A processing model for the DASH client processing model to assist in handling DASH event updates.

Abstract

A method, apparatus, and computer-readable storage medium for processing a media stream. The method may include receiving a DASH event associated with a DASH media stream, determining that the DASH event is an update to a previously received DASH event, and processing the DASH event based on an emission mode of the DASH event, the DASH event including at least one of an in-band event, a Media Presentation Description (MPD) event, or a sample of timed metadata transmitted with a first media slice in a content set, the content set including one or more media slices, and the emission mode of the DASH event includes an on-receive mode and an on-start mode.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of priority to U.S. Non-Provisional Application No. 18 / 301,596, filed April 17, 2023, which claims the benefit of priority to U.S. Provisional Application No. 63 / 332,599, filed April 19, 2022, and U.S. Provisional Patent Application No. 63 / 332,596, filed April 19, 2022, each of which is incorporated by reference in its entirety.

[0002] The present disclosure relates generally to media streaming technologies, including Dynamic Adaptive Streaming over Hypertext Transfer Protocol (DASH). More specifically, the disclosed technologies include methods and apparatus for handling DASH event updates under a DASH client processing model. [Background technology]

[0003] The Background section of this specification is intended to provide a general background to the disclosure. To the extent described in this Background section, the work of the currently named inventors and other aspects of the description that may not have been prior art at the time of filing this application are not admitted, expressly or implicitly, as prior art to the present disclosure.

[0004] The Moving Picture Expert Group (MPEG) Dynamic Adaptive Streaming over Hypertext Transfer Protocol (DASH) provides a standard for streaming multimedia content over IP networks. In the DASH standard, a Media Presentation Description (MPD) is used to provide information for a DASH client to adaptively stream media content by downloading media segments from a DASH server. The DASH standard enables streaming of multi-rate content. One aspect of the DASH standard includes the delivery of MPD events and in-band events, and a client processing model for handling these events. Summary of the Invention [Means for solving the problem]

[0005] Aspects of the present disclosure provide methods and apparatuses for media stream processing, and more particularly, for handling DASH event updates under a DASH client processing model. In some example implementations, a method for processing a media stream, such as a DASH media stream, is disclosed. The method may include receiving a DASH event associated with a DASH media stream, determining that the DASH event is an update to a previously received DASH event, and processing the DASH event based on an emission mode of the DASH event, where the DASH event includes at least one of an in-band event, a Media Presentation Description (MPD) event, or a sample of timed metadata transmitted with a first media slice in a content set, the content set including one or more media slices, and the emission mode of the DASH event includes an on-receive mode and an on-start mode.

[0006] Aspects of the present disclosure also provide a media stream processing device or apparatus that includes circuitry configured to perform any of the above method implementations.

[0007] Aspects of the present disclosure also provide a non-transitory computer-readable medium for storing instructions that, when executed by a computer for media stream processing, cause the computer to perform any of the above method implementations.

[0008] Further features, nature and various advantages of the subject matter of the present disclosure will become more apparent in the following detailed description and accompanying drawings. [Brief explanation of the drawings]

[0009] [Figure 1] 1 illustrates a system according to one embodiment of the present disclosure. [Figure 2] 1 illustrates a Dynamic Adaptive Streaming over HTTP (DASH) system according to one embodiment of the present disclosure. [Figure 3] 1 illustrates a DASH client architecture according to one embodiment of the present disclosure. [Figure 4] 1 illustrates exemplary active and posted events tables. [Figure 5a] 1 illustrates an exemplary common process for an updated event in the DASH event processing model. [Figure 5b] 10 illustrates an exemplary on-receive process for an updated event in the DASH event processing model. [Figure 5c] 10 illustrates an exemplary start-up process for an updated event in the DASH event processing model. [Figure 6] 1 shows a flowchart of a method according to one embodiment of the present disclosure. [Figure 7] 1 shows a schematic diagram of a computer system according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0010] Dynamic Adaptive Streaming over Hypertext Transfer Protocol (DASH) and Media Presentation Description (MPD) One common format for media streaming includes Dynamic Adaptive Streaming over Hypertext Transfer Protocol (DASH), as defined in ISO / IEC 23009-1. DASH is an adaptive bitrate streaming technology that enables streaming of media content using Hypertext Transfer Protocol (HTTP) infrastructure, such as web servers, content delivery networks (CDNs), and various proxies and caches. DASH supports both on-demand and live streaming from DASH servers to DASH clients and allows DASH clients to control the streaming session, so DASH servers do not need to deal with the additional burden of stream adaptation management in large-scale deployments. DASH also allows DASH clients to select streaming from various DASH servers, thus achieving further network load balancing for the benefit of DASH clients. DASH provides dynamic switching between different media tracks, for example, by varying the bitrate to adapt to network conditions.

[0011] In DASH, a Media Presentation Description (MPD) file provides information for a DASH client to adaptively stream media content by downloading media segments from a DASH server. The MPD may be in the form of an Extensible Markup Language (XML) document. The MPD file may be fragmented and delivered in parts to reduce session start delays. The MPD file may also be updated during a streaming session. In some examples, the MPD file supports content accessibility features, ratings, and camera view representation. DASH also supports the delivery of multi-view and scalable encoded content.

[0012] An MPD file may contain a sequence of one or more periods. Each of the one or more periods may be defined, for example, by a period element in the MPD file. The MPD file may contain an MPD availableStartTime attribute and a start attribute for each period. For a media presentation of dynamic type (e.g., used for live services), the sum of a period's start attribute, the MPD attribute availableStartTime, and the media segment's duration may indicate the period in Coordinated Universal Time (UTC) format, in particular the availability time of the first media segment of each representation in the corresponding period. For a media presentation of static type (e.g., used for on-demand services), the start attribute of the first period may be 0. For any other periods, the start attribute may specify a time offset between the start time of the corresponding period relative to the start time of the first period. Each period may extend until the start of the next period, or, in the case of the last period, until the end of the media presentation. The period start time may be precise and may reflect the actual timing resulting from the playback of the media of all previous periods. In an example implementation, an MPD is provided such that the next period is a continuation of the content in the previous period, possibly the immediately following period or a later period (e.g., after an advertising period has been inserted).

[0013] Each period can contain one or more adaptation sets, each of which can contain one or more representations of the same media content. A representation can be one of several alternative encoded versions of the audio or video data. Representations can vary by encoding type, e.g., by bitrate, resolution, and / or codec of the video data and bitrate, and / or codec of the audio data. The term representation can be used to refer to a section of encoded audio or video data that corresponds to a particular period of multimedia content and is encoded in a particular way.

[0014] An adaptation set for a particular period can be assigned to a group indicated by the group attribute in the MPD file. Adaptation sets within the same group are generally considered alternatives to each other. For example, each adaptation set for video data for a particular period can be assigned to the same group so that any adaptation set can be selected for decoding to display the video data of the multimedia content for the corresponding period. In some examples, media content within a period can be represented by either one adaptation set from group 0, if present, or a combination of at most one adaptation set from each non-zero group. Timing data for each representation of a period can be expressed relative to the start time of the period.

[0015] A representation can contain one or more segments. Each representation can contain an initialization segment, or each segment of a representation can be self-initializing. If present, the initialization segment can contain initialization information for accessing the representation. In some cases, the initialization segment does not contain media data. A segment can be uniquely referenced by an identifier such as a Uniform Resource Locator (URL), Uniform Resource Name (URN), or Uniform Resource Identifier (URI).

[0016] In an exemplary implementation, the URL is defined according to IETF RFC 3986, with a fixed scheme, e.g., "http" or "https." <absolute-uri>, possibly constrained by a byte range if a range attribute is provided with the URL. The byte range may be expressed as a byte-range-spec, for example as specified in IETF RFC 2616. This may be constrained to a single expression identifying a contiguous byte range. In one embodiment, the segment may be included in an MPD with a data URL, for example as defined in IETF RFC 2397.

[0017] An MPD file can provide an identifier for each segment. In some examples, an MPD file can also provide byte ranges in the form of range attributes that can correspond to data for a segment within a file that is accessible by a URL, URN, or URI.

[0018] A sub-representation can be embedded (or contained) in a regular representation and described by a sub-representation element (e.g., SubRepresentation). A sub-representation element can describe properties of one or more media content components embedded in the representation. For example, a sub-representation element can describe the properties of an embedded audio component (e.g., codec, sampling rate, etc.), an embedded subtitle (e.g., codec), or a sub-representation element can describe some embedded lower-quality video layers (e.g., some lower frame rate, etc.). Sub-representations and representation elements can share some common attributes and elements.

[0019] Each representation may also include one or more media components, each of which may correspond to an encoded version of one particular media type, such as audio, video, or timed text (e.g., for closed captioning). Media components may be temporally contiguous across boundaries of consecutive media segments within a representation.

[0020] In some example implementations, a DASH client can access and download an MPD file from a DASH server. That is, the DASH client can obtain an MPD file used to start a live session. Based on the MPD file and for each selected representation, the DASH client can make several decisions, including determining what the latest segment is available on the server, determining the segment availability start time of the next segment and possibly future segments, determining from which timeline within the segment to start playing the segment, and determining when to obtain / fetch a new MPD file. As the service plays, the client can track drift between the live service and its own playback, which needs to be detected and compensated for.

[0021] DASH Events In a DASH system, events provide a means of signaling additional information to DASH clients and their associated applications. In an exemplary implementation, events are timed and therefore have a start time and a duration. Event information can include metadata describing the content of the media presentation. Additionally or alternatively, event information can include control messages for the media player associated with specific times during playback of the media presentation, such as ad insertion cues. Events may be realized, for example, as MPD events or in-band events. They can be part of a manifest file (e.g., MPD) or embedded in an ISOBMFF-based media file such as an event message (emsg) box.

[0022] A Media Presentation Description (MPD) event is an event that can be signaled in the MPD. A set of events assigned to a media presentation time can be provided in the MPD at the period level. Events of the same type can be specified in an event stream element (e.g., EventStream) within a period element. An event ends at the end of the period, even if its start time is after the period boundary or its duration extends beyond the period boundary. An event stream element contains a message scheme identification (e.g., @schemeIdUri) and an optional value for the event stream element (e.g., @value). Furthermore, since event streams contain timed events, a timescale attribute (e.g., @timescale) can be provided to assign the event to a specific media presentation time within the period. The timed event itself can be described by an event element contained in the event stream element.

[0023] In-band event streams can be multiplexed with representations by adding event messages as part of media segments. An event stream can be present in selected representations, only one or a few selected adaptation sets, or all representations. For example, one possible configuration is where only the audio adaptation set contains in-band events, or only the video adaptation set contains in-band events. In-band event streams present within a representation can be indicated by in-band event stream elements (e.g., InbandEventStream) at various levels, such as the adaptation set level or the representation level. Furthermore, a representation can contain multiple in-band event streams, each indicated by a separate in-band event stream element.

[0024] 1 illustrates a system 100 according to one embodiment of the present disclosure. The system 100 includes a content server 110 and an information processing device 120. The content server 110 can provide a content stream including primary content (e.g., a main program) and one or more timed metadata tracks.

[0025] The information processing device (120) can interface with the content server (110). For example, the information processing device (120) can play content received from the content server (110). The content can be played based on a manifest file (e.g., an MPD) received by the information processing device (120) (e.g., from the content server (110)). The manifest file can further include signaling for one or more timed metadata tracks.

[0026] An exemplary DASH system is shown in Figure 2. The DASH system (200) may include a content server (210), an advertisement server (220), and an information processing device (230) connected to a network (250). The DASH system (200) may also include one or more supplemental content servers.

[0027] The content server (210) can provide primary content (e.g., a main program) and a manifest file (e.g., an MPD) to the information processing device (230). The manifest file can be generated, for example, by an MPD generator (214). In other embodiments, the primary content and the manifest file can be provided by different servers.

[0028] The information processing device (230) can receive the MPD and retrieve primary content from the HTTP server (212) of the content server (210) based on the MPD. The MPD can be processed by a DASH client (232) running on the information processing device (230). The DASH client (232) can also retrieve advertising content from the advertising server (220) and other content (e.g., interactive content) from one or more supplemental content servers. The main content and advertising content can be processed by the DASH client (232) and output for display on a display device (236). The display device (236) can be integrated into the information processing device (230) or external to the information processing device (230). The DASH client (232) can also extract event information from one or more timed metadata tracks and send the extracted event information to an application (234) for further processing. The application (234) can be configured to, for example, display supplemental content based on the event information.

[0029] The advertisement server (220) can store advertisement content in an advertisement storage unit such as a memory, etc. The information processing device (230) can request the stored advertisement content based on the event information.

[0030] 3 illustrates an exemplary DASH client architecture for processing DASH events, according to one embodiment of the present disclosure. A DASH client (or DASH player) communicates with an application (390) and can be configured to process various types of events, including (i) MPD events, (ii) in-band events, and (iii) timed metadata events.

[0031] The manifest parser (305) parses a manifest (e.g., an MPD). The manifest is provided, for example, by a content server (110, 210). The manifest parser (305) extracts event information about MPD events, in-band events, and timed metadata events embedded in timed metadata tracks. The extracted event information can be provided to the DASH logic (310) (e.g., the control, selection, and heuristic logic of a DASH player). Based on the event information, the DASH logic (310) can notify the application (390) of the event scheme signaled in the manifest.

[0032] The event information can include event scheme information for distinguishing between different event streams. An application (390) can use the event scheme information to subscribe to event schemes of interest. The application (390) can further indicate a desired delivery mode for each of the subscribed schemes via one or more subscription APIs. For example, the application (390) can send a subscription request to a DASH client that identifies one or more event schemes of interest and any desired corresponding delivery modes.

[0033] If an application (390) subscribes to one or more event schemes that are delivered as part of one or more timed metadata tracks, the in-band event and "moof" parser (325) can stream one or more timed metadata tracks to the timed metadata track parser (330). For example, the in-band event and "moof" parser (325) parses the movie fragment box ("moof") and subsequently parses the timed metadata tracks based on control information from the DASH logic (310).

[0034] The timed metadata track parser (330) can extract event messages embedded in the timed metadata tracks. The extracted event messages can be stored in the event and timed metadata buffer (335). The synchronizer / dispatcher module (340) (e.g., the event and timed metadata synchronizer and dispatcher) can emit (or send) subscribed events to the application (390).

[0035] The MPD events described in the MPD can be parsed by the manifest parser (305) and stored in a buffer (335). For example, the manifest parser (305) parses each event stream element of the MPD and each event described in each event stream element. For each event signaled in the MPD, event information such as presentation time and event duration can be stored in a buffer (335) associated with the event.

[0036] The in-band event and "moof" parser (325) can parse the media segments to extract in-band event messages. Any such identified in-band events and associated presentation times and durations can be stored in a buffer (335).

[0037] Thus, the buffer (335) can store MPD events, in-band events, and / or timed metadata events therein. The buffer (335) can be, for example, a first-in, first-out (FIFO) buffer. The buffer (335) can be managed in correspondence with the media buffer (350). For example, as long as a media segment is present in the media buffer (350), any events or timed metadata corresponding to that media segment can be stored in the buffer (335).

[0038] The DASH Access Application Programming Interface (API) (315) can manage the fetching and reception of content streams (or data flows) containing media content and various metadata via the HTTP protocol stack (320). The DASH Access API (315) can separate the received content stream into different data flows. The data flows provided to the in-band event and moof parser can include media segments, one or more timed metadata tracks, and in-band event signaling contained in the media segments. In one embodiment, the data flows provided to the manifest parser (305) can include an MPD.

[0039] The DASH Access API (315) can forward the manifest to the manifest parser (305). In addition to describing events, the manifest can also provide information about media segments to the DASH logic (310), which can communicate with the application (390) and the in-band event and "moof" parser (325). The application (390) can be associated with media content processed by the DASH client. Control and synchronization signals exchanged between the application (390), DASH logic (310), manifest parser (305), and DASH Access API (315) can control the fetching of media segments from the HTTP stack (320) based on the information about the media segments described in the manifest.

[0040] The in-band event and moof parser (325) can parse the media data flow into media segments containing media content, timed metadata in timed metadata tracks, and any signaled in-band events within the media segments. The media segments containing the media content can be parsed by the file format parser (345) and stored in the media buffer (350).

[0041] The events stored in the buffer (335) can enable the synchronizer / dispatcher (340) to communicate available events relevant to (or of interest to) the application via an event / metadata API. Applications can be configured to process available events (e.g., MPD events, in-band events, or timed metadata events) and subscribe to specific events or timed metadata by notifying the synchronizer / dispatcher (340). Any events stored in the buffer (335) that are not relevant to an application but instead are relevant to the DASH client itself can be forwarded by the synchronizer / dispatcher (340) to the DASH logic (310) for further processing.

[0042] In response to an application (390) subscribing to a particular event, the synchronizer / dispatcher (340) can communicate the application's event instances (or samples of timed metadata) corresponding to the event scheme to which the application subscribes. The event instances can be communicated according to a dispatch mode indicated by the subscription request or a default dispatch mode (e.g., for a particular event scheme). For example, in a dispatch-on-receive mode, event instances can be sent to the application (390) upon receipt in the buffer (335). Meanwhile, in a dispatch-on-start mode, event instances can be sent to the application (390) at their associated presentation times, for example, synchronized with a timing signal from the media decoder (355).

[0043] In some example implementations, a DASH player processes the received MPD. For every period, the MPD can contain one or more event streams. (Each event stream can be scoped by a scheme / value pair.) Each period can contain one or more adaptation sets that carry representations / tracks for timed metadata. Some or all of these event streams / timed metadata tracks may be suitable for consumption by an application.

[0044] DASH player-specific events are emitted to the DASH player's control, selection, and heuristic logic ( FIG. 3 , DASH logic 310), and application-related events and timed metadata track samples are emitted to applications as follows: When an application subscribes to a particular event stream or timed metadata stream, the corresponding event instances or timed metadata samples may be emitted to the application according to the emission mode.

[0045] In emit-on-receive mode, the DASH client (or DASH player) should emit the entire event or timed metadata information before the Latest Arrival Time (LAT) or as soon as possible. The application that receives the event prepares for the event from the LAT until the Start Time (ST) of the event. In some example implementations, the LAT is the latest time the event should be delivered to the application and is used to ensure that the application has enough time to prepare for the event. In some example implementations, the DASH client can emit the entire event or timed metadata information as soon as it is appended to the event and timed metadata buffer.

[0046] In the at-start dispatch mode, the DASH client (or DASH player) should dispatch the event exactly at ST, the start / presentation time of the event / metadata sample. The DASH client (or DASH player) should dispatch the event to the application at the presentation time of the corresponding media sample. If the start time of the event has passed but the current moment is within the event duration, the DASH client can dispatch the event at the earliest time within the event duration.

[0047] In FIG. 3, the file format parser 345, the media buffer 350, and the media decoder 355 may together form an MSE buffer.

[0048] MPD Events Events can be signaled in the MPD. A series of events assigned to a media presentation time can be provided in the MPD at the Period level. Events of the same type can be summarized into an event stream specified by an EventStream element within a Period element. For example, an event can end at the end of a Period even if its start time is after the Period boundary or if the event's duration extends beyond the Period boundary.

[0049] In some example implementations, in a period, all events of the same type may be clustered into one event stream. A period can have multiple event streams, and each event stream can have a different combination of values ​​for the @schemeIdUri attribute and the @value attribute.

[0050] Table 1 provides example event stream semantics.

[0051] [Table 1A] [Table 1B]

[0052] Table 2 provides example event semantics according to one embodiment of the present disclosure.

[0053] [Table 2A] [Table 2B]

[0054] In-band event signaling and event message boxes In some example implementations, an event stream may be multiplexed with a representation by adding an event message as part of a segment. The event stream can be present in selected representations, only one or a few selected adaptation sets, or all representations.

[0055] In some example implementations, an in-band event stream may be present within a representation, which, if expected to be processed by a DASH client, may be indicated by an InbandEventStream element in the adaptation set or at the representation level.

[0056] An InbandEventStream can be defined based on the semantics of EventStream, as shown in Table 1. Further restrictions can be added to the InbandEventStream-specific semantics.

[0057] The event message box ("emsg") may be used, for example, to provide signaling of general events related to media presentation time. The same semantics may apply as for the events defined in Tables 1 and 2.

[0058] A Media Segment, if based on an ISO BMFF container, may contain one or more event message ("emsg") boxes. If present, the "emsg" boxes may be arranged as follows:

[0059] It may be placed before the first "moof" box in the segment, or

[0060] It may be placed between any "mdat" (media data) box and a "moof" box, in which case the first "moof" box of any segment shall be preceded by an equivalent "emsg" with the same id value.

[0061] In some example implementations, "emsg" may be defined as follows: Box Type: Box Type: "emsg" Container:Segment Required: No Quantity: 0 or more

[0062] Table 3 shows an example of the "emsg" syntax.

[0063] [Table 3]

[0064] For example, the semantics of the above syntax are explained below: scheme_id_uri is a string that identifies a message scheme. The semantics and syntax of message_data[] are defined, for example, by the owner of the identified scheme. The string may use URN or URL syntax. A URL may resolve to an Internet location, and the resolving location may store the message scheme specification. value is a string that specifies the value of the event. The value space and semantics may be defined by the owner of the scheme identified in the scheme_id_uri field. timescale provides the timescale of the event duration and presentation_time_delta or presentation_time fields in ticks per second. The value should be the same as the timescale of the tracks contained in the transport segment. In some implementations, the value should be the same for all events in an event stream. presentation_time_delta provides the media presentation time delta between the media presentation time of the event and the earliest presentation time in this segment. If a segment index is present, the earliest presentation time is determined by the field earliest_presentation_time of the first 'sidx' box. If a segment index is not present, the earliest presentation time is determined as the earliest presentation time of any access unit within the media segment. The timescale is provided in the timescale field. presentation_time provides the media presentation time of the event measured on the Movie timeline within the timescale provided in the timescale field, and adjusted by InbandEventStream@presentationTimeOffset within the timescale provided by InbandEventStream@timescale. The value must not be less than the earliest presentation time of the transport segment. event_duration provides the duration of the event in media presentation time. The timescale is indicated in the timescale field. In some example implementations, the value 0xFFFFFFFF indicates an unknown duration. The interpretation of this value must be defined by the owner of the event method. id: A field that identifies this instance of a message. The scope of this identifier for each event is the same scheme_id_uri and value pair. Messages with the same id within the same scheme_id_uri and value pair range are equivalent, i.e. processing any one event message box with the same id is sufficient. message_data: The body of the message that fills the rest of the message box. This may be empty depending on the information above. The syntax and semantics of this field must be defined by the owner of the scheme identified in the scheme_id_uri field. flags may be a variable with multiple bit fields (e.g., 8-bit characters). Each bit may be used to indicate the state, condition, or type of the emsg. In an example implementation, (flags&1) equal to 1 indicates that the emsg is an update of another emsg with identical values ​​of scheme_id_uri, value, and id fields. "&" is the bitwise OR operator. Other bits in flags may also be selected to indicate that the emsg is an update.

[0065] In a DASH system, in addition to signaling events, it may be beneficial to have the ability to dynamically update previously signaled events. For example, the start time, duration, and / or media content specified by an event may be updated. In some scenarios, there is breaking news that may be signaled as an event. The news may need to be updated with the latest developments. Thus, the event may need to be updated accordingly. In some scenarios, an already scheduled event may need to be removed, for example, due to a schedule update, or a conflict occurs.

[0066] In the current DASH system, the updated event is not clearly defined. DASH clients can be implemented by many different vendors on many different platforms (e.g., operating systems, browsers, etc.), so cross-vendor or cross-platform compatibility cannot be achieved. The signaled updated event may not be identified or recognized by a specific DASH client. Furthermore, the behavior for handling the updated event is also not defined, which causes further inconsistency in vendor-dependent DASH client implementations.

[0067] In this disclosure, various embodiments are described for accurately defining and identifying updated events. Some embodiments related to the DASH client processing model are described for updated event support, including monitoring and identifying updated events, updating an event buffer, and emitting an updated event only if a previous event has not previously been emitted.

[0068] In some example implementations, an MPD updated event is defined. An Event with @status='update' is an updated instance of an Event with the same @schemeIdUri, @value, and @id attributes that may have been previously processed by a DASH client. If the previous event has not yet been dispatched, the DASH client may replace the previous event with the updated instance. An Event with @status='update' may differ from the previous event except for the @schemeIdUri, @value, and @id attributes.

[0069] In some example implementations, an in-band updated event is defined. An emsg box with (flags&1)=1 (& is the bitwise AND operator) is an updated instance of an emsg box with the same scheme_id_uri, value, and id fields that may have been previously processed by a DASH client. If the previous event has not yet been dispatched, the DASH client can replace the previous event with the updated instance. The updated emsg may differ from the previous emsg except for the scheme_id_uri, value, and id fields.

[0070] In some example implementations, when replacing an event, the event and any old copies of the event stored in the timed metadata buffer may be updated with the updated instance of the event.

[0071] In some example implementations, when replacing an event, the (possibly updated) presentation time of the updated event may be checked. The updated event may be added to the event and timed metadata buffer according to its presentation time, and the old event may be removed from the event and timed metadata buffer. Note that because the presentation time may be updated, the updated event may have a different position in the event and timed metadata buffer.

[0072] This disclosure improves the DASH client event processing model to support updated events. For example, when one or more updated versions of an event are received, the DASH client can emit only the latest event updates and avoid emitting old / obsolete events.

[0073] As described above, the event processing model (hereinafter also referred to as the processing model) can process events in two dispatch modes: on-receive mode and on-start mode. The processing model shares common processing applicable to both dispatch modes and then uses separate processing for each dispatch mode.

[0074] 3, an application 390 subscribes to a particular event stream identified by a (scheme_uri / value) pair with a particular dispatch mode of either initiating or receiving. The method may be indicated, for example, by the @schemeIdUri attribute (or field) of EventStream or the scheme_id_uri field defined in "emsg." Common processing and dispatch mode dependent processes are described below.

[0075] In some example implementations, an EventStream acts as a container for events of the same type. An EventStream may be identified by a (scheme id / value) pair or by just the scheme id. The scheme id may include, for example, a scheme_id_uri.

[0076] overhead processing In some example implementations, a DASH client can set up an active event table for each subscribed EventStream identified by a (scheme_uri / value) pair. For example, an active event table may only need to be configured for events with an emit mode at start. The active event table maintains a single list of event ids for events waiting to be emitted. Figure 4 shows an example active event table configuration. In Figure 4, k active event tables are created, each corresponding to a unique (scheme_uri / value) pair. Each table may be implemented, for example, as a list. Each list can maintain a list of event identifiers (e.g., Event@id, "emsg" id) that identify events waiting to be emitted.

[0077] In some example implementations, the DASH client can also set up a dispatched events table for each subscribed EventStream identified by a (scheme_uri / value) pair. The dispatched events table maintains a single list of event ids for dispatched events. Figure 4 shows an example dispatched events table configuration. In Figure 4, x dispatched events tables are created, each corresponding to a unique (scheme_uri / value) pair. Each table may be implemented, for example, as a list. Each list can maintain a list of event identifiers (e.g., Event@id, "emsg" id) that identify the dispatched events.

[0078] The DASH client parses the event (e.g., sample "emsg" / timed metadata) and obtains the scheme_uri / (value).

[0079] If no applications are subscribed to the scheme_uri / (value) pair, the DASH client can finish processing this event. The DASH client then proceeds to derive / obtain the start time (ST or presentation time) of the event instance / metadata sample and derives the end time (ET) of the event instance / metadata sample using the formula ET=ST+DU, where DU is the event duration, which may be signaled in the event.

[0080] FIG. 5a illustrates exemplary common processing logic.

[0081] Receiving process In some example implementations, when the transmission mode is receive, the DASH client can implement the following process.

[0082] Step 1: If the current presentation time value is greater than the ET of the updated event, then end the process.

[0083] Step 2: Check whether the newly received updated event has been dispatched. The DASH client can compare the id of the event with an entry in the dispatched events table that is identified by the same scheme_uri / (value) pair as the updated event. If an entry with the same id value exists, end the process.

[0084] Step 3: Publish the event / timed metadata including ST, id, DU, timescale, and message_data (i.e., the updated version in the updated event) and add the event to the corresponding published events table identified by the same scheme_uri / (value) pair as the updated event.

[0085] FIG. 5b illustrates exemplary receive processing logic.

[0086] Start-up processing In some example implementations, when the origination mode is at initiation, the DASH client can implement the following example process.

[0087] Step 1: If the event is an update, remove the deferred event (which is replaced by the updated event) with the same id from the corresponding active events table that is identified by the same scheme_uri / (value) pair as the updated event, if any.

[0088] Step 2: Derive / obtain the ST for the event instance / metadata sample.

[0089] Step 3: If the current media presentation time value is less than ST, go to step 6.

[0090] Step 4: Derive the event end time: ET = ST + DU.

[0091] Step 5: If the current presentation time value is greater than ET, end the process.

[0092] Step 6: Compare the id of the event with the corresponding active events table and corresponding dispatched events table entries (identified by the same scheme_uri / (value) pair as the updated event). If an entry with the same event id value exists, the process ends, or if an entry with the same event id value exists in the corresponding dispatched event table, the process ends. Otherwise, add an event id (e.g., Event@id, "emsg" id) identifying the updated event instance / metadata sample to the corresponding active event table. In some example implementations, the updated event may be added to an event and time metadata buffer, and the deferred event may be removed from the same buffer.

[0093] Step 7: Dispatch the updated event / metadata (eg, message_data) at time ST, or immediately if the current presentation time is greater than ST.

[0094] The DASH client can add the updated event (eg, Event@id, "emsg" id) to the corresponding posted events table after the updated event is posted.

[0095] Optionally, the DASH client may further check whether there is an entry for the updated event in the corresponding active event table, and if there is an entry, the DASH client may remove the entry from there.

[0096] FIG. 5c illustrates exemplary start-up processing logic.

[0097] The steps described above for common processing, on-receive processing, and on-init processing are for illustrative purposes only. Other implementations may include, for example, a subset of the steps.

[0098] In this disclosure, the terms "field" or "attribute" may be interchangeable for in-band events, MPD events, and samples of timed metadata. An event may generally refer to an MPD event, an in-band event, or a sample of timed metadata, unless otherwise specified.

[0099] In some example implementations, the in-band events and the MPD events are each identified by or associated with a scheme identifier, which may include, for example, a scheme identifier Uniform Resource Identifier (URI).

[0100]

[0023] Embodiments of the present disclosure apply to DASH and other media streaming technologies / standards. Figure 6 shows an example method 600 for processing a media stream, such as a DASH media stream. The method may be performed by a DASH client hosted by a media streaming device in a media streaming system, and may include some or all of the following: step 610: receiving a DASH event associated with a DASH media stream; step 620: determining that the DASH event is an update to a previously received DASH event; and step 630: processing the DASH event based on an emission mode of the DASH event, where the DASH event is an in-band event transmitted with a first media slice in a content set, the content set including one or more media slices, including at least one of an in-band event, a media presentation description (MPD) event, or a sample of timed metadata, and the emission mode of the DASH event includes an on-receive mode and an on-start mode.

[0101] The embodiments of the present disclosure may be used separately or combined in any order. Furthermore, each of the methods (or embodiments) and DASH clients may be implemented by processing circuitry (e.g., one or more processors or one or more integrated circuits). In one example, the one or more processors execute a program stored on a non-transitory computer-readable medium. The embodiments of the present disclosure may be applied to DASH and / or other media streaming technologies / standards.

[0102] The techniques described above may be implemented as computer software using computer-readable instructions and physically stored on one or more computer-readable media. For example, Figure 7 illustrates a computer system (1800) suitable for implementing certain embodiments of the disclosed subject matter.

[0103] Computer software can be coded using any suitable machine code or computer language that is amenable to mechanisms such as assembly, compilation, linking, etc., to create code containing instructions that can be executed by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc., directly or via interpretation, microcode execution, etc.

[0104] The instructions may be executed on various types of computers or computer components including, for example, personal computers, tablet computers, servers, smartphones, gaming consoles, Internet of Things devices, and the like.

[0105] 7 for computer system (1800) are exemplary in nature and are not intended to suggest any limitation as to the scope of use or functionality of the computer software implementing embodiments of the present disclosure, nor should the arrangement of components be interpreted as having any dependency or requirement relating to any one or combination of components illustrated in the exemplary embodiment of computer system (1800).

[0106] The computer system (1800) may include several human interface input devices. Such human interface input devices may be responsive to input by one or more human users through, for example, tactile input (e.g., keystrokes, swipes, data glove movements, etc.), audio input (e.g., voice, clapping, etc.), visual input (e.g., gestures), and olfactory input (not shown). The human interface devices may also be used to capture certain media not necessarily directly associated with conscious human input, such as audio (e.g., voice, music, ambient sounds), images (e.g., scanned images, photographic images obtained from still image cameras), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic video, etc.).

[0107] The input human interface devices may include one or more of a keyboard (1801), a mouse (1802), a trackpad (1803), a touch screen (1810), a data glove (not shown), a joystick (1805), a microphone (1806), a scanner (1807), and a camera (1808) (only one of each is shown).

[0108] Additionally, the computer system (1800) may include several human interface output devices. Such human interface output devices may stimulate one or more of the human user's senses through, for example, tactile output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touchscreen (1810), data gloves (not shown), or joystick (1805), although haptic feedback devices that do not function as input devices may also be present), audio output devices (speakers (1809), headphones (not shown), etc.), visual output devices (such as screens (1810) including CRT, LCD, plasma, or OLED screens, each with or without touchscreen input capability and each with or without haptic feedback capability, some of which may be capable of outputting two-dimensional visual output or output in more than three dimensions by means of stereoscopic graphic output, virtual reality glasses (not shown), holographic displays, or smoke tanks (not shown), etc.), and printers (not shown).

[0109] The computer system (1800) may also include human-accessible storage devices and their associated media, such as optical media (1821) including CD / DVD ROM / RW (1820) using CD / DVD or similar media, thumb drives (1822), removable hard drives or solid state drives (1823), legacy magnetic media (not shown) such as tape and floppy disks, and specialized ROM / ASIC / PLD-based devices (not shown) such as security dongles.

[0110] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the subject matter of this disclosure does not encompass transmission media, carrier waves, or other transitory signals.

[0111] The computer system 1800 may also include an interface 1854 to one or more communications networks 1855. The networks may be, for example, wireless, wired, or optical. The networks may further be local, wide-area, metropolitan, vehicular, industrial, real-time, delay-tolerant, etc. Examples of networks include local area networks such as Ethernet; cellular networks including WLAN, GSM, 3G, 4G, 5G, LTE, etc.; television wired or wireless wide-area digital networks including cable, satellite, and terrestrial television; and vehicular and industrial networks including CAN bus. Some networks require an external network interface adapter that attaches in a standard manner to some general-purpose data port or peripheral bus 1849 (e.g., a USB port on the computer system 1800); other networks are typically integrated into the core of the computer system 1800 by attaching to a system bus, as described below (e.g., an Ethernet interface for a PC computer system or a cellular network interface for a smartphone computer system). Any of these networks may be used by the computer system 1800 to communicate with others. Such communication may be one-way receive-only (e.g., television broadcast), one-way transmit-only (e.g., from the CANbus to a specific CANbus device), or bidirectional to other computer systems, for example, using local or wide-area digital networks. Specific protocols and protocol stacks may be used with each of these networks and network interfaces, as described above.

[0112] The above-mentioned human interface devices, human-accessible storage devices, and network interfaces may be attached to the core (1840) of the computer system (1800).

[0113] The cores (1840) may include one or more central processing units (CPUs) (1841), graphics processing units (GPUs) (1842), specialized programmable processing units in the form of field programmable gate arrays (FPGAs) (1843), task-specific hardware accelerators (1844), graphics adapters (1850), etc. These devices may be connected via a system bus (1848), along with read-only memory (ROM) (1845), random access memory (1846), and internal mass storage devices (1847) such as internal hard drives and SSDs that are not user-accessible. In some computer systems, the system bus (1848) may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripheral devices may be attached directly to the core's system bus (1848) or via a peripheral bus (1849). In one example, a screen (1810) may be connected to the graphics adapter (1850). Architectures for peripheral buses include PCI, USB, etc.

[0114] The CPU (1841), GPU (1842), FPGA (1843), and accelerator (1844) can execute several instructions that can be combined to form the above-mentioned computer code. This computer code can be stored in ROM (1845) or RAM (1846). Transient data can also be stored in RAM (1846), while permanent data can be stored, for example, in internal mass storage (1847). Rapid storage and retrieval of any memory device can be enabled through the use of cache memory, which can be closely associated with one or more of the CPU (1841), GPU (1842), mass storage (1847), ROM (1845), RAM (1846), etc.

[0115] The computer-readable medium may bear computer code for performing various computer-implemented operations. The medium and computer code may be those specially designed and constructed for the purposes of the present disclosure, or they may be of the kind well known and available to those skilled in the computer software arts.

[0116] As a non-limiting example, a computer system having architecture (1800), and in particular core (1840), can provide functionality as a result of one or more processors (including CPUs, GPUs, FPGAs, accelerators, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media can be media associated with mass storage devices directly accessible by a user, as described above, or even specific storage devices of the core (1840), such as the core's internal mass storage device (1847) or non-transitory storage devices, such as ROM (1845). Software implementing various embodiments of the present disclosure can be stored on such devices and executed by the core (1840). The computer-readable media can include one or more memory devices or chips, depending on particular needs. Software enables the cores (1840), and in particular the processors (including CPUs, GPUs, FPGAs, etc.) in the cores (1840), to execute the particular processes or particular portions of the particular processes described herein, including defining data structures stored in RAM (1846) and modifying such data structures in accordance with the processes defined by the software. Additionally or alternatively, logic may be hardwired into or otherwise implemented in circuitry (e.g., accelerators (1844)) that can act in place of or cooperate with software to execute the particular processes or particular portions of the particular processes described herein, thereby providing the computer system with functionality. Where appropriate, references to "software" may encompass logic, and vice versa. References to computer-readable media may encompass circuitry (e.g., integrated circuits (ICs)) that stores software for execution, circuitry that embodies logic for execution, or both, where appropriate. The present disclosure encompasses any appropriate combination of hardware and software.

[0117] While this disclosure describes several exemplary embodiments, alterations, substitutions, and various substitute equivalents exist and fall within the scope of this disclosure. It will thus be appreciated that those skilled in the art will be able to devise numerous systems and methods that, although not explicitly shown or described herein, embody the principles of the present disclosure and are therefore within the spirit and scope of the present disclosure. [Explanation of symbols]

[0118] 100 systems 110 Content Server 120 Information processing equipment 200 DASH System 210 Content Server 212 HTTP Server 214 MPD Generator 220 Ad Server 230 Information processing equipment 232 DASH client 234 Applications 236 Display Devices 250 Network 305 Manifest Parser 310 DASH Logic 315 DASH Access API 320 HTTP stack 325 in-band events and the 'moof' parser 330 Timed Metadata Track Parser 335 Event and Timed Metadata Buffers 340 Event / Metadata Synchronizer / Dispatcher 345 File Format Parser 350 Media Buffer 355 Media Decoder 390 Applications 1800 Computer Systems 1801 keyboard 1802 Mouse 1803 Trackpad 1805 Joystick 1806 Mike 1807 Scanner 1808 Camera 1809 Speaker 1810 screen 1820 CD / DVD ROM / RW 1821 Optical media 1822 thumb drive 1823 Removable Hard Drive or Solid State Drive 1840 Core 1841 Central Processing Unit (CPU) 1942 Graphics Processing Unit (GPU) 1843 Field Programmable Gate Area (FPGA) 1844 Accelerator 1845 ROM 1846 RAM 1847 mass storage 1849 Peripheral Bus 1850 graphics adapter 1854 Interface 1855 Network Interface

Claims

1. 1. A method for processing a Dynamic Adaptive Streaming over HTTP (DASH) media stream executed by a DASH client hosted by a media streaming device in a media streaming system, the method comprising: receiving a DASH event associated with the DASH media stream; determining that the DASH event is an update to a previously received DASH event; determining that the DASH event is an updated version based on one of a status field of the DASH event or a flag associated with the DASH event; determining that the DASH event is the update to the previously received event in response to a combination of fields in the DASH event matching a combination of fields in the previously received event, the combination of fields including at least one of a scheme identifier field, a value field, or an ID field; and processing the DASH event based on a dispatch mode of the DASH event; wherein the DASH event comprises: an in-band event sent with a first media slice in a content set, the content set including one or more media slices; a Media Presentation Description (MPD) event, or Timed Metadata Sample and The transmission mode of the DASH event includes an on-receive mode and an on-start mode. method.

2. A method for processing a Dynamic Adaptive Streaming over HTTP (DASH) media stream executed by a DASH client hosted by a media streaming device in a media streaming system, the method comprising: receiving a DASH event associated with the DASH media stream; determining that the DASH event is an update to a previously received DASH event; processing the DASH event based on a dispatch mode of the DASH event, wherein the dispatch mode of the DASH event is a start mode; removing the previously received event from an active event list in response to the previously received event being in the active event list associated with the same scheme identifier and value pair as the DASH event, the active event list including a list of events to be dispatched; Including steps and wherein the DASH event comprises: an in-band event sent with a first media slice in a content set, the content set including one or more media slices; a Media Presentation Description (MPD) event, or Timed Metadata Sample at least one of: method.

3. determining that the event is the updated version, in response to the event being an MPD event, determining that the event is the updated version based on a state field of the event; or The method of claim 1 , comprising: in response to the event being an in-band event, determining that the event is the updated version based on a flag associated with the event.

4. a current media presentation time of the DASH media stream that is less than the start time of the event; adding the previously received event to the active event list in response to the event not being in a dispatched event list associated with the same scheme identifier and value pair as the event, the dispatched event list including a list of dispatched events. The method of claim 2 further comprising:

5. halting processing of the event in response to the previously received event being in a posted event list associated with the same scheme identifier and value pair as the event. The method of claim 2 further comprising:

6. broadcasting the event at the start time of the event; adding the event to the dispatched events list associated with the same scheme identifier and value pair as the event; The method of claim 4 further comprising:

7. After the step of transmitting the event at the start time of the event, the method further comprises: removing the event from the active event list. The method of claim 6 further comprising:

8. a current media presentation time of the DASH media stream that is greater than a start time of the event; deriving an end time of the event based on the start time of the event and a duration of the event; in response to the current media presentation time of the DASH media stream being less than the end time of the event and the previously received event not being in a posted event list associated with the same scheme identifier and value pair as the event. At the start time of the event, dispatching the event to applications subscribed to the event type to which the event belongs; adding the event to the dispatched events list associated with the same scheme identifier and value pair as the event; The method of claim 2 further comprising:

9. Prior to the step of posting the event at the start time of the event, the method further comprises the step of: adding the event to the active event list. further comprising After the step of transmitting the event at the start time of the event, the method further comprises: removing the event from the active event list. The method of claim 8 further comprising:

10. The transmission mode of the event is the on-receive mode, and In response to the previously received event not being in a posted event list associated with the same scheme identifier and value pair as the event, transmitting the event to applications subscribed to the event type to which the event belongs; adding the event to the dispatched events list associated with the same scheme identifier and value pair as the event; The method of claim 1 , comprising:

11. 11. A device for processing DASH media streams, said device comprising: a memory for storing computer instructions; and a processor in communication with said memory, said processor, when executing said computer instructions, causing said device to perform a method according to any one of claims 1 to 10.

12. 11. A computer program comprising computer readable instructions which, when executed by a processor in a device for processing DASH media streams, cause the processor to perform the method of any one of claims 1 to 10.

Images