Method, apparatus and non-transitory computer readable storage medium for receiving media data

Abstract

Aspects of the disclosure provide methods, apparatuses and non-transitory computer-readable storage medium for receiving media data. An apparatus includes processing circuitry that receives a timed metadata track comprising a plurality of segments of a plurality of metadata samples, each of the plurality of segments including only one of the plurality of metadata samples, each of the plurality of metadata samples including one or more event message boxes. A fragmentation and reassembly process is performed on the timed metadata track. The processing circuitry determines a start time and an active duration of each event message box. The processing circuitry processes event information included in the event message box based on the start time and the active duration of the event message box.

Description

[0001] INCORPORATED BY REFERENCE

[0002] This application claims priority to U.S. Patent Application No. 17 / 031,227, filed September 24, 2020, entitled “METHODS AND APPARATUS FOR HTTP BASED DYNAMIC ADAPTIVE STREAMING,” which claims priority to U.S. Provisional Application No. 62 / 908,238, filed September 30, 2019, entitled “SINGLE-SAMPLE SPARSE TIMING METADATA DASH SEGMENTS AND CMAF SEGMENTS WITH EMBEDDED EVENT MESSAGE BOXES,” the entire disclosures of which are incorporated by reference herein. TECHNICAL FIELD

[0003] Embodiments of the present disclosure describe methods and apparatus generally related to dynamic adaptive streaming over hypertext transfer protocol. BACKGROUND

[0004] The background description provided herein is intended to present generally the context of the present disclosure. The work of the inventors, to the extent the inventors are aware of it, in the field of the present disclosure, and the background art described herein are not necessarily prior art to the present disclosure, and are not admitted to be prior art by virtue of their inclusion in this section of the specification.

[0005] Dynamic adaptive streaming over hypertext transfer protocol (HTTP) (DASH) provides a standard for streaming multimedia content over IP networks. The DASH standard allows event message boxes to be carried in media segments. SUMMARY

[0006] Aspects of the present disclosure provide an apparatus for receiving media data. An apparatus includes processing circuitry that receives a timing metadata track of a plurality of segments including a plurality of metadata samples, each of the plurality of segments including only one of the plurality of metadata samples, each of the plurality of metadata samples including one or more event message boxes. The processing circuitry determines a start time and an active duration of each event message box. The processing circuitry processes event information included in the event message boxes based on the start time and the active duration of the event message boxes. A fragmentation and fragment reordering process is performed on the timing metadata track.

[0007] In an embodiment, a time stamp of each event message box is equal to a time stamp of the timing metadata track.

[0008] In an embodiment, a presentation time and a duration of each of the plurality of metadata samples is equal to an earliest presentation time and a duration of one of the plurality of segments including the respective metadata sample.

[0009] In an embodiment, the presentation time of each of the plurality of metadata samples is an anchor point of one of a presentation time value and a presentation time delta value of an event message box included in the respective metadata sample.

[0010] In an embodiment, a sum of a start time and an active duration of each event message box is limited by an end presentation time of the timed metadata track.

[0011] In an embodiment, each of the plurality of segments is one of a Common Media Application Format (CMAF) fragment and a Dynamic Adaptive Streaming over Hypertext Transfer Protocol (DASH) segment.

[0012] In an embodiment, each event message box included in one of the plurality of metadata samples includes a different scheme identifier.

[0013] In an embodiment, the fragmentation and segment reorganization process is based on an International Organization for Standardization (ISO) / International Electrotechnical Commission (IEC) Base Media File Format (ISOBMFF) fragmentation and segment reorganization process.

[0014] Aspects of the disclosure provide methods of receiving media data. In one method, a timed metadata track including a plurality of segments of a plurality of metadata samples is received, each of the plurality of segments including only one of the plurality of metadata samples, each of the plurality of metadata samples including one or more event message boxes. A start time and an active duration of each event message box is determined. Event information included in the event message boxes is processed based on the start time and the active duration of the event message box. A fragmentation and segment reorganization process is performed on the timed metadata track.

[0015] Aspects of the disclosure also provide non-transitory computer-readable storage media storing instructions that, when executed by a computer for receiving media data, cause the computer to perform any one or combination of the methods of receiving media data. BRIEF DESCRIPTION OF DRAWINGS

[0016] Other features, properties, and various advantages of the disclosed subject matter will become further apparent during the following detailed description, the appended claims and the drawings, in which:

[0017] Figure 1 An exemplary Dynamic Adaptive Streaming over Hypertext Transfer Protocol (DASH) system according to embodiments of the disclosure is shown.

[0018] Figure 2 An exemplary DASH client architecture according to embodiments of the disclosure is shown.

[0019] Figure 3An example in-band event timing model is shown in accordance with embodiments of the disclosure.

[0020] Figure 4 An example of media tracks and timing metadata tracks that can be included in a content stream is shown in accordance with embodiments of the disclosure.

[0021] Figure 5 A flow diagram showing an overview process example is shown in accordance with some embodiments.

[0022] Figure 6 is a schematic diagram of a computer system in accordance with embodiments. DETAILED DESCRIPTION

[0023] I. Dynamic adaptive streaming over hypertext transfer protocol (DASH) and media presentation description (MPD)

[0024] Dynamic adaptive streaming over hypertext transfer protocol (DASH) is an adaptive bitrate streaming technique that enables streaming of media content using hypertext transfer protocol (HTTP) infrastructure, such as web servers, content delivery networks (CDNs), various proxies and caches, etc. DASH supports both on-demand and live streaming from a DASH server to a DASH client and allows the DASH client to control the streaming session so that the DASH server does not need to handle additional stream adaptation management load in large scale deployments. DASH also allows the DASH client to select streaming from various DASH servers, thereby further enabling load balancing of the network for the benefit of the DASH client. DASH provides dynamic switching between different media tracks, e.g., by changing the bitrate to adapt to network conditions.

[0025] In DASH, a media presentation description (MPD) file provides information to a DASH client to adaptively stream media content by downloading media segments from a DASH server. The MPD file can be fragmented and partially delivered to reduce session startup delay. The MPD file can also be updated during a streaming session. In some examples, the MPD file supports representation of content accessibility features, ratings, and camera views. DASH also supports delivery of multi-view and scalable encoded content.

[0026] An MPD file can contain a sequence of one or more periods. Each of the one or more periods can be defined by a period element in the MPD file. An MPD file can include an availableStartTime attribute of the MPD and a start attribute of each period. For a media presentation with dynamic type (e.g., for a live service), the sum of the start attribute of a period and the availableStartTime attribute of the MPD and the duration of the media segments can indicate the availability time of the period, in particular for the first media segment of each presentation in the corresponding period, in Coordinated Universal Time (UTC) format. For a media presentation with static type (e.g., for an on-demand service), the start attribute of the first period can be 0. For any other period, the start attribute can specify a time offset between the start time of the corresponding period and the start time of the first period. Each period can extend to the start of the next period, or to the end of the media presentation if it is the last period. The period start time can be precise and reflect the actual timing resulting from playing the media of all previous periods.

[0027] Each period can contain one or more adaptation sets, and each adaptation set can contain one or more presentations of the same media content. A presentation can be one of a plurality of alternative encoded versions of audio or video data. The plurality of presentations can differ by encoding type, e.g., by bitrate, resolution, and / or codec of the video data and by bitrate and / or codec of the audio data. The term presentation can be used to refer to a piece of encoded audio or video data corresponding to a particular period of multimedia content and encoded in a particular way.

[0028] An adaptation set of a particular period can be assigned to a group indicated by a group attribute in the MPD file. Adaptation sets in the same group are generally considered to be mutually substitutable. For example, each adaptation set of video data of a particular period can be assigned to the same group, such that any one adaptation set can be selected for decoding to display the video data of the multimedia content for the corresponding period. In some examples, the media content within a period can be presented by one adaptation set from group 0, if present, or by a combination of at most one adaptation set from each non-zero group. The timing data for each presentation of a period can be expressed relative to the start time of the period.

[0029] A presentation can include one or more segments. Each presentation can include an initialization segment, or each segment of a presentation can be self-initializing. When present, the initialization segment can contain initialization information for accessing the presentation. In some cases, the initialization segment does not contain media data. A segment can be uniquely referenced by an identifier, such as a uniform resource locator (URL), uniform resource name (URN), or uniform resource identifier (URI). An MPD file can provide an identifier for each segment. In some examples, the MPD file can also provide a byte range in the form of a range attribute, which can correspond to the data of a segment within a file accessible by a URL, URN, or URI.

[0030] Each presentation can also include one or more media components, where each media component can correspond to an encoded version of a single media type, such as audio, video, or timed text (e.g., for closed captioning). The media components can be time continuous across the boundaries of consecutive media segments within a presentation.

[0031] In some embodiments, a DASH client can access and download an MPD file from a DASH server. That is, the DASH client can retrieve an MPD file for initiating a live session. Based on the MPD file, for each selected presentation, the DASH client can make a number of decisions, including determining the latest segment available on the server, determining the segment availability start times for the next segment and possibly future segments, determining when to start playing the segment and from which timeline in the segment to start playing, and determining when to fetch / acquire a new MPD file. Once the service is played, the client can track the drift between the live service and its own playback, which needs to be detected and compensated.

[0032] II. Event Message Box

[0033] The International Organization for Standardization (ISO) / International Electrotechnical Commission (IEC) 23009-1 DASH standard introduces an event message box for carrying events with media segments. The ISO / IEC 23000-19 Common Media Application Format (CMAF) allows including an event message box at the beginning of each CMAF chunk. The transport of event messages as part of a CMAF sparse metadata track has been discussed. However, a question arose as to whether such a track satisfies the fragmentation / refragmentation requirements of an ISO / IEC 14496-12 ISO / IEC Base Media File Format (ISOBMFF) track.

[0034] The present disclosure includes embodiments for signaling and processing of event information, such as event information included in a timed metadata track. Event information can correspond to media timing events associated with a time point or period in a media presentation, such as a continuous audio and / or video presentation. For example, event information can be used for dynamic content replacement, advertisement insertion, presentation of supplemental content with audio and / or video, changes to a web page, and execution of application code triggered at a specific point on the media timeline of a media presentation, such as an audio and / or video media stream. Furthermore, event information can be provided by different methods.

[0035] Media timing events can be used to carry information that is synchronized with a media stream. For example, event information can include metadata (or timed metadata) that describes the content of a media presentation, such as a program or chapter title, or geographic location information. Furthermore, event information can include control messages for a media player that are associated with a specific time during playback of a media presentation, such as an advertisement insertion cue.

[0036] Embodiments of the present disclosure can be implemented in MPEG-DASH. A timed metadata track can include embedded event message boxes (emsg) that carry event information. The timed metadata track can be used to carry information similar to the MPD events and in-band events described above. Event information can include metadata that is time-synchronized with information provided in other tracks. Due to this synchronization, metadata in a timed metadata track can be provided irregularly or discontinuously, and such a timed metadata track can be referred to as a sparse timed metadata track.

[0037] When a timed metadata track carries event message boxes as part of metadata samples, each event message box can include its own timing that uses the earliest presentation time of the CMAF box or track (including the event message box) as an anchor point. However, the ISO / IEC Base Media File Format (ISOBMFF) requires that the timing of data be preserved if the ISOBMFF track goes through any fragmentation and fragment reorganization processes. Due to the loss of box or track boundaries in the fragmentation and fragment reorganization processes, a sparse timed metadata track carrying embedded events can not be able to support this requirement.

[0038] The present disclosure proposes methods for a sparse timed metadata track carrying embedded events to support any arbitrary fragmentation and fragment reorganization processes, thereby becoming a valid ISOBMFF track.

[0039] Figure 1An exemplary DASH system (100) is shown in accordance with embodiments of the present disclosure. In the DASH system (100), an MPD file is sent from a DASH server (101) (e.g., a content server) to a DASH client (102). The DASH client (102) can receive media segments from the DASH server (101) based on the MPD file. The DASH client (102) can send a request to the DASH server (101) to update the MPD file. The DASH server (101) can provide a content stream including a main content (e.g., a main program) and one or more timed metadata tracks.

[0040] Figure 2 An exemplary DASH client architecture is shown in accordance with embodiments of the present disclosure. A DASH client (or DASH player) can be configured to communicate with an application (212) and handle various types of events including (i) MPD events, (ii) in-band events, and (iii) timed metadata events.

[0041] A manifest parser (210) can parse a manifest (e.g., an MPD). For example, the manifest can be provided by a DASH server (101). The manifest parser (210) can extract event information about MPD events, in-band events, and timed metadata events embedded in timed metadata tracks. The extracted event information can be provided to DASH logic (211) (e.g., DASH player control, selection, and heuristics logic). The DASH logic (211) can inform the application (212) of event schemes signaled in the manifest based on the event information.

[0042] The event information can include event scheme information for distinguishing different event streams. The application (212) can use the event scheme information to subscribe to event schemes of interest. The application (212) can also indicate a desired scheduling mode for each subscribed scheme through one or more subscription application program interfaces (APIs). For example, the application (212) can send a subscription request to the DASH client identifying one or more event schemes of interest and any desired corresponding scheduling modes.

[0043] If the application (212) subscribes to one or more event schemes delivered as part of one or more timed metadata tracks, the in-band event and'moof' parser (203) can stream the one or more timed metadata tracks to the timed metadata track parser (204). For example, the in-band event and'moof' parser (203) parses movie fragment boxes ('moof') and subsequently parses the timed metadata tracks based on control information from the DASH logic (211).

[0044] The timed metadata track parser (204) can extract event messages embedded in the timed metadata tracks. The extracted event messages can be stored in the event and timed metadata buffer (206). The synchronizer / scheduler module (208) (e.g., event and timed metadata synchronizer and scheduler) can schedule (or send) the subscribed events to the application (212).

[0045] The MPD events described in the MPD can be parsed by the manifest parser (210) and stored in the event and timed metadata buffer (206). For example, the manifest parser (210) parses each event stream element of the MPD and parses each event described in each event stream element. For each event signaled in the MPD, event information such as presentation time and event duration can be stored in the event and timed metadata buffer (206) associated with the event.

[0046] The in-band event and'moof' parser (203) can parse media segments to extract in-band event messages. Any such identified in-band events and associated presentation times and durations can be stored in the event and timed metadata buffer (206).

[0047] Accordingly, the event and timed metadata buffer (206) can store MPD events, in-band events, and / or timed metadata events therein. The event and timed metadata buffer (206) can be, for example, a first-in-first-out (FIFO) buffer. The event and timed metadata buffer (206) can be managed in correspondence with the media buffer (207). For example, as long as a media segment exists in the media buffer (207), any events or timed metadata corresponding to the media segment can be stored in the event and timed metadata buffer (206).

[0048] The DASH access API (202) can manage the fetching and receiving of content streams (or data streams) including media content and various metadata through the HTTP protocol stack (201). The DASH access API (202) can separate the received content streams into different data streams. The data stream provided to the in-band event and'moof' parser (203) can include media segments, one or more timed metadata tracks, and in-band event signaling included in the media segments. In an embodiment, the data stream provided to the manifest parser (210) can include the MPD.

[0049] The DASH access API (202) can forward the manifest to a manifest parser (210). In addition to describing events, the manifest can also provide information about media segments to DASH logic (211), which can communicate with an application (212) as well as an in-band event and'moof' parser (203). The application (212) can be associated with media content handled by the DASH client. Control / synchronization signals exchanged between the application (212), DASH logic (211), manifest parser (210), and DASH access API (202) can control the fetching of media segments from the HTTP protocol stack (201) based on the information about media segments provided in the manifest.

[0050] The in-band event and'moof' parser (203) can parse the media data stream into media segments, including media content, timed metadata in timed metadata tracks, and any signaled in-band events in the media segments. The media segments including media content can be parsed by a file format parser (205) and stored in a media buffer (207).

[0051] Events stored in the event and timed metadata buffer (206) can allow a synchronizer / scheduler (208) to communicate available events (or events of interest) to the application (212) through an event / metadata API. The application (212) can be configured to handle the available events (e.g., MPD events, in-band events, or timed metadata events) and subscribe to specific events or timed metadata by notifying the synchronizer / scheduler (208). Any events stored in the event and timed metadata buffer (206) that are not relevant to the application (212) but are relevant to the DASH client itself can be forwarded by the synchronizer / scheduler (208) to the DASH logic (211) for further processing.

[0052] In response to the application (212) subscribing to specific events, the synchronizer / scheduler (208) can communicate event instances (or timed metadata samples) corresponding to the event schemes to which the application (212) has subscribed to the application (212). The event instances can be communicated according to a scheduling mode indicated by the subscription request (e.g., for specific event schemes) or a default scheduling mode. For example, in an on-receive scheduling mode, the event instances can be sent to the application (212) as they are received in the event and timed metadata buffer (206). On the other hand, in an on-start scheduling mode, the event instances can be sent to the application (212) at their associated presentation times (e.g., synchronized with timing signals from a media decoder (209)).

[0053] It should be noted that in the DASH client architecture, the thick data flow line represents the media data stream, the thin data flow line represents the event and timed metadata data stream, and the dashed data flow line represents the control and synchronization. Furthermore, the same processing model can be used for CMAF events.

[0054] Figure 3 An exemplary timing model for event message box is shown. In the timing model, each event message box can be described by three timing parameters on the media timeline: (i) event arrival time (AT), which is the earliest presentation time of the segment that includes the event message box; (ii) event presentation / start time (ST), which is the time in the media (MPD) timeline when the event becomes active; and (iii) event duration (DU), during which the event is active.

[0055] The event message box can be inserted at the beginning of a media segment. Therefore, the earliest presentation time of the media segment carrying the event message box can be considered as the location of the event message box on the media timeline. The DASH client can fetch and parse the media segment before or at the AT of the event message box.

[0056] The ST of the event message box can be offset from the location of the media segment (e.g., DASH segment or CMAF chunk) in the track that carries the event message box. The anchor point of the ST can be different depending on the version of the event message box. For version 0 of the event message box, the anchor point can be the earliest presentation time of the CMAF fragment / DASH segment that carries the event message box. For version 1 of the event message box, the anchor point can be the earliest presentation time of the CMAF track / DASH period that carries the event message box.

[0057] According to ISO / IEC 23000-19, the timescale of the event message box can be equal to the timescale in the MediaHeaderBox of the CMAF track. The event message box can be inserted at the beginning of a CMAF chunk, CMAF fragment, or CMAF segment. The anchor point of the version 0 event message box in a CMAF fragment is the earliest presentation time of the fragment. Furthermore, according to ISO / IEC 23009-1, the event message box can be placed before the first ‘moof’ box of a segment, or can be placed between any media data box (‘mdat’) and ‘moof’ box. In the latter case, an equivalent ‘emsg’ with the same id value should appear before the first ‘moof’ box of any segment.

[0058] III. Fragmentation / Fragment Reordering Process

[0059] The fragmentation / fragment reordering process of ISOBMFF can perform fragmentation and fragment reordering on an ISOBMFF track with fragments of any size. In either process, the resulting non-fragmented or fragmented track should be a valid ISOBMFF track. In this case, all event message boxes included in the new track must maintain the correct timing.

[0060] According to aspects of the disclosure, the fragmentation / fragment reordering process can be performed on a metadata track in units of metadata samples. Thus, during the fragmentation / fragment reordering process, the correct timing of a single metadata sample can be maintained. However, in the case where a segment includes multiple metadata samples, the correct timing of the segment can not be maintained during the fragmentation / fragment reordering process. Furthermore, if the segment includes a version 0 event message box and the correct timing of the segment is not maintained during the fragmentation / fragment reordering process, the correct timing of the event message box can not be maintained since the anchor point of the ST of the version 0 event message box is the earliest presentation time of the segment.

[0061] The survivability of CMAF fragments / DASH segments under any arbitrary fragmentation and fragment reordering can be analyzed as follows.

[0062] If a DASH segment / CMAF fragment includes a version 0 event message box, the fragmentation / fragment reordering process can fail since the anchor point of the event message box is the earliest presentation time of the DASH segment / CMAF fragment and can be lost during the fragmentation / fragment reordering process of the DASH / CMAF media track.

[0063] If a DASH segment / CMAF fragment includes a version 1 event message box, the fragmentation / fragment reordering process can pass since the anchor point of the event message box is the earliest presentation time of the DASH / CMAF media track and is preserved during the fragmentation / fragment reordering process of the DASH / CMAF media track.

[0064] Table 1 summarizes the status of DASH and CMAF tracks for the fragmentation / fragment reordering process.

[0065] Table 1

[0066] Track Fragmentation Fragment reorganization DASH w / emsg V0 X X DASH w / emsg V1 OK OK CMAF w / emsg V0 X X CMAF w / emsg V1 OK OK

[0067] Figure 4 An example of media tracks and timing metadata tracks that can be included in a content stream according to embodiments of the disclosure is shown. The metadata tracks can include event message boxes. Event message boxes can be used to signal general events related to media presentation times. In some embodiments, if a DASH client detects an event message box with a scheme that is not defined in the MPD, the DASH client ignores the event message box.

[0068] An event message box can include message scheme identification information (e.g., scheme id uri), optional values of events contained in the event message box, timing information, and event data. The timing information can indicate a time stamp of other time information (e.g., in units of ticks per second), such as a media presentation time delta of an event’s media presentation time relative to a reference presentation time (e.g., start of a segment or metadata sample), an event’s media presentation time, and an event duration (e.g., in media presentation time).

[0069] A metadata track can carry event message boxes as part of the metadata samples included in the metadata track. Further, a metadata sample can include one or more event message boxes. Each event message box can belong to one event scheme, which is defined by a scheme URI id and optional values of the corresponding event message box. Since event instances from multiple schemes can be included in one or more metadata samples, it is necessary to identify event schemes in the DASH manifest so that a DASH client can discover these schemes.

[0070] DASH includes two elements that can be used to describe event schemes in an MPD. The two elements are an event stream element (e.g., EventStream) for MPD events and an inband event stream element (e.g., InbandEventStream) for inband events. The two event schemes can use the same construction.

[0071] IV. Single-Sample Sparse Timing Metadata

[0072] According to aspects of the present disclosure, a metadata segment / tile including an embedded event message box can be a single-sample DASH segment / CMAF tile or a multi-sample DASH segment / CMAF tile. A single-sample DASH segment / CMAF tile can include only one metadata sample, and the duration of the metadata sample is equal to the duration of the DASH segment / CMAF tile. A multi-sample DASH segment / CMAF tile can include multiple metadata samples.

[0073] If a single-sample DASH segment / CMAF tile includes an event message box of version 0, the segmenting / tile-izing process can pass through. Since the earliest presentation time of the segment / tile is the same as the presentation time of the only one metadata sample included in the segment / tile, the timing of the event message box can be preserved during the segmenting / tile-izing process if the anchor point of the event message box is considered to be the presentation time of the metadata sample that includes the event message box.

[0074] If single-sample DASH segments / CMAF fragments include event message boxes of version 1, then the fragmentation / fragment re-arrangement process can pass since the earliest presentation time of the track is the anchor point of the event message boxes.

[0075] If multi-sample DASH segments / CMAF fragments include event message boxes of version 0, then the fragmentation / fragment re-arrangement process can fail since the anchor point of the event message boxes is the earliest presentation time of the fragments / fragments and can be lost during the fragmentation / fragment re-arrangement process.

[0076] If multi-sample DASH segments / CMAF fragments include event message boxes of version 1, then the fragmentation / fragment re-arrangement process can pass since the earliest presentation time of the track is the anchor point of the event message boxes.

[0077] Table 2 summarizes the status of single-sample and multi-sample metadata segments / fragments with embedded event message boxes in the fragmentation / fragment re-arrangement process.

[0078] Table 2

[0079]

[0080] According to aspects of the present disclosure, the following constraints can be applied on CMAF sparse metadata tracks to meet the ISOBMFF fragmentation / fragment re-arrangement process: (i) each CMAF fragment / DASH segment includes only one metadata sample (or is limited to one metadata sample), and the duration of the metadata sample is the duration of the fragment / segment; (ii) the earliest presentation time of the CMAF fragment / DASH segment is the presentation time of the metadata sample; (iii) in non-fragmented tracks, each event message box of version 0 can use the presentation time of the metadata sample (including the respective event message box) as the anchor point of the presentation time delta parameter (e.g., presentation time delta) of the respective event message box; (iv) in non-fragmented tracks, each event message box of version 1 can use the earliest presentation time of the track as the anchor point of the presentation time parameter (e.g., presentation time) of the respective event message box; (v) in all cases (e.g., DASH or CMAF, fragmented or non-fragmented), the timestamp of each event message box (e.g., version 0 or version 1) can be equal to the timestamp of the track; and (vi) the end time of the event message box (e.g., event start time plus event duration) does not exceed the end (or last) presentation time of the track, even if the value of the event duration indicates that the end time of the event message box exceeds the end presentation time of the track.

[0081] Because the presentation time of the metadata samples carrying the event message box does not change during any fragmentation / defragmentation process, constraints (i) and (ii) can allow fragmentation / defragmentation of the track without changing the timing of the event message box of version 0.

[0082] Constraints (iii) and (iv) are constraints on non-fragmented tracks, allowing to maintain the relationship between fragmented / segmented single-sample sparse metadata tracks and non-fragmented tracks.

[0083] Constraint (v) can ensure that the timing of the events is aligned with the track sample timing, so that there is no fractional drift when integer algorithms are used by the file format parser (205) and the media decoder (209).

[0084] Constraint (vi) can limit the event duration to the maximum track duration, so that operations in the file format parser (205) can be simplified, especially because the application events must be passed to the application (212) that does not necessarily have a track duration. Thus, if the duration of each event message box is set so that the event message box ends before the track or at the end of the track, the file format parser (205) does not need to truncate the event duration if the event duration exceeds the track duration before passing the event to the application (212).

[0085] Some advantages of single-sample sparse timing metadata tracks can be listed as follows. Because each fragment / segment only includes one (synchronized) metadata sample, the structure of single-sample sparse timing metadata tracks is simple. Each metadata sample can include one or more event message boxes with various schemes / sub-schemes. If the fragmentation / defragmentation process maintains the single-sample fragment / segment constraint, fragmented tracks can pass through the fragmentation / defragmentation process. The constraint on the anchor point of the event message box in the non-fragmented track is very simple and easy to maintain. The properties of the event message box can be maintained so that it is delivered using a separate track from the media track, so that it is delivered independent of any particular media track.

[0086] According to aspects of the present disclosure, a sparse timing metadata track can be generated, where each segment / fragment includes only one metadata sample. The metadata sample can include one or more event message boxes. Each event message box can include a different scheme identifier and a different value of a related sub-scheme identifier carrying a message data payload. The time stamp of the event message box is equal to the time stamp of the track. The presentation time and the duration of the metadata sample are equal to the earliest presentation time and the duration of the segment / fragment including the metadata sample. In case of fragment re-arrangement, the anchor point of the presentation time (e.g., presentation time) and / or the presentation time offset (e.g., presentation time delta) of the event message box is the presentation time of the metadata sample including the event message box.

[0087] V. Flowchart

[0088] Figure 5 A flowchart of an overview process (500) according to embodiments of the present disclosure is shown. In various embodiments, the process (500) is performed by processing circuitry, such as processing circuitry in a DASH client (102). In some embodiments, the process (500) is implemented in software instructions, and thus the processing circuitry performs the process (500) when the processing circuitry executes the software instructions. The process (500) starts at (S510), where the process (500) receives a timing metadata track including a plurality of segments of a plurality of metadata samples. Each of the plurality of segments can include only one of the plurality of metadata samples. Each of the plurality of metadata samples includes one or more event message boxes. Then, the process (500) proceeds to step (S520).

[0089] At step (S520), the process (500) determines a start time and an active duration of each event message box. Then, the process (500) proceeds to step (S530).

[0090] At step (S530), the process (500) processes event information included in the event message box based on the start time and the active duration of the event message box. Then, the process (500) ends.

[0091] In an embodiment, the time stamp of each event message box is equal to the time stamp of the timing metadata track.

[0092] In an embodiment, the presentation time and the duration of each of the plurality of metadata samples are equal to the earliest presentation time and the duration of one of the plurality of segments including the respective metadata sample, and a fragmentation and fragment re-arrangement process is performed on the timing metadata track.

[0093] In embodiments, the presentation time of each of the plurality of metadata samples is an anchor point that includes one of a presentation time value and a presentation time delta value of an event message box included in the respective metadata sample.

[0094] In embodiments, a sum of a start time and an active duration of each event message box is limited by an end presentation time of the timed metadata track.

[0095] In embodiments, each of the plurality of segments is one of a CMAF fragment and a DASH segment.

[0096] In embodiments, each event message box included in one of the plurality of metadata samples includes a different scheme identifier.

[0097] In embodiments, the fragmentation and segment reorganization process is based on the ISO / IEC ISOBMFF fragmentation and segment reorganization process.

[0098] VI. Computer System

[0099] The techniques described above, can be implemented as computer software using computer-readable instructions and physically stored in one or more computer-readable media. For example, Figure 6 A computer system (600) is shown which is suitable for implementing certain embodiments of the disclosed subject matter.

[0100] The computer software can be coded using any suitable machine code or computer language, that can be subject to assembly, compilation, linking, or like mechanisms to create code comprising instructions that can be executed directly, or through interpretation, micro-code execution, or the like, by one or more computer central processing units (CPUs), Graphics Processing Units (GPUs), and the like.

[0101] The instructions can be executed, in various embodiments, on a variety of different types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, internet appliances, and the like.

[0102] Figure 6 The components shown in the computer system (600) are exemplary and not limiting, and are not intended to suggest any limitation as to the scope of use or functionality of the computer software implementing embodiments of this disclosure. Neither should the configuration of components be interpreted as having any dependency or requirement relating to any one or combination of components illustrated in the exemplary embodiment of a computer system (600).

[0103] Computer system (600) can include certain human interface input devices. Such a human interface input device can be responsive to user input. The couple of human interface input devices can take many forms, including but not limited to one or more of a keyboard (601), mouse (602), touchpad (603), touchscreen (610), data glove (not shown), joystick (605), microphone (606), scanner (607), camera (608), and so on. A monitor (609) or other display device can also be included as a human interface output device to provide visual output to a user.

[0104] The human interface input devices can include one or more of: a keyboard (601), mouse (602), trackpad (603), touchscreen (610), data glove (not shown), joystick (605), microphone (606), scanner (607), and camera (608) to provide

[0105] Computer system (600) can also include certain human interface output devices. Such human interface output devices can be stimulating one or more of the human senses, including but not limited to visual sense, tactile sense, auditory sense, olfactory sense, and gustatory sense. The human interface output devices can include tactile output (e.g., to output tactile effects such as vibrations, to

[0106] Computer system (600) might also include certain human accessible storage and / or storage interface devices. An interface for such devices can be implemented using, for example, an industry

[0107] Those skilled in the art will further appreciate that the term "computer-readable medium" as used herein does not include transitory mainlines, carriers, or other transitory signals.

[0108] Computer system (600) can also include a network interface (654) to one or more communication networks (655). These networks can include wireless, wireline, optical, etc. The one or more networks (655) can be local, wide-area, metropolitan, etc. networks, and can also be telecommunications, data networks, etc. Examples of such networks include local area networks, telephone networks, plastic networks, etc. The one or more networks (655) can be the same or different networks and can include a global computer network, such as the Internet.

[0109] The aforementioned human interface devices, human-accessible storage devices, and network interfaces can be connected to the core (640) of the computer system (600) using various types of communication media, such as communication lines, cables, pins, etc.

[0110] The core (640) can include one or more Central Processing Units (CPU) (641), Graphics Processing Units (GPU) (642), specialized programmable processing units in the form of Field Programmable Gate Arrays (FPGA) (643), hardware accelerators for certain tasks (644), and so forth. These devices, along with Read-only memory (ROM) (645), Random-access memory (RAM) (646), internal mass storage such as internal non-user accessible hard drives, solid state drives, and so forth (647), can be connected through a system bus (648). In some computer systems, the system bus (648) can be accessible in the form of one or more physical plugs to enable extensions of the system bus (648) through additional CPUs, GPU, and so forth. The peripheral devices can be attached either directly to the core’s system bus (648), or through a peripheral bus (649). Architectures for a peripheral bus include Peripheral Component Interconnect (PCI) and variants, Universal Serial Bus (USB), and so forth.

[0111] CPUs (641), GPUs (642), FPGAs (643), and accelerators (644) can execute certain instructions that, in combination, can make up the aforementioned computer code. That computer code can be stored in ROM (645) or RAM (646). Transitional data for the CPU (641) or GPU (642) can be stored in RAM (646), while permanent data can be stored in internal mass storage (647). Fast storage and retrieval can be achieved using cache memory, which can be closely associated with one or more CPU (641), GPU (642), mass storage (647), ROM (645), RAM (646), and so forth.

[0112] The computer software can be implemented as computer code that is stored on a computer readable medium, which can be any medium that is readable by a machine. The computer software can be implemented in a variety of programming languages, including but not limited to assembly language, machine language, high-level procedural, high-level object-oriented, and so forth. The computer software can be implemented using a variety of protocols and formats, including but not limited to C, C++, C#, Java, Pascal, Fortran, COBOL, Perl, Objective-C, Ruby, Eiffel, Haskell, ML, Lisp, F#, and so forth. The computer software can be implemented using different programming techniques, including but not limited to object-oriented programming, procedural programming, and so forth.

[0113] By way of example and not limitation, a computer system having an architecture (600), particularly a core (640), can provide functionality as a processor (including a CPU, GPU, FPGA, accelerator, etc.) to execute software contained in one or more tangible computer-readable media. Such computer-readable media can be media associated with the aforementioned user-accessible mass storage, as well as specific memory of the core (640) that is non-volatile, such as internal mass storage (647) or ROM (645). Software implementing various embodiments of this disclosure can be stored in such a device and executed by the core (640). Depending on specific needs, the computer-readable medium may include one or more storage devices or chips. The software can cause the core (640), particularly the processor therein (including a CPU, GPU, FPGA, etc.), to execute specific processes or specific portions of specific processes described herein, including defining data structures stored in RAM (646) and modifying such data structures according to software-defined processes. Alternatively or as an alternative, the computer system may provide logic hardwired or otherwise incorporated into circuitry (e.g., an accelerator (644)) that may replace or operate with the software to perform the specific process or a specific portion of the specific process described herein. References to software may include logic, and vice versa, where appropriate. References to computer-readable media may include, where appropriate, circuitry storing the execution of software (such as an integrated circuit (IC)), circuitry containing execution logic, or both. This disclosure includes any suitable combination of hardware and software.

[0114] While this disclosure has described several exemplary embodiments, various modifications, arrangements, and equivalent substitutions of the embodiments are within the scope of this disclosure. Therefore, it should be understood that those skilled in the art can design various systems and methods that, while not explicitly shown or described herein, embody the principles of this disclosure and are thus within its spirit and scope.

Claims

1. A method of receiving media data, characterized by, The method comprises: receiving a timed metadata track comprising a plurality of segments of a plurality of metadata samples, each of the plurality of segments comprising only one of the plurality of metadata samples, each of the plurality of metadata samples comprising one or more event message boxes; determining a start time and an active duration of each event message box, the sum of the start time and the active duration of each event message box being limited by an end presentation time of the timed metadata track, wherein, in a non-fragmented track, each event message box of version 0 uses a presentation time of the metadata sample comprising the respective event message box as an anchor point for a presentation time offset parameter of the respective event message box, in a fragmented track, each event message box of version 1 uses an earliest presentation time of the non-fragmented track as an anchor point for a presentation time parameter of the respective event message box; and processing event information comprised in the event message boxes based on the start time and the active duration of the event message boxes, wherein a fragmentation and fragment reordering process is performed on the timed metadata track.

2. The method of claim 1, wherein, a timescale of each event message box is equal to a timescale of the timed metadata track.

3. The method of claim 1, wherein, further comprising: receiving a start presentation time of each of the plurality of metadata samples, the start presentation time being equal to a start presentation time of a different segment of the plurality of segments comprising the respective metadata sample.

4. The method of claim 3, wherein, a duration of each of the plurality of metadata samples is equal to a duration of a different segment of the plurality of segments comprising the respective metadata sample.

5. The method of claim 3, wherein, a start presentation time and a duration of each of the plurality of metadata samples is equal to an earliest presentation time and a duration of one of the plurality of segments comprising the respective metadata sample.

6. The method according to any one of claims 1 to 5, characterized in that, each of the plurality of segments is a Common Media Application Format, CMAF, segment or a Dynamic Adaptive Streaming over Hypertext Transfer Protocol, DASH, segment.

7. The method according to any one of claims 1 to 5, characterized in that, each event message box comprised in one of the plurality of metadata samples comprises a different scheme identifier.

8. The method according to any one of claims 1 to 5, characterized in that, the fragmentation and fragment reordering process is based on an International Organization for Standardization, ISO / International Electrotechnical Commission, IEC, Base Media File Format, ISOBMFF, fragmentation and fragment reordering process.

9. An apparatus for receiving media data, the apparatus comprising: The apparatus comprises processing circuitry configured to perform the method of any one of claims 1-8.

10. An apparatus for receiving media data, the apparatus comprising: The apparatus comprises: a receiving module configured to receive a timed metadata track comprising a plurality of segments of a plurality of metadata samples, each of the plurality of segments comprising only one of the plurality of metadata samples, each of the plurality of metadata samples comprising one or more event message boxes; a determining module configured to determine a start time and an active duration of each event message box, the sum of the start time and the active duration of each event message box being limited by an end presentation time of the timed metadata track, wherein, in a non-fragmented track, each event message box of version 0 uses a presentation time of the metadata sample comprising the respective event message box as an anchor point for a presentation time offset parameter of the respective event message box, in a fragmented track, each event message box of version 1 uses an earliest presentation time of the non-fragmented track as an anchor point for a presentation time parameter of the respective event message box; and a processing module configured to process event information comprised in the event message boxes based on the start time and the active duration of the event message boxes, wherein a fragmentation and fragment reordering process is performed on the timed metadata track. determining module configured to determine a start time and an active duration of each event message box, a sum of the start time and the active duration of each event message box being limited by an end presentation time of the timed metadata track, wherein in a non-fragmented track, each event message box of version 0 uses a presentation time of a metadata sample of the respective event message box as an anchor point for a presentation time offset parameter of the respective event message box, and in a fragmented track, each event message box of version 1 uses an earliest presentation time of the non-fragmented track as an anchor point for a presentation time parameter of the respective event message box; and processing module configured to process event information included in the event message box based on the start time and the active duration of the event message box, wherein performing a fragmentation and defragmentation process on the timed metadata track.

11. A non-transitory computer readable storage medium, comprising: instructions for causing a computer for receiving media data to perform the method of any of claims 1-8 when the instructions are executed by the computer.

Images