Method, device, and computer-readable medium for processing alternative media presentation descriptions

Abstract

The present disclosure generally relates to media streaming technologies, and more particularly to a method and apparatus for processing an alternate media presentation description (MPD) using a main media presentation description (MPD) from a content server. The method includes receiving a manifest of the alternate MPD from a content server by a Dynamic Adaptive Streaming over HTTP (DASH) media streaming device, parsing the manifest to extract a set of parameters of the alternate MPD, the set of parameters including at least one of a value of a DASH event stream, a presentation time, or a duration, switching from the main media presentation to the alternate media presentation based on the presentation time, and playing the main media presentation at a return point according to the value of the DASH event stream in response to an end of the alternate media presentation.

Description

Technical Field

[0001] Cross - Reference to Related Applications This application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 332,590, filed on April 19, 2022, which is hereby incorporated by reference in its entirety. This application also claims the benefit of priority to U.S. Non - Provisional Patent Application No. 18 / 135,250, filed on April 17, 2023, which is hereby incorporated by reference in its entirety.

[0002] The present disclosure generally relates to media streaming technologies, and more particularly, to methods and apparatuses for processing alternative media presentation descriptions (MPDs) in adaptive streaming.

Background Art

[0003] The description of the background art provided herein is for the purpose of generally presenting the context of the present disclosure. To the extent that the description in this background art section is concerned, the research of the inventors described herein, as well as aspects of the description that may not be recognized as prior art at the time of the effective filing of this application, are not to be regarded as prior art to the present disclosure, either explicitly or implicitly.

[0004] Dynamic Adaptive Streaming over HTTP (DASH) via the MPEG (Moving Picture Expert Group) HyperText Transfer Protocol provides a standard for streaming multimedia content over an IP network. In the DASH standard, a Media Presentation Description (MPD) is used to provide information for adaptively streaming media content by a DASH client downloading media segments from a DASH server. Streaming of multi-rate content is made possible by the DASH standard. One aspect of the DASH standard includes the conveyance of MPD events and in-band events and client processing models used for handling such events.

[0005] Common Media Application Format (CMAF) is a standard used to package and deliver various forms of HTTP-based media. This standard simplifies the delivery of media to playback devices, for example, by working with HTTP Live Streaming (HLS) and the DASH protocol to package data under a uniform transport container file. It also uses chunked encoding and chunked transfer encoding to reduce latency.

[0006] MPEG DASH can provide several means for streaming multimedia content over an IP network, along with alternative MPD events for signaling media presentations between two timelines. There are various issues / problems regarding how to handle alternative MPD events.

[0007] This disclosure describes various embodiments for handling alternative MPD events that address at least one of the problems / issues and advance the art of media streaming. SUMMARY OF THE INVENTION

Means for Solving the Problem

[0008] The present disclosure generally relates to media streaming technology, and more specifically to methods and apparatuses for processing alternative MPD events in dynamic adaptive streaming. The alternative MPD event may be sent by a media content server to a media streaming client and then processed by the media streaming client.

[0009] According to one aspect, an embodiment of the present disclosure provides a method by a media streaming device for processing an alternative MPD using a main media presentation description (MPD) from a content server. The method includes receiving, by the media streaming device, a manifest for the alternative MPD from the content server; parsing the manifest to extract a set of parameters for the alternative MPD, the set of parameters including at least one of a value of an event stream, a presentation time, or a duration, the presentation time indicating an offset at which an alternative media presentation starts in a timeline of the main media presentation, and the duration indicating a period during which the alternative media presentation is active; switching from the main media presentation to the alternative media presentation based on the presentation time; and resuming the main media presentation at a return point according to the value of the event stream in response to the end of the alternative media presentation. The media streaming device includes a dynamic adaptive streaming over HTTP (DASH) media streaming device, and the event stream includes a DASH event stream.

[0010] According to another aspect, one embodiment of the present disclosure provides a method by a media streaming content server for constructing an alternative MPD using a main media presentation description (MPD) and transmitting the alternative MPD to a media streaming device. The constructed MPD is configured to cause the media streaming device to execute any one of the method embodiments described in the present disclosure. The media streaming device includes a dynamic adaptive streaming over HTTP (DASH) media streaming device.

[0011] Aspects of the present disclosure also provide a media streaming device or apparatus including a circuit configured to execute any one of the above method embodiments.

[0012] Aspects of the present disclosure also provide a non-transitory computer-readable medium storing instructions that, when executed by a media streaming device, are configured to cause the media streaming device to perform any one of the above method embodiments.

[0013] The above and other aspects and their embodiments are described in more detail in the drawings, the specification, and the claims.

[0014] Further features, properties, and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings.

Brief Description of the Drawings

[0015]

Figure 1

Figure 2

Figure 3A

Figure 3B

Figure 4

Figure 5

Mode for Carrying Out the Invention

[0016] Hereinafter, a part of the present invention will be described in detail with reference to the accompanying drawings showing specific examples of embodiments as illustrative examples. However, it should be noted that the present invention may be embodied in various different forms, and thus it is intended that the subject matter covered or claimed is not limited to any of the embodiments described below. It should also be noted that the present invention may be embodied as a method, device, component, or system. Therefore, embodiments of the present invention may take the form of, for example, hardware, software, firmware, or any combination thereof.

[0017] Throughout this specification and the claims, terms may have subtle meanings suggested or implied within the context beyond the explicitly described meaning. The phrase "in one embodiment" or the phrase "in some embodiments" used in the present disclosure does not necessarily refer to the same embodiment, and the phrase "in another embodiment" or "in other embodiments" used in the present disclosure does not necessarily refer to different embodiments. Similarly, the phrases "in one aspect" or "in some aspects" used in this specification do not necessarily refer to the same aspect, and the phrases "in another aspect" or "in other aspects" used in this specification do not necessarily refer to different aspects. For example, the claimed subject matter is intended to include combinations of all or part of the exemplary embodiments / aspects.

[0018] Generally, technical terms may be understood at least in part from their usage in context. For example, terms such as "and," "or," or "and / or" as used herein may include a variety of meanings that may depend at least in part on the context in which such terms are used. Typically, "or" as used to associate a list such as A, B, or C is intended here to mean A, B, and C in an inclusive sense, as well as A, B, or C in an exclusive sense here. In addition, the terms "one or more" or "at least one" as used herein may, at least in part, depend on the context, be used to describe any feature, structure, or property in a singular sense, or be used to describe a combination of features, structures, or properties in a plural sense. Similarly, terms such as "a," "an," or "the" may also, at least in part, depend on the context, be understood to convey a singular usage or a plural usage. In addition, the terms "based on" or "determined by" may not necessarily be intended to convey an exclusive set of factors, and instead may, at least in part, also depend on the context and may allow for the existence of additional factors that are not necessarily explicitly described.

[0019] Streaming via the Hypertext Transfer Protocol (HTTP) Figure 1 shows an exemplary content delivery system 100 configured such that a remote information processing device 120 requests content from one or more centralized or distributed content servers 110 via a communication network 130. In particular, the information processing device 120 can include a dedicated hardware component, a software component operating on general-purpose hardware, or a combination thereof that functions as a content consumption application. The content consumption application can generate one or more requests that specify the requested content and the characteristics of the requested content. Each request is constructed based on a stack of network protocols and can be communicated to the content server 110 via the communication network 130. In response, the content server can generate a bitstream according to the request, package the bitstream using a stack of network protocols, and communicate the bitstream package to the content consumption application.

[0020] In some exemplary embodiments, the content may be requested all at once. In other words, the entire media content can be requested, received, and locally stored by the content consumption application. The locally stored content can be either part of the content consumption application or separated from the content consumption application and can be processed and consumed as needed (e.g., extracted, decoded, played) by, for example, a media player. Such processing is sometimes referred to as a download.

[0021] In some other embodiments, rather than being downloaded for later consumption, the content may be streamed while being consumed. In such embodiments, it may not be necessary for the entire requested content to be stored in the content consumption application. Instead, only a limited amount of content is continuously received from the content server 110 on a rolling basis and managed by an input / output local buffer for content processing and playback. Such embodiments are sometimes referred to as streaming. Some media playback functions such as rewind, fast forward, and seek can involve complex media bitstream control and buffering, but media streaming is generally more general-purpose and better suited for the delivery of content that includes a time-sequenced media that is not repeatedly consumed.

[0022] In the following disclosure, the terms "content" and "media" may be used interchangeably. The requested content can include various information items necessary for its consumption, including but not limited to the content itself and various metadata. The content itself can further include various media components such as different tracks including, but not limited to, video components / tracks, audio components / tracks, subtitles, etc. Metadata for describing the media content or providing additional processing information may be treated as one or more separate tracks. Such content with its metadata can be generated by content server 120 as a bitstream that can be parsed and decoded according to a set of protocols or rules known to content consumption applications. The singular term "content server" is used to represent a single server or multiple servers located centrally or distributed across various geographical locations. Such a content server may be implemented as a dedicated computing machine, or alternatively, as a virtual machine and / or virtually housed in a cloud computing environment. Further, in the following disclosure, the terms "information processing device" (referring to 120 in FIG. 1) and "content consumption application" may be used interchangeably. Alternatively, these terms may sometimes be referred to as "client", "client device / equipment", "playback device / equipment / client", etc. Note that in FIG. 1, only one information processing device 120 is shown, but there can be multiple independent information processing devices. In other words, a set of content servers 110 may be configured to simultaneously and independently provide streaming services to multiple content consumption applications.

[0023] In some exemplary embodiments, the content generated for delivery by content server 110 can be segmented to facilitate their streaming. For example, the time sequence of media content such as a movie can be chopped into time segments, each containing several media frames. Each media segment can be self - contained such that its processing, including, for example, parsing, decoding, and playback, does not require information for other media segments. The media content may be pre - segmented. Thus, the media content can be stored and managed by content server 120 on a per - segment basis. Alternatively, the media segments may be generated in real - time from the adjacent stored media content as required during the streaming process. In some further embodiments, the segmentation of the media may be hierarchical, including multiple levels of segmentation.

[0024] In some specific embodiments for streaming, the decision of which media segment or which part of a media segment to request from content server 110 can be determined in real - time by the content consumption application under the control of user playback commands via the user application interface. In this way, the content server can be configured to respond to the requests, generate or retrieve segments or parts of segments of the content with their metadata according to the requests, and deliver the segments or parts of segments to the content consumption application requesting via network 130.

[0025] In some exemplary embodiments, the same media track of media content can be prepared as different versions. For example, the same movie track may be prepared at different resolutions and / or frame rates. As another example, the same movie track may be prepared at different bitrates. As another example, the same audio movie may be prepared with different audio qualities and / or different numbers of audio channels (e.g., 5-channel sound, or 7-channel sound). Thus, the content consumption application can determine which version of the media track to stream and include such a selection in the requirements for the media content. Such a decision by the content consumption application can be made based on one or more of several exemplary factors including, but not limited to, the playback capabilities of the information processing device 120 (e.g., display resolution, decoding speed, processing power, buffer size, etc.), network bandwidth and throughput, etc. Thus, the streaming session can be adapted among different media consumption applications according to their device capabilities. Such a configured streaming architecture may be referred to as adaptive streaming. The streaming process can be further adaptive within each media consumption application in that different versions of the media track can be selected and requested at different times during the streaming session, for example, according to real-time network conditions (e.g., bandwidth and throughput, and the bitrate supported by the network bandwidth). Such a configured streaming architecture may be further referred to as dynamic adaptive streaming. In particular, a streaming architecture configured to adapt to the bitrate of media content may be referred to as dynamic adaptive bitrate streaming.

[0026] In some exemplary embodiments, requests for a particular version of a segment of media content or a portion of a segment by a content consumption application in dynamic adaptive streaming may be constructed based on a media manifest as the streaming session progresses. The term "manifest" may be used to represent any set of information items that describe media content, including segmentation, versioning, network location, and any other information that may be required by the content consumption application to determine how and what to request at different times during the streaming session. A manifest is generally sometimes referred to as a "Media Presentation Description" (MPD).

[0027] Such a manifest may be prepared on the content server side when the particular media content is created or generated. Such a manifest may be requested by the content consumption application and received from the content server at the start of the streaming session. The content consumption application may further request any updates to the manifest during the streaming session. Such a manifest may be used by the content consumption device as a blueprint for constructing subsequent requests for a particular version of a segment of media content or a portion of a segment during the streaming session.

[0028] In some exemplary embodiments, the media server may be configured to function in a manner similar to a web server from the standpoint of an external application. Accordingly, requests for a media manifest and / or media segments or portions of media segments by a content consumption application may be made, for example, based on the Hypertext Transfer Protocol (HTTP). In this way, the requests may be constructed as URLs and the requested content may be delivered as a response to an HTTP request from the content server.

[0029] Details of how a manifest is specified, content is segmented, composed, versioned, and an HTTP request is constructed may depend on specific adaptive streaming protocols such as Dynamic Adaptive Streaming over HTTP (DASH), HTTP Live Streaming (HLS), Smooth Streaming Transport Protocol (SSTP), etc. The various additional exemplary embodiments below can be described in the context of DASH. However, the underlying principles are applicable to any type of adaptive streaming over HTTP. Further, the underlying principles are applicable to media content request mechanisms based on network protocols other than HTTP.

[0030] Dynamic Adaptive Streaming over HTTP (DASH) One exemplary protocol for implementing adaptive media streaming includes Dynamic Adaptive Streaming over HTTP (DASH). As described above, DASH represents one of the adaptive bitrate streaming implementations that enables streaming of media content using a content delivery network (CDN) based on the Hypertext Transfer Protocol (HTTP) infrastructure, including content servers configured as web servers with various proxies and caches. Such a content server may be referred to as a DASH server. Accordingly, the content consumption application described above can be called a DASH client.

[0031] DASH supports live streaming from a DASH server to a DASH client and enables the DASH client to control the streaming session, so the DASH server does not need to handle the additional load of stream adaptation management in large-scale deployment. As described above, DASH also enables the DASH client to select streaming from various DASH servers, thereby achieving further load distribution of the network for the benefit of the DASH client. DASH further provides dynamic switching between different media versions of a media track, for example, by changing the bitrate to adapt to the network conditions and processing capabilities of the DASH client.

[0032] In DASH, the above media manifest may be specifically called an MPD (however, the term MPD may sometimes be generally used to refer to any type of manifest in an adaptive streaming system other than those based on DASH). For example, the MPD in DASH can be fully or partially downloaded by the DASH client and is constructed as a file that provides information items used by the DASH client to stream media content by selectively and adaptively requesting streaming media segments from the DASH server.

[0033] MPD can be composed in various formats. For example, MPD may be constructed in the form of an Extensible Markup Language (XML) document or file. An MPD file may be requested and delivered to a DASH client. The MPD file may be requested via HTTP, for example, through an HTTP GET request. The MPD file may be delivered completely at the start of a streaming session. Alternatively, the MPD file may be fragmented and delivered partially. Thus, a part of the MPD file may be requested and delivered before the start of streaming, and other parts of the MPD file may be requested and delivered to reduce session start delay (thereby, streaming can start from the previous media segment without having to wait for information items related to subsequent segments of the media). The MPD file can also be updated during a streaming session (for example, using segment information that is required but not yet acquired).

[0034] In some exemplary embodiments, the MPD file describes the segmentation of media content, the composition of segments, and the available versions of segments. MPD may support representations such as content accessibility features, ratings, camera views, metadata, etc. DASH may also support the delivery of multi-view and scalable-coded content.

[0035] In some exemplary embodiments, the MPD file can include a series of descriptions for one or more periods along a media consumption timeline (e.g., the playback time of video content). Each of the one or more periods may be defined, for example, by an "Period" information element tag within the MPD file. The media content may be represented by an MPD file grouped into a plurality of consecutive periods. The MPD file can identify the start time of each period in the playback timeline. The start time may be defined as an absolute start time from the beginning of the media content or as a relative offset from another reference point in the playback timeline.

[0036] In some exemplary embodiments, for each media period, the MPD file can further specify one or more adaptation sets. Different adaptation sets can be specified to incorporate one or more different combinations (or subsets) of media components. For example, video and audio can be in different adaptation sets. Different versions of audio (stereo audio or multi-channel audio) may be in different adaptation sets. Audio in different languages may be in different adaptation sets. In one particular example, the MPD file can specify that each period includes one video adaptation set, a plurality of audio adaptation sets, one for each supported language. Also, an adaptation set can include subtitles and any metadata.

[0037] In some exemplary embodiments, an adaptation set for a particular period may be assigned to a group indicated by a group attribute in the MPD file. Adaptation sets within the same group are generally considered to be alternatives to each other. For example, each adaptation set for a particular period of video data may be assigned to the same group, such that any adaptation set may be selected for the video data of the multimedia content for the corresponding period. The media content within one period may be from either one adaptation set, or a combination of adaptation sets, where each group contributes at most one adaptation set.

[0038] In some exemplary embodiments, each adaptation set may be specified by the MPD file as including one or more representations of the same media component for the corresponding period. For example, a representation may be one of several alternative encoded versions of audio or video data. Representations may differ by encoding type, e.g., bitrate, resolution, and / or codec for video data, and bitrate, and / or codec for audio data. The term representation may be used to refer to a section of encoded media data corresponding to a particular period of multimedia content and encoded in a particular way to achieve a particular range of average bitrates. In some exemplary embodiments, for each representation within an adaptation set, the MPD file may specify representation attributes including, but not limited to, video / audio type, video / audio codec, video frame width in pixel units, video frame height in pixel units, video / audio frame rate, and bandwidth (representing the average encoded bitrate).

[0039] Each representation of an adaptation set can also include one or more media components, depending on the combination of media components included in the adaptation set. Each media component within a representation can correspond to an encoded version of one individual media type, such as audio, video, or timed text (e.g., for closed captions). Media components can be temporally continuous across the boundaries of successive media segments within one representation.

[0040] In some exemplary embodiments, a representation can include one or more segments. Each representation can include an initialization segment, or each segment of a representation can be self-initializing. If present, the initialization segment can include initialization information for accessing the representation. In some cases, the initialization segment does not include media data. Segments that include media data can represent time-segmented content. Segments between different representations may be temporally aligned. For each media segment, the MPD file can include a unique identifier. Such an identifier, when combined with a base URL, base URN, or base uniform resource identifier (URI), can form a unique URL, URN, or URI that represents the network location of the media segment, which is included in the HTTP request for this media segment and can be used by the content server to find the requested segment for delivery.

[0041] For example, the URL for requesting a media segment is in a fixed scheme of "http" or "https" <absolute-uri>It can be defined as, and in some cases, further supplemented by byte ranges when range attributes are provided along with a URL. The byte range can be expressed to identify a contiguous byte range within a segment.

[0042] In some further exemplary embodiments, the sub - representation may be specified and described in the MPD file as being embedded (or included) in the normal representation, for example, using sub - representation elements / indicators. The sub - representation elements can be used to describe properties of one or more media content components embedded in the representation. For example, the sub - representation elements may be used to describe properties of an embedded audio component (e.g., codec, sampling rate, etc.), an embedded subtitle (e.g., codec), or the sub - representation elements may be used to describe some embedded low - quality video layers (e.g., some lower frame rates, etc.). The sub - representation and the representation elements can share some common attributes and elements.

[0043] In some exemplary embodiments, a DASH client can be configured to access, download, and request all or part of an MPD file from a DASH server. That is, the DASH client can obtain the MPD file used at the start of a live streaming session. Based on the MPD file and the selection of the representation, the DASH client can make several further determinations, including determining what the latest segment available on the server is, determining the segment availability start time of the next segment and possibly future segments, determining when to start playing a segment, and determining when to obtain / fetch / request a new MPD file.

[0044] In some exemplary embodiments, the MPD can further include information regarding DASH events in order to signal aperiodic information to a DASH client or DASH application. The events may be time-adjusted starting from a specific media presentation time having a duration. Additionally or alternatively, the event information can include control messages for a media player associated with a specific time during the playback of a media presentation, such as an advertisement insertion queue. The media that can be inserted during streaming may be provided from a separate server, such as an advertisement server. In addition to signaling events by the MPD separate from the media representation, the events may be multiplexed in-band within a selected media representation, either in only one or some selected adaptation sets, or in all representations.

[0045] An exemplary DASH system 200 is shown in FIG. 2. The DASH system 200 may include one or more centralized or distributed content servers 210 and information processing devices 230 connected by a network 250. The DASH system (200) may also include one or more supplementary content servers, such as one or more advertisement servers 220.

[0046] The content server 210 may provide the primary content (e.g., the main program) and its MPD to the information processing device 230. The manifest file can be generated by an MPD generator 214. The primary content and the manifest file may be provided by the same server or different servers.

[0047] The information processing apparatus 230 may include a DASH client 232 that communicates directly with the content server 210. The DASH client 232 may be controlled by the DASH application 234 of the information processing apparatus 230, request and / or receive an MPD, and may request and obtain primary content from the HTTP server 212 of the content server 210 based on the MPD. The MPD may be processed by the DASH client 232. Further, the DASH client 232 may obtain advertisement content from the advertisement server 220 or other content (e.g., interactive content) from one or more supplementary content servers in response to a DASH event. The main content and the advertisement content may be processed by the DASH client 232 or the DASH application 234 and displayed and output on the display device 236 of the information processing apparatus 230. The display device 236 may be integrated with the information processing apparatus 230 or may be external to the information processing apparatus. Further, the DASH client 232 can extract other event information from one or more timed metadata tracks and transmit the extracted event information to the DASH application 234 for further processing. The DASH application 234 may be configured to display supplementary content based on the event information, for example.

[0048] An example of the DASH client 232 is shown in FIG. 3A. The exemplary DASH client 232 can include a DASH access engine 304, a selection logic 302, and media engines 306 and 308. The DASH access engine 302 can be configured to communicate with a content server to obtain, for example, some or all of the MPD of the streaming media, request and obtain segment data of the streaming media that is dynamically requested, and request supplementary media (advertisements) according to the MPD DASH events. The selection logic 304 can be configured to determine the next one or more segments to request, including the selection of adaptation sets and representations. Such a determination can be made, for example, by user instructions and other real-time information such as network bandwidth and throughput. The media engine 306 can be configured to process the segment data received by the DASH access engine 302 according to the format of the media segment (e.g., MPEG) and the timing of the media segment to generate a main media output. The media engine 308 can be configured to process the media content associated with the timed DASH events from the DASH access engine 302 to generate a supplementary media output (such as an advertisement) that can be inserted into the main media output.

[0049] FIG. 3B shows another exemplary DASH / CMAF client architecture for processing DASH and / or CMAF events according to an embodiment of the present disclosure. The DASH / CMAF client (or DASH / CMAF player) can be configured to communicate with an application (390) and process various types of events including (i) MPD events, (ii) in-band events, and (iii) timed metadata events.

[0050] The manifest parser (305) parses a manifest (e.g., MPD). The manifest is provided, for example, by a content server (110, 210). The manifest parser (305) extracts event information regarding MPD events, in-band events, and time-domain metadata events embedded in a time-domain metadata track. The extracted event information can be provided to DASH logic (310) (e.g., DASH player control, selection, and discovery logic). The DASH logic (310) can notify an application (390) of an event scheme signaled within the manifest based on the event information.

[0051] The event information can include event scheme information for distinguishing different event streams. The application (390) can subscribe to an event scheme of interest using the event scheme information. The application (390) can further indicate a desired transmission mode for each of the subscribed schemes via one or more subscription APIs. For example, the application (390) can send a subscription request to the DASH client identifying one or more event schemes of interest and any desired corresponding transmission modes.

[0052] When the application (390) subscribes to one or more event schemes delivered as part of one or more time-domain metadata tracks, the in-band events and the "moof" parser (325) can stream one or more time-domain metadata tracks to the time-domain metadata track parser (330). For example, the in-band events and the "moof" parser (325) parse a movie fragment box ("moof") and subsequently parse the time-domain metadata track based on control information from the DASH logic (310).

[0053] The time domain metadata track parser (330) can extract event messages embedded in the time domain metadata track. The extracted event messages can be stored in an event buffer (335) (e.g., an event buffer). The synchronizer / dispatcher module (340) (e.g., an event and time domain metadata synchronizer and dispatcher) can send (or transmit) subscribed events to the application (390).

[0054] The MPD events described in the MPD can be parsed by the manifest parser (305) and stored in the buffer (335). For example, the manifest parser (305) can parse each event stream element of the MPD and parse each event described in each event stream element. For each event signaled within the MPD, event information such as presentation time and event duration can be stored in the buffer (335) associated with the event.

[0055] The in-band event and "moof" parser (325) can parse the media segment to extract in-band event messages. Any such identified in-band events and associated presentation times and durations can be stored in the buffer (335).

[0056] Accordingly, the buffer (335) can store MPD events, in-band events, and / or time domain metadata events therein. The buffer (335) can be, for example, a first-in-first-out (FIFO) buffer. The buffer (335) can be managed corresponding to the media buffer (350). For example, as long as the media segment exists within the media buffer (350), any events or time domain metadata corresponding to that media segment can be stored in the buffer (335).

[0057] The DASH access application programming interface (API) (315) can manage the fetching and receiving of a content stream (or data flow) including media content and various metadata via an HTTP protocol stack (320). The DASH access API (315) can separate the received content stream into different data flows. The data flows provided to the in-band event and moof parser can include media segments, one or more timed metadata tracks, and in-band event signaling included in the media segments. In one embodiment, the data flow provided to the manifest parser (305) can include an MPD.

[0058] The DASH access API (315) can transfer a manifest to the manifest parser (305). In addition to describing events, the manifest can also provide information about media segments to the DASH logic (310) that can communicate with the application (390) as well as the in-band event and "moof" parser (325). The application (390) can be associated with the media content processed by the DASH client. The control and synchronization signals exchanged between the application (390), the DASH logic (310), the manifest parser (305), and the DASH access API (315) can control the fetching of media segments from the HTTP stack (320) based on the information about the media segments described in the manifest.

[0059] The in-band event and moof parser (325) can parse the media data flow into a media segment that includes media content, time-domain metadata in the time-domain metadata track, and any signaled in-band event in the media segment. The media segment including the media content can be parsed by a file format parser (345) and stored in a media buffer (350).

[0060] Events stored in buffer (335) can enable the synchronizer / dispatcher (340) to communicate available events (or events of interest) related to the application to the application via the event / metadata API. The application can be configured to subscribe to specific events or time-domain metadata by processing the available events (e.g., MPD events, in-band events, or time-domain metadata events) and notifying the synchronizer / dispatcher (340). Any events stored in buffer (335) that are not related to the application but instead related to the DASH client itself can be transferred by the synchronizer / dispatcher (340) to the DASH logic (310) for further processing.

[0061] In response to an application (390) subscribing to a particular event, the synchronizer / dispatcher (340) can communicate with an event instance of the application (or a sample of time-domain metadata) corresponding to the event scheme to which the application is subscribed. The event instance can be communicated according to a transmission mode indicated by a subscription request (e.g., for a particular event scheme) or a default transmission mode. For example, in the receive-time transmission mode, the event instance may be transmitted to the application (390) upon reception in the buffer (335). On the other hand, in the start-time transmission mode, the event instances may be transmitted to the application (390) at their associated presentation times, e.g., in synchronization with a timing signal from the media decoder (355).

[0062] In some embodiments, a media segment engine (MSE) (395) can include a file format parser (345), a media buffer (350), and a media decoder (355). The MSE can receive media segments and generate a decoded media output.

[0063] Alternative MPD Configuration and Processing In some embodiments, alternative MPD events are utilized to enable streaming of multi-rate content and to switch media presentations between two timelines, i.e., one timeline for the main MPD event for the main media presentation and another timeline for the alternative MPD event for the alternative media presentation. There can be several issues / problems with the method of providing a post-processing model for the client.

[0064] For example, some embodiments using DASH design may include one or more of the following problems / issues, namely, no return point for on-demand content, the assumption that the switching time is the same as the event start time, and / or the assumption that the ad duration is the same as the event duration.

[0065] Various embodiments describe a processing model for a DASH client to provide, dispatch, and post-process alternative MPD events. Some embodiments provide different signaling and improved post-processing models for correctly processing alternative MPD events, such as enabling preroll and / or midroll ads efficiently and effectively.

[0066] Various embodiments can also address one or more of the problems described in this disclosure. For example, some embodiments can improve the media streaming field by adding a return point for on-demand content, separating these two parameters (switching time and event start time) to accurately achieve preroll and midroll, and / or separating these two parameters (ad duration and event duration) to correctly achieve switching to and from ads.

[0067] Referring to FIG. 3B, the client can request media segments based on the addresses described in the manifest. The MSE buffer can include a pipeline of a file format parser, a media buffer, and a media decoder.

[0068] FIG. 4 shows an exemplary flow diagram of an exemplary method 400 by a media streaming device (e.g., a dynamic adaptive streaming over HTTP (DASH) media streaming device) for processing an alternative MPD using a main media presentation description (MPD) from a content server. The method 400 includes step 410 of receiving, by the DASH media streaming device, a manifest for the alternative MPD from the content server, step 420 of parsing the manifest to extract a set of parameters for the alternative MPD, where the set of parameters includes at least one of a value of an event stream (e.g., a DASH event stream), a presentation time, or a duration, the presentation time indicating an offset at which the alternative media presentation starts in the timeline of the main media presentation, and the duration indicating a period during which the alternative media presentation is active, step 420, step 430 of switching from the main media presentation to the alternative media presentation based on the presentation time, and / or step 440 of playing the main media presentation at a return point according to the value of the DASH event stream in response to the end of the alternative media presentation, and may include some or all of them.

[0069] In various embodiments, the present disclosure provides a method for a media streaming content server to configure an alternative MPD using a main media presentation description (MPD). The media streaming content server can send the configured alternative MPD to the media streaming device and can configure the media streaming device to perform some or all of method 400 or any other embodiment described in the present disclosure.

[0070] In some embodiments, the presentation time is a parameter different from the actual switching time.

[0071] In some embodiments, the duration is a parameter that is different from the actual duration of the alternative media presentation.

[0072] In some embodiments, the value of the event stream includes a first value indicating a timeshift and / or the main media presentation is played until the point in time at which the main media presentation is switched to the alternative media presentation.

[0073] In some embodiments, the value of the event stream includes a second value indicating a replacement and / or depending on a first MPD type indicating dynamic, the main media presentation is played up to the live edge of the main media presentation and / or depending on a second MPD type indicating static, the main media presentation is played until the sum of the actual duration of the alternative media presentation at the point in time at which the main media presentation is switched to the alternative media presentation.

[0074] In some embodiments, the alternative MPD is processed and dispatched according to dynamic adaptive streaming over HTTP (DASH) event processing.

[0075] In some embodiments, the step of ending the alternative media presentation includes at least one of the following, namely, ending the playback of the alternative media presentation at the end of the duration or stopping the playback of the alternative media presentation when a stop command is received.

[0076] In some embodiments, depending on the MPD type indicating dynamic, depending on the value indicating replacement, the return point for playing the main media presentation is the live edge of the main media presentation, and / or depending on the value indicating time shift, the return point for playing the main media presentation is the actual switch time or the earliest available segment thereafter.

[0077] In some embodiments, depending on the MPD type indicating static, depending on the value indicating replacement, the return point for playing the main media presentation is the sum of the actual switch time and the actual duration of the alternative media presentation, and / or depending on the value indicating time shift, the return point for playing the main media presentation is the actual switch time.

[0078] The parameterized descriptor for media segment requests may be referred to as an MPD data structure. As a non-limiting example, an instantiated data structure of such a type within the MPD called "UrlQueryInfo" can include various attributes or descriptors used to provide static and dynamic parameterization mechanisms for media segments. In some further exemplary embodiments, such static and dynamic parameterization mechanisms can be extended to other types of HTTP requests in addition to media segment requests. An exemplary syntax for an extended data structure type can be specified and used to instantiate an MPD that includes corresponding data structures for signaling and defining parameterization of various types (or attributes or values).

[0079] Table 1 shows non-limiting examples of modified and / or improved semantics for alternative MPD signaling.

[0080]

Table 1

[0081] Table 2 shows other non - limiting examples of modifications and / or improved semantics for alternative MPD signaling.

[0082] [Table 2]

[0083] Various embodiments include a client post - processing model for alternative MPD events. In some embodiments, alternative MPD events are processed and dispatched according to general DASH event processing. In some embodiments, alternative MPD events may be post - processed after being dispatched. Table 3 shows non - limiting examples of parameters on which the post - processing procedure of events may depend.

[0084] [Table 3]

[0085] In the present disclosure, a uniform resource name (URN), such as "urn:mpeg:dash:event:alternative:2022", is defined and can be used with any substring defined by the owner of the URN / URL scheme.

[0086] In one non - limiting example of the client's alternative MPD switching event post - processing procedure, the method can include one or more of the following steps.

[0087] For step 1, the media streaming client (or media streaming device, or client) checks whether the alternative MPD uniform resource locator (URL) in the message is within the previously played list (PPL).

[0088] For Step 2, if the client determines that the alternative MPD URL is within its PPL, the client may not take any further action, and / or if the client determines that the alternative MPD URL is not within its PPL, the client may proceed with one or more of the following steps.

[0089] In the case of Step 3, the client downloads the alternative MPD.

[0090] For Step 4, the client determines and executes one of the following cases.

[0091] In the case of Step 4-1, if the client determines that the current playback time ≥ presentation time, it immediately proceeds to the next step (Step 5). In some embodiments, the current playback time can indicate the current time of the main media presentation, and / or the presentation_time can indicate the "target" time to switch to the alternative media presentation.

[0092] In the case of Step 4-2, when the client determines that the current playback time < presentation time, the client continues to play the main media presentation until the current playback time = presentation time, which satisfies Step 4-1, and then proceeds to the next step.

[0093] For Step 5, the client sets switch_time = current playback time. And as long as the main media presentation does not end, it switches the playback from the main media presentation to the alternative media presentation. When the main media presentation ends, it stops and clears its switch_time and PLL buffer.

[0094] For step 6, the client stores the main MPD URL and switch_time.

[0095] In step 7, the client adds a message to its PPL.

[0096] In step 8, the client downloads the main MPD from the main MPD URL at the end of the alternative media presentation. The alternative media presentation can be ended by at least one of the following: ending the playback of the alternative media presentation at the end of its duration (i.e., the alternative media presentation completes its configured duration), or stopping the playback of the alternative media presentation when a stop command is received (i.e., the user or content server ends the playback of the alternative media presentation before it completes its configured duration).

[0097] For step 9, the client continues to play the main media presentation based on one of the following steps according to the value and / or MPD type.

[0098] For step 9-1, in response to MPD@type = 'dynamic', when value ='replacement', the playback time position is from the live edge, and / or when value = 'timeshift', the playback time position is from the earliest available segment after switch_time. In some embodiments, @timeshiftBufferDepth may be set to a value greater than or equal to the maximum alternative media presentation duration to ensure that a media segment is available at switch_time when playback returns to the main media presentation.

[0099] For step 9-2, depending on MPD@type='static', when the value='replace', the playback time position is from (switch_time + the duration of the alternative media presentation), and / or when the value='timeshift', the playback time position is from switch_time.

[0100] In some embodiments, the DASH client starts from the initial parsing of the main MPD, clears its URL, switch_time, and PPL values, and continues to maintain them throughout playback.

[0101] In some embodiments, the event presentation_time and duration indicate the active time interval of the media presentation during which the media presentation is switched to the alternative media presentation. The exact time of the switch (switch_time) depends on how the player reaches the active time interval, for example, by linear playback up to its start time or by random access to a point in the middle.

[0102] Various embodiments can include a method for processing alternative MPD events, where the event start time and event duration are maintained as different parameters from the switch time and alternative advertisement duration. The event start time and duration define the time interval during which the switching time can occur while the switching time can change according to the linear playback of the content by the player or random access to the content, and the time of the switchback is defined by the alternative advertisement duration rather than the event duration. In the processing model, these values are maintained and treated separately, and the signaling of the alternative MPD event semantics is defined considering the described processing model.

[0103] The techniques described above can be implemented as computer software using computer-readable instructions and can be physically stored on one or more computer-readable media. For example, FIG. 5 shows a computer system (500) suitable for implementing certain embodiments of the disclosed subject matter.

[0104] The computer software can be coded using any suitable machine code or computer language that can be subjected to mechanisms such as assembly, compilation, linking, etc. to create code that includes instructions that can be directly executed by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc., or executed via interpretation, microcode execution, etc.

[0105] The instructions can be executed on various types of computers or their components, including, for example, personal computers, tablet computers, servers, smartphones, gaming machines, Internet of Things devices, etc.

[0106] The components shown in FIG. 5 with respect to the computer system (500) are exemplary in nature and are not intended to imply any limitation regarding the use or functionality scope of the computer software implementing embodiments of the present disclosure. Also, the configuration of the components should not be construed as having dependencies or requirements regarding any one or combination of the components shown in the exemplary embodiments of the computer system (500).

[0107] The computer system (500) may include a specific human interface input device. Such a human interface input device may respond to input by one or more users, such as, for example, tactile input (keystrokes, swipes, movement of a data glove, etc.), voice input (voice, clapping, etc.), visual input (gestures, etc.), olfactory input (not shown), etc. By using the human interface device, it is also possible to acquire some medium that is not necessarily directly related to conscious input by humans, such as voice (speech, music, ambient sound, etc.), images (scanned images, photographic images obtained from a still image camera, etc.), video (2D video, 3D video including stereoscopic video, etc.).

[0108] The input human interface device may include one or more of a keyboard (501), a mouse (502), a trackpad (503), a touch screen (510), a data glove (not shown), a joystick (505), a microphone (506), a scanner (507), a camera (508) (only one of each is shown).

[0109] The computer system (500) may also include certain human interface output devices. Such human interface output devices may, for example, stimulate the senses of one or more human users through tactile output, sound, light, and smell / taste. Such human interface output devices may include tactile output devices (e.g., a touch screen (510), a data glove (not shown), or a joystick (505) with tactile feedback, although there may also be tactile feedback devices that do not function as input devices), audio output devices (such as speakers (509), headphones (not shown), etc.), visual output devices (regardless of whether each has a touch screen input function and regardless of whether each has a tactile feedback function, there are also those that can output two-dimensional visual output or three-dimensional or higher-dimensional output by means such as stereographic output, virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown), including screens (510) such as CRT screens, LCD screens, plasma screens, and OLED screens), and may include printers (not shown).

[0110] The computer system (500) may also include a memory device accessible by humans, and related media such as optical media (521) such as CD / DVD ROM / RW (520) including CD / DVDs, thumb drives (522), removable hard drives or solid state drives (523), legacy magnetic media such as tapes and floppy disks (not shown), and dedicated ROM / ASIC / PLD-based devices such as security dongles (not shown).

[0111] Those skilled in the art will also naturally understand that the term "computer-readable medium" as used in connection with the subject matter disclosed herein does not include transmission media, carrier waves, or other transient signals.

[0112] The computer system (500) can further include an interface (554) for one or more communication networks (555). The network can be, for example, wireless, wired, or optical. The network can further be local, wide area, metropolitan, vehicular and industrial, real-time, delay-tolerant, etc. Examples of networks include local area networks such as Ethernet, wireless LAN, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., cable TV, satellite TV, and wired or wireless wide area digital networks including terrestrial broadcast TV, vehicular and industrial including CAN bus. A particular network generally requires an external network interface adapter connected to a particular general-purpose data port or peripheral bus (549) (e.g., a USB port of the computer system (500)), and others are generally integrated into the core of the computer system (500) by connection to the system bus as described below (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system (500) can communicate with other entities. Such communication can be unidirectional, receive-only (e.g., broadcast TV), transmit-only (e.g., from a CANbus to a particular CANbus device), or bidirectional, e.g., communication to other computer systems using a local area digital network or a wide area digital network. Specific protocols and protocol stacks can be used for each of those networks and network interfaces as described above.

[0113] The aforementioned human interface device, human-accessible storage device, and network interface can be connected to the core (540) of the computer system (500).

[0114] The core (540) can include one or more central processing units (CPUs) (541), a graphics processing unit (GPU) (542), a dedicated programmable processing device in the form of a field programmable gate array (FPGA) (543), a hardware accelerator for specific tasks (544), a graphics adapter (550), etc. Such devices may be connected via a system bus (548) together with internal mass storage devices such as read-only memory (ROM) (545), random access memory (546), internal hard drives that are not accessible to internal users, SSDs, etc. (547). In some computer systems, the system bus (548) may be accessible in the form of one or more physical plugs so as to be expandable by additional CPUs, GPUs, etc. Peripheral devices can be attached directly to the system bus of the core (548) or via a peripheral bus (549). In the example, the screen (510) can be connected to the graphics adapter (550). Architectures for peripheral buses include PCI, USB, etc.

[0115] The CPU (541), GPU (542), FPGA (543), and accelerator (544) can execute specific instructions that can together constitute the aforementioned computer code. The computer code can be stored in the ROM (545) or RAM (546). Migration data can be stored in the RAM (546), while persistent data can be stored, for example, in the internal mass storage device (547). By using cache memory that can be closely associated with one or more CPUs (541), GPUs (542), mass storage devices (547), ROM (545), RAM (546), etc., fast storage and reading for any memory device becomes possible.

[0116] A computer-readable medium can carry computer code for performing various computer-implemented operations. The media and the computer code can be specially designed and configured for this disclosure, or can be of the kind well known to and available from those skilled in the computer software arts.

[0117] As a non-limiting example, a computer system having an architecture (500), specifically a core (540), can provide functionality as a result of a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software embodied on one or more tangible computer-readable media. Such computer-readable media can be associated with the user-accessible mass storage devices introduced above, and media associated with specific storage devices of the core (540) having a non-transitory nature, such as the core internal mass storage device (547) and ROM (545).

[0118] The software implementing various embodiments of the present disclosure can be stored in such a device and executed by a core (540). The computer-readable medium can include one or more memory devices or chips according to specific needs. The software can cause the core (540), particularly the processors (including CPUs, GPUs, FPGAs, etc.) therein, to define data structures stored in a RAM (546) and modify such data structures according to processes defined by the software, and can execute specific processes or specific parts of specific processes described herein. In addition to, or instead of, the software, a circuit (e.g., an accelerator (544)) that can operate with or instead of the software can provide functions as a result of wired or otherwise embodied logic to execute specific processes or specific parts of specific processes described in this application. References to software can, if necessary, include logic, and vice versa. References to a computer-readable medium can, if necessary, include a circuit (such as an integrated circuit (IC)) that stores software for execution, a circuit that embodies logic for execution, or both. The present disclosure includes any suitable combination of hardware and software.

[0119] In embodiments and implementations of the present disclosure, if desired, any steps and / or operations may be combined or arranged in any amount or order. Two or more of the steps and / or operations may be executed in parallel. The embodiments and implementations of the present disclosure may be used individually or combined in any order. Further, each of a method (or embodiment), a client, and a server can be implemented by a processing circuit (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored in a non-transitory computer-readable medium.

[0120] Although the present disclosure describes several exemplary embodiments, there are changes, substitutions, and various alternative equivalents that fall within the scope of the present disclosure. Therefore, it will be understood that those skilled in the art can devise numerous systems and methods that embody the principles of the present disclosure and thus are within the spirit and scope of the present disclosure, even though not explicitly shown or described herein.

Explanation of Signs

[0121] 100 Content Delivery System 110 Content Server 120 Information Processing Device 130 Communication Network 200 DASH System 210 Content Server 212 HTTP Server 214 MPD Generator 220 Advertising Server 230 Information Processing Device 232 DASH Client 234 DASH Application 236 Display Device 250 Network 302 DASH Access Engine 304 Selection Logic 305 Manifest Parser 306 Media Engine 308 Media Engine 310 DASH Logic 315 DASH Access API 320 HTTP Protocol Stack 325 In-Band Event and "moof" Parser 330 Time-Domain Metadata Track Parser 335 Event Buffer 340 Synchronizer / Dispatcher 345 File Format Parser 350 Media Buffer 355 Media Decoder 390 Application 400 Method 500 Computer System 501 Keyboard 502 Mouse 503 Track Pad 505 Joystick 506 Microphone 507 Scanner 508 Camera 509 Audio Output Device Speaker 510 Touch Screen 520 CD / DVD ROM / RW 521 Optical Media 522 Thumb Drive 523 Solid State Drive 540 Core 541 Central Processing Unit (CPU) 542 Graphics Processing Unit (GPU) 543 Field Programmable Gate Array (FPGA) 544 Accelerator 545 Read Only Memory (ROM) 546 Random Access Memory 547 Mass Storage Device 548 System Bus 549 Peripheral Bus 550 Graphics Adapter 554 Interface 555 Communication Network

Claims

1. A method performed by an HTTP-based Dynamic Adaptive Streaming over HTTP (DASH) media streaming device for processing an alternative MPD using a main media presentation description (MPD) from a content server, the method comprising: receiving, by the DASH media streaming device, a manifest for the alternative MPD from the content server; parsing the manifest to extract a set of parameters for the alternative MPD, the set of parameters including at least one of a value of a DASH event stream, a presentation time, or a duration, the presentation time indicating an offset at which an alternative media presentation starts in a timeline of a main media presentation, and the duration indicating a period during which the alternative media presentation is active; switching from the main media presentation to the alternative media presentation based on the presentation time; resuming the main media presentation at a return point according to the value of the DASH event stream in response to an end of the alternative media presentation; A method as described above.

2. The presentation time is a parameter different from an actual switching time. The method according to claim 1.

3. The duration is a parameter different from an actual duration of the alternative media presentation. The method according to claim 1.

4. The value of the DASH event stream includes a first value indicating a time shift. The main media presentation is played until the point in time when the main media presentation is switched to the alternative media presentation. The method according to claim 1.

5. The value of the DASH event stream includes a second value indicating replacement. Depending on a first MPD type indicating dynamic, the main media presentation is played up to the live edge of the main media presentation. Depending on a second MPD type indicating static, the main media presentation is played until the time when the actual duration of the alternative media presentation is added at the point in time when the main media presentation is switched to the alternative media presentation. The method according to claim 1.

6. The alternative MPD is processed and dispatched according to dynamic adaptive streaming over hypertext transfer protocol (DASH) event processing. The method according to claim 1.

7. The step of ending the alternative media presentation is the step of ending the playback of the alternative media presentation at the end of the duration, or the step of stopping the playback of the alternative media presentation when a stop command is received. The method according to claim 1, including at least one of the above.

8. In the case of the first MPD type indicating dynamic, depending on the value indicating replacement, the return point for playing the main media presentation is the live edge of the main media presentation. Depending on the value indicating time shift, the return point for playing the main media presentation is the actual switching time or the earliest segment available thereafter. The method according to claim 5.

9. In the case of the second MPD type indicating static, Depending on the value indicating replacement, the return point for playing the main media presentation is the sum of the actual switching time and the actual duration of the alternative media presentation. Depending on the value indicating time shift, the return point for playing the main media presentation is the actual switching time. The method according to claim 5.

10. A DASH media streaming device configured to perform the method according to any one of claims 1 to 9.

11. A computer program for causing a computer to execute the method according to any one of claims 1 to 9.

Images