Method and apparatus for MPEG DASH supporting pre-roll and mid-roll content during media playback

Abstract

A method and apparatus may be provided for signaling auxiliary media including pre-roll and mid-roll media content in a primary media stream, the method including receiving a primary media presentation descriptor (MPD) including one or more primary media segments and at least one auxiliary descriptor referencing one or more auxiliary MPDs. The auxiliary MPD may include one or more auxiliary media segments independent of the one or more primary media segments. The method may include retrieving one or more auxiliary MPDs referenced in the one or more primary MPDs using the at least one auxiliary descriptor, the auxiliary MPDs of the one or more auxiliary MPDs including the one or more auxiliary media segments and the auxiliary media presentation duration. The one or more auxiliary media segments and the one or more primary media segments may be added to a media source extension (MSE) source buffer and dispatched.

Description

[Technical field]

[0001] (Reference to Related Application) This application claims priority to U.S. Provisional Application No. 63 / 253,050, filed October 6, 2021, and U.S. Application No. 17 / 949,592, filed September 21, 2022, the contents of which are expressly incorporated by reference in their entirety into this application.

[0002] (Technical field) Embodiments of the present disclosure are directed to streaming media content, and more specifically, streaming media, advertisements, and live content in accordance with the Moving Picture Experts Group (MPEG) dynamic adaptive streaming over hypertext transfer protocol (DASH). [Background technology]

[0003] MPEG DASH provides a standard for streaming media content over IP networks. In MPEG DASH, media presentation descriptions (MPDs) and events are used to deliver media timeline-related events to clients. The ISO / IEC 23009-1 DASH standard enables streaming of multi-rate content. The DASH standard provides a single linear timeline where periods are successive of each other in a single timeline. ISO / IEC 23009-1 also provides tools for MPD chaining, i.e., signaling the URL of the next MPD to play in an MPD that can be used for pre-roll ad insertion.

[0004] MPEG DASH provides a standard for streaming multimedia content over IP networks. Although the standard addresses linear playback of media content, the standard does not address situations of non-linear operation, e.g., when media segments are associated with different timelines that are independent of each other. MPD chaining and pre-roll ad insertion may be used to overcome the above-mentioned shortcomings. However, even MPD chaining and pre-roll ad insertion fail when a DASH player uses W3C Media Source Extensions, because it is very difficult to deal with such non-linear playback with one single MSE source buffer. Furthermore, MPD chaining and pre-roll ad insertion cannot be used when W3C Encrypted Media Extensions for playback of protected live content when an advertisement is expected to be played before the advertisement uses MPD chaining or pre-roll elements. Summary of the Invention [Problem to be solved by the invention]

[0005] Therefore, a need exists for a method for combining auxiliary or independent content that is distinct from the primary media content. Specifically, a need exists for a method and apparatus for combining auxiliary content with the primary media content as pre-roll or mid-roll playback. Additionally, a need exists for a method for processing the W3C Encrypted Media Extensions for playback of protected live content using MPD chaining. [Means for solving the problem]

[0006] The present disclosure addresses one or more technical problems. The present disclosure includes a method, process, apparatus, and non-transitory computer-readable medium that implements a new concept, namely, auxiliary presentations and auxiliary MPDs for the DASH standard that may enable describing a media presentation that is auxiliary or independent from a primary media presentation. In addition, embodiments of the present disclosure also provide support for including pre-roll and mid-roll content in a primary presentation.

[0007] SUMMARY Embodiments of the present disclosure may provide a method for signaling auxiliary media, including pre-roll and mid-roll media content, in primary Dynamic Adaptive Streaming over HTTP (DASH). The method may be performed by at least one processor and may include receiving one or more primary media presentation descriptors (MPDs) including one or more primary DASH media segments and at least one auxiliary descriptor including uniform resource locators (URLs) referencing one or more auxiliary MPDs, where the auxiliary MPDs among the one or more auxiliary MPDs include one or more auxiliary media segments independent from the one or more primary DASH media segments; searching for one or more auxiliary MPDs referenced by the URLs in the one or more primary MPDs using the at least one auxiliary descriptor, where the auxiliary MPDs of the one or more auxiliary MPDs include the one or more auxiliary media segments and an auxiliary media presentation duration; appending the one or more auxiliary media segments and the one or more primary DASH media segments to a media source extension (MSE) source buffer; and dispatching the one or more auxiliary media segments and the one or more primary DASH media segments from the MSE source buffer based on presentation time offsets associated with the one or more auxiliary media segments and the one or more primary DASH media segments.

[0008] An embodiment of the present disclosure may provide an apparatus for signaling auxiliary media including pre-roll and mid-roll media content in primary Dynamic Adaptive Streaming over HTTP (DASH). The apparatus may include at least one memory configured to store computer program code and at least one processor configured to access the computer program code and operate as instructed by the computer program code. The computer program code may include receiving code configured to cause the at least one processor to receive one or more primary Media Presentation Descriptors (MPDs) including one or more primary DASH media segments and at least one auxiliary descriptor including a Uniform Resource Locator (URL) referencing the one or more auxiliary MPDs, where the auxiliary MPDs in the one or more auxiliary MPDs include one or more auxiliary media segments independent from the one or more primary DASH media segments; and searching code configured to cause the at least one processor to search for one or more auxiliary MPDs referenced by the URL in the one or more primary MPDs using the at least one auxiliary descriptor, where the one or more auxiliary MPDs include one or more auxiliary media segments independent from the one or more primary DASH media segments. The auxiliary MPD of the auxiliary MPD may include: a searching code including one or more auxiliary media segments and an auxiliary media presentation duration; an appending code configured to cause at least one processor to append the one or more auxiliary media segments and the one or more primary DASH media segments to a media source extension (MSE) source buffer; and a dispatching code configured to cause the at least one processor to dispatch the one or more auxiliary media segments and the one or more primary DASH media segments from the MSE source buffer based on presentation time offsets associated with the one or more auxiliary media segments and the one or more primary DASH media segments.

[0009]

[0005] Embodiments of the present disclosure may provide a non-transitory computer-readable medium storing instructions. The instructions may include one or more instructions that, when executed by one or more processors of a device for signaling auxiliary media including pre-roll and mid-roll media content in primary Dynamic Adaptive Streaming over HTTP (DASH), cause the one or more processors to receive one or more primary Media Presentation Descriptors (MPDs) including one or more primary DASH media segments and at least one auxiliary descriptor including a Uniform Resource Locator (URL) referencing the one or more auxiliary MPDs, the one or more auxiliary MPDs including one or more auxiliary media segments independent of the one or more primary DASH media segments. and causing the one or more auxiliary MPDs referenced by the URLs in the one or more primary MPDs using the at least one auxiliary descriptor, the one or more auxiliary MPDs including the one or more auxiliary media segments and the auxiliary media presentation duration, to append the one or more auxiliary media segments and the one or more primary DASH media segments to a media source extension (MSE) source buffer, and to dispatch the one or more auxiliary media segments and the one or more primary DASH media segments from the MSE source buffer based on a presentation time offset associated with the one or more auxiliary media segments and the one or more primary DASH media segments. [Brief description of the drawings]

[0010] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings. [Figure 1] 1 is a simplified illustration of a communication system according to an embodiment.

[0011] [Diagram 2] 1 is an exemplary illustration of an arrangement of components in a streaming environment according to an embodiment.

[0012] [Diagram 3] FIG. 1 is a simplified block diagram of a DASH processing model according to an embodiment.

[0013] [Figure 4A] 1 is a simplified illustration of a Media Source Extension (MSE) source buffer implementation of a supplemental media presentation descriptor (MPD) according to an embodiment.

[0014] [Figure 4B] 1 is an exemplary script for an auxiliary MPD element for playing pre-roll and mid-roll auxiliary media content according to an embodiment.

[0015] [Diagram 5] 1 is an exemplary illustration of a stacked presentation based on an auxiliary MPD according to an embodiment.

[0016] [Figure 6] 1 illustrates an exemplary flow chart for signaling auxiliary media including pre-roll and mid-roll media content within a primary media stream according to an embodiment.

[0017] [Figure 7] 1 is a simplified diagram of a computer system according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] The proposed configurations discussed below may be used separately or combined in any order. Additionally, the embodiments may be implemented by processing circuitry (e.g., one or more processors or one or more integrated circuits). In one example, the one or more processors execute a program stored on a non-temporal computer-readable medium.

[0019] FIG. 1 illustrates a simplified block diagram of a communication system 100 according to an embodiment of the present disclosure. The communication system 100 may include at least two terminals 102 and 103 interconnected via a network 105. For one-way transmission of data, a first terminal 103 may code video data at a local location for transmission to the other terminal 102 via the network 105. A second terminal 102 may receive the coded video data of the other terminal from the network 105, decode the coded data, and display the recovered video data. One-way data transmission may be common in media serving applications, etc.

[0020] 1 shows a second pair of terminals 101 and 104 provided to support bidirectional transmission of coded video, such as may occur during a video conference. For the bidirectional transmission of data, each terminal 101 and 104 may code video data captured at a local location for transmission to the other terminal over network 105. Each terminal 101 and 104 may receive coded video data transmitted by the other terminal, may decode the coded data, and may display the recovered video data on a local display device.

[0021] In FIG. 1, terminals 101, 102, 103, and 104 may be depicted as servers, personal computers, and smartphones, although the principles of the present disclosure are not so limited. Embodiments of the present disclosure find application with laptop computers, tablet computers, media players, and / or dedicated video conferencing equipment. Network 105 represents any number of networks that convey coded video data between terminals 101, 102, 103, and 104, including, for example, wired and / or wireless communication networks. Network 105 may exchange data in circuit-switched and / or packet-switched channels. Representative networks include communication networks, local area networks, wide area networks, and / or the Internet. For purposes of this description, the architecture and topology of network 105 may not be important to the operation of the present disclosure, unless described below.

[0022] 2 shows, by way of example, the arrangement of video encoders and decoders in a streaming environment. Embodiments may be applicable to other video-enabled applications including, for example, video conferencing, digital TV, storage of compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0023] A streaming system may include a video source 201, a capture subsystem 203 that may include, for example, a digital camera, that generates an uncompressed video sample stream 213. The sample stream 213 may be highlighted as a high data volume when compared to an encoded video bitstream and may be processed by an encoder 202 coupled to the video source 201. The encoder 202 may include hardware, software, or a combination thereof to enable or implement aspects of the embodiments as described in more detail below. The encoded video bitstream 204, which may be highlighted as a lower data volume when compared to the sample stream, may be stored on a streaming server 205 for future use. One or more streaming clients 212 and 207 may access the streaming server 205 to retrieve encoded video bitstreams 208 and 206, which may be copies of the encoded video bitstream 204. The client 212 may include a video decoder 211 that decodes an input copy of the encoded video bitstream 208 and generates an output video sample stream 210 that may be rendered on a display 209 or other rendering device. In some streaming systems, the encoded video bitstreams 204, 206, and 208 may be encoded according to a particular video coding / compression standard, examples of which are mentioned above and further described herein.

[0024] 3 illustrates a sample DASH processing model 300, such as a sample client architecture for processing DASH and CMAF events. In the DASH processing model 300, a client's requests for media segments (e.g., advertisement media segments and live media segments) may be based on addresses listed in a manifest 303. The manifest 303 also describes the metadata tracks, from which the client may access segments of the metadata tracks, parse them, and send them to the application 301.

[0025] The manifest 303 includes MPD events or events, and the in-band event and "moof" parser 306 may parse the MPD event or event segments and append the event segments to the event and metadata buffer 330. The in-band event and "moof" parser 306 may also fetch media segments and append the media segments to the media buffer 340. The event and metadata buffer 330 may send the event and metadata information to an event and metadata synchronizer and dispatcher 335. The event and metadata synchronizer and dispatcher 335 may dispatch certain events to the DASH player control, selection, and discovery logic 302 and may dispatch application related events and metadata tracks to the application 301.

[0026] According to some embodiments, the MSE may include a pipeline including a file format parser 350, a media buffer 340, and a media decoder 345. The MSE 320 is a logical buffer of media segments, where the media segments may be tracked and ordered based on the presentation time of the media segments. The media segments may include, but are not limited to, advertising media segments associated with an advertising MPD and live media segments associated with a live MPD. Each media segment may be added or appended to the media buffer 340 based on a timestamp offset of the media segment, which may be used to order the media segments in the media buffer 340.

[0027] Embodiments of the present application may be directed to using MPD chaining to build a linear media source extension (MSE) buffer from two or more non-linear media sources, which may be an advertising MPD and a live MPD, and a file format parser 350 may be used to process different media and / or codecs used by live media segments included in the live MPD. In some embodiments, the file format parser may issue a change type based on the codec, profile, and / or level of the live media segments.

[0028] As long as a media segment is present in the media buffer 340, the event and metadata buffer 330 maintains the corresponding event segment and metadata. The sample DASH processing model 300 may include a timed metadata track parser 325 to track metadata associated with in-band and MPD events. According to FIG. 3, the MSE 320 includes only a file format parser 350, a media buffer 340, and a media decoder 345. The event and metadata buffer 330 and the event and metadata synchronizer and dispatcher 335 are not native to the MSE 320 and prevent the MSE 320 from natively processing events and sending them to applications.

[0029] Supporting Presentations

[0030] An embodiment of the present disclosure defines an auxiliary media presentation to be a media presentation independent of a primary media presentation of an MPD. As an example, an advertisement media segment or a live media segment independent of a primary media segment may be an auxiliary presentation. Updates to an auxiliary media presentation or an auxiliary media segment do not affect the primary media segment. Similarly, updates to a primary media segment do not affect the auxiliary media segment. Thus, an auxiliary media segment (also referred to as an auxiliary media presentation or an auxiliary presentation) may be completely independent of a primary media segment (also referred to as a primary media presentation and a media presentation in this disclosure).

[0031] Auxiliary MPD

[0032] An MPD is a media presentation description that may contain a media presentation in a hierarchical organization. An MPD may contain one or more period sequences, where each period may contain one or more adaptation sets. Each adaptation set in the MPD may contain one or more representations, where each representation contains one or more media segments. These one or more media segments carry the actual media data and associated metadata to be encoded, decoded, and / or played. An auxiliary MPD may contain one or more auxiliary media segments, and according to one aspect, each auxiliary media segment included in the auxiliary MPD may be made available at the start of the primary MPD availability. According to another aspect or the same aspect, an auxiliary MPD must include @mediationPresentationDuration. The media presentation duration may include the presentation duration of one or more auxiliary media segments. In some embodiments, the media presentation duration may include the total media presentation time of all auxiliary media segments included in the MPD.

[0033] As mentioned above, embodiments of the present disclosure define an auxiliary MPD that represents auxiliary content independent of the primary media content. According to one aspect, the primary MPD may include a reference to at least one, or in some embodiments, each auxiliary MPD using an auxiliary descriptor. The auxiliary descriptor may have a specific syntax. As an example, the auxiliary descriptor may include an element known as EssentialProperty or may include an element known as SupplementalProperty that may describe or identify the auxiliary MPD. The auxiliary descriptor may include a specific universal resource name for the scheme of the auxiliary descriptor. As an example, a specific URN for the auxiliary descriptor may include an @scheme such as "urn:mpeg:dash:auxiliary:2021". The auxiliary descriptor may include an @value that may include the auxiliary MPD URL, a start time value, and a return time value.

[0034] According to one aspect, the begin time value (also referred to in this disclosure as a begin value, a begin time attribute, and simply begin) may represent a moment in the primary media presentation at which the primary media presentation is stopped and the auxiliary media presentation is played. In some embodiments, if begin=0 for the auxiliary media segment, the auxiliary media presentation may be a pre-roll auxiliary media segment. If the auxiliary media presentation is a pre-roll auxiliary media segment, the auxiliary media segment may be played for the first time before playing the primary media segment.

[0035] According to one aspect, the return time value (sometimes referred to as "returnType") may represent an offset from a moment in the primary media timeline at which the primary media presentation is stopped to play the auxiliary MPD or one or more auxiliary media segments. In some embodiments, the default value for the returnTime value is the media presentation duration associated with the auxiliary MPD. A returnTime value of zero results in the joining of the primary media presentation at the moment the primary media presentation is stopped to play the auxiliary media segments.

[0036] FIG. 4B is an example script 450 showing xml code that enables auxiliary MPD elements to play pre-roll and mid-roll auxiliary media content, according to an embodiment.

[0037] 4B, the auxiliary MPD described by the EssentialProperty descriptor with @id="e0" may be a pre-roll media segment, while the auxiliary MPD described by the EssentialProperty descriptors with @id="e1" and @id="e2" may be mid-roll media segments. In some embodiments, the auxiliary MPD may be disclosed before one or more periods in the auxiliary MPD.

[0038] Implementing auxiliary MPD on MSE source buffer.

[0039] The W3C MSE is a sequential buffer. As long as the MSE source buffer TimeStamp Offset (TSO) is set appropriately so that content is expected to be played in order and is appended to the correct location, the MSE source buffer will play content correctly. According to one embodiment, each time the primary presentation switches to an auxiliary presentation or back to the primary presentation, the MSE timestamp offset may be adjusted to place the primary and auxiliary media segments in the correct location in the MSE buffer.

[0040] FIG. 4A is a diagram 400 of the MSE source buffer, which is explained using the xml code of FIG. 4B.

[0041] As shown in diagram 400, auxiliary MPD Aux e0 may be a pre-roll MPD, and Aux e1 and Aux e2 may be mid-roll MPDs. Live content p0 may be a primary MPD that includes one or more primary media segments.

[0042] Since Aux e0 is a pre-roll MPD, one or more auxiliary media segments from Aux e0 may be fetched and placed at the beginning of the MSE source buffer before any main media segments. When fetching and adding pre-roll auxiliary media segments in Aux e0, the MSE TSO is zero.

number

[0043] In some embodiments, following fetching and appending each pre-roll auxiliary media segment, the MSE TSO may be updated based on the start time of the live media p0, the presentation time of the live media p0, and the time-shift buffer depth. As an example, in FIG. 4A, T1 may be based on the time-shift buffer depth, the duration of the pre-roll auxiliary media segment, and the presentation time offset associated with the live media.

number

[0044] For mid-roll auxiliary MPDs aux e1 and aux e2:

number

number

number

number

[0045] where e0@PTO, e1@PTO, and e2@PTO are the presentation time offsets of the first period in the auxiliary MPD linked by the basic descriptor in that order, and ei@starttime and ei@returnTime are the startTime and returnTime values ​​in each corresponding descriptor with id=ei.

[0046] According to an embodiment, updating the MSE TSO may be based on the following:

[0047] The pre-roll auxiliary media segment may start from the MSE Add Window Start (0). Since the pre-roll auxiliary media segment is fetched and added first, the MSE TSO is set equal to the PTO of the auxiliary MPD (see, for example, equation (1)).

[0048] In some embodiments, a live edge associated with the live media or primary media segment may be calculated after adding the last segment of the pre-roll auxiliary media segment. Using the size of the DVR window, the live edge, the period start and duration of the pre-roll auxiliary MPD or media segment, and the PTO of the live content, the MSE TSO may be adjusted so that there is a threshold time range between the first media segment added to the MSE buffer and the last segment of the previously added pre-roll auxiliary MPD. See equation (2) as an example. In some embodiments, the threshold may be less than, equal to, or greater than the DVR window. The DVR window may include the ability for the client to scrub back or start playing at an earlier point in the primary media stream.

[0049] According to an embodiment, for each added auxiliary MPD or media segment, its offset from the live edge is calculated. The MSE TSO is increased by this offset and adjusted based on the live period PTO and the PTO of the auxiliary MPD / media segment. In another embodiment or the same embodiment, at the edge of each auxiliary MPD / media segment, the MSE TSO is adjusted based on the returnTime of the auxiliary MPD / media segment, the live period PTO, and the PTO of the auxiliary MPD / media segment.

[0050] In some embodiments, in playing a pre-roll auxiliary MPD / media segment, the client may decide to remove the pre-roll auxiliary MPD / media segment time range from the MSE, so the pre-roll may be played only once. In some embodiments, if a DVR is accessed, the pre-roll may not be played. After joining a live session, the client may remove the time range from the MSE additional window buffer according to the value of timeShiftBufferDepth.

[0051] FIG. 5 is an example stacker presentation 500 illustrating staking presentations based on an auxiliary MPD.

[0052] As seen in Figure 5, a combined presentation may start with live content (primary media) and may also include one or more independent auxiliary media segments. Based on Aux MPD1 being referenced in the primary MPD, the presentation of the live content (primary media segment) may be stopped and the auxiliary media segment from Aux1 may be presented instead.

[0053] In some implementations, based on Aux MPD2 being referenced in Aux MPD1, the presentation of the auxiliary media segment from Aux1 may be stopped and the auxiliary media segment from Aux2 may be played and / or dispatched instead. At the end of Aux1 and the auxiliary media segment from Aux1 and at the end of Aux2 and the auxiliary media segment from Aux2, the presentation may switch back to the live content. Thus, when the primary media stream is unrelated to the auxiliary media stream, the auxiliary media segment (or presentation) may be supported by the MSE source buffer as pre-roll or mid-roll auxiliary media.

[0054] Thus, embodiments of the present disclosure may provide a method and apparatus for signaling from a primary media presentation defined in an MPD to an auxiliary media presentation by inserting pre-roll and mid-roll content into the media presentation. Each auxiliary presentation may be represented by a mandatory or supplemental descriptor in the MPD, which may include a URL to the auxiliary MPD. There may be additional constraints including the time of stopping and rejoining the primary media presentation, which may be defined by startTime and returnTime values ​​in the same descriptor. In some embodiments, the auxiliary MPD may be used for playing pre-roll auxiliary content before playing the primary content or inserting the auxiliary content mid-roll. Constraints may also define whether the primary media presentation may need to be time-shifted after the mid-roll or whether the mid-roll replaces the primary content during that period of the auxiliary MPD. In some embodiments, the MPD period may be updated independently from the auxiliary MPD, and the MSE and EME may have a single MSE buffer for playing the primary and auxiliary content, because the buffer may be set taking into account the primary content characteristics.

[0055] Embodiments of the present disclosure may also provide a method and apparatus for fetching and playing auxiliary content along with the primary content using an MSE source buffer by defining operations for adding various segments for pre-roll auxiliary content, mid-roll auxiliary content, and primary content based on an MSE TSO adjusted according to an embodiment of the present disclosure.

[0056] Embodiments of the present disclosure may be directed to combining presentations by stacking auxiliary MPDs according to any embodiment of the present specification, where each main MPD may have a reference to an auxiliary MPD, and the auxiliary MPD may reference another auxiliary MPD that generates a stack of the referenced MPDs resulting in a stacked presentation.

[0057] FIG. 6 is an example flow chart of a process 600 for signaling auxiliary media including pre-roll and mid-roll media content in a primary media stream.

[0058] At operation 610, one or more primary media presentation descriptors including one or more primary media segments and at least one auxiliary descriptor referencing one or more auxiliary MPDs may be received. As an example, the MSE 320 may receive one or more primary media presentation descriptions including one or more primary media segments and at least one auxiliary descriptor referencing one or more auxiliary MPDs. In some embodiments, the auxiliary MPDs in the one or more auxiliary MPDs may include one or more auxiliary media segments independent from the one or more primary media segments. The auxiliary descriptor may include a start time attribute indicating a first time offset at which the primary media segment in the one or more primary media segments may be stopped to dispatch the auxiliary media segment in the one or more auxiliary media segments.

[0059] The auxiliary descriptor may include a return time attribute indicating a second time offset at which the primary media segment in the one or more primary media segments may be resumed after dispatching the auxiliary media segment in the one or more auxiliary media segments. In some embodiments, the default return time attribute value may be the auxiliary media presentation duration associated with the auxiliary media segment. In some embodiments, the auxiliary descriptor may include a mandatory or supplemental characteristic descriptor at the MPD level, a uniform resource name (URN) for a scheme associated with the at least one auxiliary descriptor, or a uniform resource locator associated with the auxiliary MPD in the one or more auxiliary MPDs.

[0060] At operation 615, one or more auxiliary MPDs referenced in the one or more primary MPDs may be retrieved using the at least one auxiliary descriptor. As an example, the MSE 320 may retrieve one or more auxiliary MPDs referenced in the one or more primary MPDs and may use the at least one auxiliary descriptor. The auxiliary MPDs of the one or more auxiliary MPDs may include one or more auxiliary media segments and an auxiliary media presentation duration.

[0061] At operation 620, the one or more auxiliary media segments and the one or more primary media segments may be added to a media source extension (MSE) source buffer, such as MSE 320.

[0062] In some embodiments, based on an auxiliary media segment among the one or more auxiliary media segments having a start time attribute equal to zero, the auxiliary media segment may be a pre-roll auxiliary media segment, and the pre-roll auxiliary media segment may be dispatched before the one or more primary media segments.

[0063] At operation 625, the one or more primary media segments and the one or more auxiliary media segments from the MSE source buffer may be dispatched based on presentation time offsets associated with the one or more auxiliary media segments and the one or more primary media segments. As an example, the MSE 320 may dispatch the one or more auxiliary media segments and the one or more primary media segments based on timing information from the timed metadata track parser 325, the event and metadata buffer 330, and the event and metadata synchronizer and dispatcher 335.

[0064] In operation 630, the MSE source buffer timestamp offset may be updated based on the primary media segment among the one or more primary media segments being stopped or the primary media segment among the one or more primary media segments being resumed. As an example, the MSE 320 may be updated.

[0065] In some embodiments, the MSE source buffer timestamp offset may be equal to a presentation time offset associated with the auxiliary media segment. In some embodiments, the MSE source buffer timestamp offset may be updated based on the depth of the time shift buffer, a start time attribute associated with the auxiliary media segment, a total duration of all pre-roll media segments, and a presentation time offset associated with a first primary media segment of the one or more primary media segments. In the same or different embodiments, the MSE source buffer timestamp offset may be updated to include a threshold time range between the last pre-roll auxiliary media segment and the first primary media segment. In some embodiments, the MSE source buffer timestamp offset may be updated based on a presentation time offset and a return time attribute associated with one of the one or more auxiliary media segments when one of the one or more auxiliary media segments in the MSE source buffer is dispatched.

[0066] 6 illustrates example blocks of process 600, in embodiments, process 600 may include additional, fewer, different, or differently arranged blocks than those illustrated in FIG 6. In embodiments, any of the blocks of process 600 may be combined or arranged in any amount or order as desired. In embodiments, two or more of the blocks of process 600 may be performed in parallel.

[0067] The techniques described above may be implemented using computer-readable instructions stored on one or more computer-readable mediums, or by one or more specially configured hardware processors. For example, Figure 7 illustrates a computer system 700 suitable for implementing various embodiments.

[0068] Computer software can be coded using any suitable machine code or computer language that can be subject to assembly, compilation, linking, or similar mechanisms to produce code containing instructions that can be executed by computer central processing units (CPUs), graphics processing units (GPUs), etc. directly, or through interpretation, microcode execution, etc.

[0069] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smart phones, gaming devices, Internet of Things devices, and the like.

[0070] 7 for computer system 700 are exemplary in nature and are not intended to suggest any limitations on the scope of use or functionality of the computer software implementing the embodiments of the present disclosure, nor should the arrangement of components be interpreted as having any dependency or requirement regarding any one or combination of components illustrated in the exemplary embodiment of computer system 700.

[0071] The computer system 700 may include certain human interface input devices. Such human interface input devices may be responsive to input by one or more human users, for example, through tactile input (such as keystrokes, swipes, data glove movements), audio input (such as voice, clapping), visual input (such as gestures), and olfactory input. Human interface devices may also be used to capture certain media that are not necessarily directly related to conscious human input, such as audio (such as speech, music, ambience), images (such as scanned images, photographic images obtained from a still image camera), and video (such as two-dimensional video, three-dimensional video including stereoscopic video).

[0072] The input human interface devices may include one or more of a keyboard 701, a mouse 702, a trackpad 703, a touch screen 710, a joystick 705, a microphone 706, a scanner 708, a camera 707 (only one of each is shown in the drawings).

[0073] The computer system 700 may also include certain human interface output devices. Such human interface output devices may stimulate one or more of the senses of a human user, for example, through haptic output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touch screen 710 or joystick 705, however, there may be haptic feedback devices that do not function as input devices), audio output devices (such as speakers 709, headphones), visual output devices (such as screens 710, including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touch screen input capability, each with or without haptic feedback capability - some of which may be capable of outputting two-dimensional visual output or output with more than three dimensions through such means as stereoscopic output, virtual reality glasses, holographic displays, and smoke tanks), and printers.

[0074] The computer system 700 may also include human accessible storage devices and their associated media, such as optical media including CD / DVD 711 or CD / DVD ROM / RW 720 with similar media, thumb drives 722, removable hard drives or solid state drives 723, legacy magnetic media such as tape and floppy disks, specialized ROM / ASIC / PLD based devices based on devices such as security dongles.

[0075] Those skilled in the art will also understand that the term "computer-readable medium" as used in connection with the presently disclosed subject matter does not encompass transmission media, carrier waves, or other transient signals.

[0076] The computer system 700 may also include an interface 799 to one or more communication networks 798. The network 798 may be, for example, wireless, wired, optical. The network 798 may further be local, wide area, metropolitan, vehicular and industrial, real-time, delay tolerant, etc. Examples of the network 798 include local area networks such as Ethernet, wireless LAN, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., TV wiring or wireless wide area digital networks including cable TV, satellite TV, terrestrial broadcast TV, vehicular and industrial including CANBus. Certain networks 798 generally require an external network interface adapter attached to a particular general-purpose data port (e.g., a USB port of the computer system 700) or peripheral bus (750 and 751), while others are generally integrated into the core of the computer system 700 by attachment to a system bus (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system) as described below. Using any of these networks 798, the computer system 700 may communicate with other entities. Such communications may be unidirectional, receive only (e.g., broadcast television), unidirectional transmit only (e.g., CANbus to a particular CANbus device), or bidirectional, for example, to other computer systems using local or wide area digital networks. Specific protocols and protocol stacks may be used in each of these networks and network interfaces, as described above.

[0077] The aforementioned human interface devices, human accessible storage devices, and network interfaces may be attached to the core 740 of the computer system 700 .

[0078] The core 740 may include one or more central processing units (CPUs) 741, graphics processing units (GPUs) 742, graphics adapters 717, specialized programmable processing units in the form of field programmable gate areas (FPGAs) 743, hardware accelerators for specific tasks, etc. These devices may be connected through a system bus 748, along with internal mass storage devices 747 such as read only memory (ROM) 745, random access memory (RAM) 746, internal non-user accessible hard drives, SSDs, etc. 774, etc. In some computer systems, the system bus 748 may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripheral devices may be attached directly to the core's system bus 748 or through a peripheral bus 751. Architectures for peripheral buses include PCI, USB, etc.

[0079] The CPU 741, GPU 742, FPGA 743, and accelerator 744 can execute certain instructions that in combination can constitute the computer code described above. That computer code can be stored in ROM 745 or RAM 746. Transient data can also be stored in RAM 746, while permanent data can be stored in, for example, internal mass storage device 747. Rapid storage and retrieval from any of the memory devices can be made possible through the use of cache memory, which can be closely associated with one or more of the CPU 741, GPU 742, mass storage device 747, ROM 745, RAM 746, etc.

[0080] The computer-readable medium can have computer code thereon for performing various computer-implemented operations. The medium and computer code can be those specially designed and constructed for the purposes of the present disclosure, or they can be of the kind well known and available to those skilled in the computer software arts.

[0081] As an example, and not by way of limitation, computer system 700 having the illustrated architecture, and in particular core 740, can provide functionality as a result of processors (including CPUs, GPUs, FPGAs, accelerators, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media can be media associated with user-accessible mass storage devices as described above, as well as specific storage devices of core 740 that are non-transitory in nature, such as core internal mass storage device 747 or ROM 745. Software implementing various embodiments of the present disclosure can be stored in such devices and executed by core 740. Computer-readable media can include one or more memory devices or chips, depending on particular needs. The software can cause core 740, and in particular the processors therein (including CPUs, GPUs, FPGAs, etc.) to perform certain processes or certain portions of certain processes described herein, including defining data structures stored in RAM 746 and modifying such data structures according to processes defined by the software. Additionally or alternatively, a computer system may provide functionality as a result of logic hardwired or otherwise embodied in circuitry (e.g., accelerator 744), which may operate in place of or in conjunction with software to perform particular processes or particular portions of particular processes described herein. Reference to software may include logic, and vice versa, where appropriate. Reference to a computer-readable medium may encompass circuitry that stores software for execution, circuitry that embodies logic for execution, or both, where appropriate. The present disclosure encompasses any suitable combination of hardware and software.

[0082] While this disclosure has described several exemplary embodiments, there are alterations, substitutions, and various substitute equivalents that are within the scope of this disclosure. Thus, it will be appreciated that those skilled in the art can devise many systems and methods that, although not explicitly shown or described herein, embody the principles of the present disclosure and are therefore within the spirit and scope of the present disclosure.

Claims

1. A method for signaling auxiliary media, including pre-roll and mid-roll media content, in a primary dynamic adaptive streaming over hypertext transfer protocol (DASH) media stream, the method being performed by at least one processor, the method comprising: receiving one or more primary Media Presentation Descriptors (MPDs) including one or more primary DASH media segments and at least one auxiliary descriptor including a Uniform Resource Locator (URL) referencing one or more auxiliary MPDs, wherein an auxiliary MPD among the one or more auxiliary MPDs includes one or more auxiliary media segments separate from the one or more primary DASH media segments; - searching for one or more auxiliary MPDs referenced by the URL in the one or more primary MPDs using the at least one auxiliary descriptor, the one or more auxiliary MPDs including the one or more auxiliary media segments and an auxiliary media presentation duration; appending the one or more auxiliary media segments and the one or more primary DASH media segments to a media source extension (MSE) source buffer; dispatching the one or more auxiliary media segments and the one or more primary DASH media segments from the MSE source buffer based on presentation time offsets associated with the one or more auxiliary media segments and the one or more primary DASH media segments. method.

2. The at least one auxiliary descriptor is a start time attribute indicating a first time offset at which a primary DASH media segment of the one or more primary DASH media segments is stopped for dispatching an auxiliary media segment of the one or more auxiliary media segments; a return time attribute indicating a second time offset at which the primary DASH media segment in the one or more primary DASH media segments resumes after dispatching the auxiliary media segment in the one or more auxiliary media segments. The method of claim 1.

3. 3. The method of claim 2, further comprising updating an MSE source buffer timestamp offset based on the primary DASH media segment in the one or more primary DASH media segments being stopped or the primary DASH media segment in the one or more primary DASH media segments being resumed.

4. The method of claim 2 , wherein based on the auxiliary media segment among the one or more auxiliary media segments having the start time attribute equal to zero, the auxiliary media segment is a pre-roll auxiliary media segment.

5. 5. The method of claim 4, wherein dispatching the one or more auxiliary media segments and the one or more primary DASH media segments from the MSE source buffer based on the auxiliary media segments being the pre-roll auxiliary media segments comprises dispatching a pre-roll auxiliary media segment prior to the one or more primary DASH media segments.

6. The method of claim 5 , wherein based on the auxiliary media segment being the pre-roll auxiliary media segment, an MSE source buffer timestamp offset is equal to a presentation time offset associated with the auxiliary media segment.

7. the MSE source buffer timestamp offset is updated based on a time shift buffer depth, the start time attribute associated with the auxiliary media segment, a total duration of all pre-roll media segments, and a presentation time offset associated with a first primary DASH media segment of the one or more primary DASH media segments; the MSE source buffer timestamp offset is updated to include a threshold time range between a last pre-roll auxiliary media segment and the first primary DASH media segment. The method according to claim 6.

8. 8. The method of claim 7, wherein the MSE source buffer timestamp offset is updated based on a presentation time offset and the return time attribute associated with the one of the one or more auxiliary media segments in the MSE source buffer when the one of the one or more auxiliary media segments is dispatched.

9. The method of claim 2 , wherein a default return time attribute value is the auxiliary media presentation duration associated with the auxiliary media segment.

10. The at least one auxiliary descriptor is Mandatory or supplemental characteristic descriptors at MPD level; a Uniform Resource Name (URN) for a scheme associated with said at least one auxiliary descriptor; and a Uniform Resource Locator (URL) associated with the auxiliary MPD among the one or more auxiliary MPDs. The method of claim 1.

11. An apparatus for signaling auxiliary media, including pre-roll and mid-roll media content, in a primary dynamic adaptive streaming over hypertext transfer protocol (DASH) media stream, comprising: at least one memory configured to store computer program code; at least one processor configured to access the computer program code and to operate as instructed by the computer program code; The computer program code comprises: receiving code configured to cause the at least one processor to receive one or more primary Media Presentation Descriptors (MPDs) including one or more primary DASH media segments and at least one auxiliary descriptor including Uniform Resource Locators (URLs) referencing one or more auxiliary MPDs, where an auxiliary MPD in the one or more auxiliary MPDs includes one or more auxiliary media segments separate from the one or more primary DASH media segments; a searching code configured to cause the at least one processor to use the at least one auxiliary descriptor to search for one or more auxiliary MPDs referenced by the URL in the one or more primary MPDs, the auxiliary MPDs of the one or more auxiliary MPDs including the one or more auxiliary media segments and an auxiliary media presentation duration; code configured to cause the at least one processor to append the one or more auxiliary media segments and the one or more primary DASH media segments to a Media Source Extension (MSE) source buffer; and a dispatching code configured to cause the at least one processor to dispatch the one or more auxiliary media segments and the one or more primary DASH media segments from the MSE source buffer based on a presentation time offset associated with the one or more auxiliary media segments and the one or more primary DASH media segments. Device.

12. The at least one auxiliary descriptor is a start time attribute indicating a first time offset at which a primary DASH media segment of the one or more primary DASH media segments is stopped for dispatching an auxiliary media segment of the one or more auxiliary media segments; a return time attribute indicating a second time offset at which the primary DASH media segment in the one or more primary DASH media segments resumes after dispatching the auxiliary media segment in the one or more auxiliary media segments.

12. The apparatus of claim 11.

13. The computer program code comprises: and a first updating code configured to cause the at least one processor to update an MSE source buffer timestamp offset based on the primary DASH media segment in the one or more primary DASH media segments being stopped or the primary DASH media segment in the one or more primary DASH media segments being resumed.

13. The apparatus of claim 12.

14. The apparatus of claim 12 , wherein based on the auxiliary media segment among the one or more auxiliary media segments having the start time attribute equal to zero, the auxiliary media segment is a pre-roll auxiliary media segment.

15. 15. The apparatus of claim 14, wherein dispatching the one or more auxiliary media segments and the one or more primary DASH media segments from the MSE source buffer based on the auxiliary media segments being the pre-roll auxiliary media segments comprises dispatching a pre-roll auxiliary media segment prior to the one or more primary DASH media segments.

16. The apparatus of claim 15 , wherein based on the auxiliary media segment being the pre-roll auxiliary media segment, an MSE source buffer timestamp offset is equal to a presentation time offset associated with the auxiliary media segment.

17. the MSE source buffer timestamp offset is updated based on a time shift buffer depth, the start time attribute associated with the auxiliary media segment, a total duration of all pre-roll media segments, and a presentation time offset associated with a first primary DASH media segment of the one or more primary DASH media segments; the MSE source buffer timestamp offset is updated to include a threshold time range between a last pre-roll auxiliary media segment and the first primary DASH media segment.

17. The apparatus of claim 16.

18. 20. The apparatus of claim 17, wherein the MSE source buffer timestamp offset is updated based on a presentation time offset and the return time attribute associated with the one of the one or more auxiliary media segments in the MSE source buffer when the one of the one or more auxiliary media segments is dispatched.

19. A computer program product causing at least one processor to carry out the method according to any one of claims 1 to 10.

Images