Method and apparatus for indicating secondary media in primary DASH media stream

Abstract

Embodiments of the present application provide a method and apparatus for indicating secondary media in primary DASH media streams, the method comprising receiving a primary media presentation description, MPD, including one or more primary media segments and at least one secondary descriptor referencing one or more secondary media presentation descriptions, MPDs. The secondary MPDs can include one or more secondary media segments independent of the one or more primary media segments. The method can include retrieving the one or more secondary MPDs referenced in the one or more primary MPDs using the at least one secondary descriptor, the secondary MPD of the one or more secondary MPDs including the one or more secondary media segments and a secondary media presentation duration. The one or more secondary media segments and the one or more primary media segments can be appended to a media source extension, MSE, source buffer and scheduled.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to U.S. Provisional Application No. 63 / 253,050, filed October 6, 2021, and U.S. Application No. 17 / 949,592, filed September 21, 2022, the entire contents of which are hereby expressly incorporated by reference. Technical Field

[0003] The embodiments of this disclosure primarily relate to the field of streaming media technology, and more specifically, to a method and apparatus for indicating secondary media in a primary DASH media stream, electronic devices, and computer-readable media. Background Technology

[0004] MPEG DASH provides a standard for streaming media content over IP networks. In MPEG DASH, media presentation descriptions (MPDs) and events are used to deliver media timeline-related events to the client. The ISO / IEC 23009-1 DASH standard allows for the streaming of multi-rate content. The DASH standard provides a single linear timeline in which time segments are continuations of each other within a single timeline. ISO / IEC 23009-1 also provides tools for MPD linking, i.e., signaling the URL of the next MPD to be played within an MPD that can be used for pre-roll ad insertion.

[0005] MPEG DASH provides a standard for streaming multimedia content over IP networks. While this standard addresses linear playback of media content, it fails to handle non-linear scenarios, such as media segments associated with different, independent timelines. MPD linking and pre-roll ad insertion can overcome these shortcomings. However, even with MPD linking and pre-roll ad insertion, DASH players will fail when using W3C media source extensions because handling such non-linear playback with a single MSE source buffer is extremely challenging. Furthermore, MPD linking and pre-roll ad insertion cannot be used with W3C encrypted media extensions used for playing protected live content when it is expected that ads will play before using MPD linking or pre-roll elements.

[0006] Therefore, methods are needed for combining supplementary or independent content that differs from the main media content. Specifically, methods and apparatus are needed for combining supplementary content with main media content for pre-roll or mid-roll playback. Furthermore, methods are needed for using MPD links to process W3C encrypted media extensions for playback of protected live content. Summary of the Invention

[0007] Implementations of this disclosure may provide a method for signaling auxiliary media, including pre-roll and mid-roll media content, in a dynamic adaptive streaming over HTTP (DASH) media stream that is primarily over HTTP. The method may include: receiving one or more primary media presentation descriptions (MPDs) comprising one or more primary DASH media segments and at least one secondary descriptor, wherein the at least one secondary descriptor includes a Uniform Resource Locator (URL) referencing one or more secondary MPDs, wherein the secondary MPDs among the one or more secondary MPDs include one or more secondary media segments independent of the one or more primary DASH media segments; retrieving one or more secondary MPDs referenced by the URLs in the one or more primary MPDs using the at least one secondary descriptor, wherein the secondary MPDs of the one or more secondary MPDs include one or more secondary media segments and secondary media presentation durations; appending one or more secondary media segments and one or more primary DASH media segments to a media source extension (MSE) source buffer; and scheduling one or more secondary media segments and one or more primary DASH media segments from the MSE source buffer based on a presentation time offset associated with one or more secondary media segments and one or more primary DASH media segments.

[0008] Embodiments of this disclosure may provide an apparatus for signaling auxiliary media, including pre-roll and mid-roll media content, within a primary dynamic adaptive streaming over HTTP (DASH) media stream. The apparatus may include: at least one memory configured to store computer program code; and at least one processor configured to access the computer program code and operate as instructed by the computer program code. The program code may include: receiving code configured to cause at least one processor to receive one or more primary media presentation descriptions (MPDs) comprising one or more primary DASH media segments and at least one secondary descriptor, wherein the at least one secondary descriptor includes a Uniform Resource Locator (URL) referencing one or more secondary MPDs, wherein the secondary MPDs among the one or more secondary MPDs include one or more secondary media segments independent of the one or more primary DASH media segments; retrieval code configured to cause at least one processor to retrieve one or more secondary MPDs referenced by the URLs in the one or more primary MPDs using at least one secondary descriptor, wherein the secondary MPDs among the one or more secondary MPDs may include one or more secondary media segments and secondary media presentation durations; and appending code configured to cause at least one processor to append one or more secondary media segments and one or more primary DASH media segments to a media source extension (media... The source extension (MSE) source buffer; and scheduling code configured to cause at least one processor to schedule one or more secondary media segments and one or more primary DASH media segments from the MSE source buffer based on the presentation time offset associated with one or more secondary media segments and one or more primary DASH media segments.

[0009] Embodiments of this disclosure may provide an electronic device including a memory and a processor, the memory storing computer-readable instructions which, when executed by the processor, cause the electronic device to perform the methods described above.

[0010] Embodiments of this disclosure may provide a non-transitory computer-readable medium that stores instructions. This instruction may include one or more instructions that, when executed by one or more processors of a device for signaling secondary media, including pre-positioned and center-positioned media content, in a dynamic adaptive streaming over HTTP (DASH) media stream, cause one or more processors to perform the following operations: receive one or more primary media presentation descriptions (MPDs) including one or more primary media segments and at least one secondary descriptor, wherein the at least one secondary descriptor includes a Uniform Resource Locator (URL) referencing one or more secondary MPDs, wherein the secondary MPDs among the one or more secondary MPDs may include one or more secondary media segments independent of the one or more primary DASH media segments; retrieve one or more secondary MPDs referenced by the URLs in the one or more primary MPDs using at least one secondary descriptor, wherein the secondary MPDs of the one or more secondary MPDs may include one or more secondary media segments and secondary media presentation durations; and append one or more secondary media segments and one or more primary DASH media segments to a media source extension. The extension (MSE) source buffer; and the scheduling of one or more auxiliary media segments and one or more main DASH media segments from the MSE source buffer based on the rendering time offset associated with one or more auxiliary media segments and one or more main DASH media segments.

[0011] In summary, embodiments of this application provide methods, processes, apparatus, and non-transitory computer-readable media for implementing the novel concept—the DASH standard—as auxiliary MPD and auxiliary presentation, which allows for the description of auxiliary or independent media presentations from the main media presentation. Furthermore, embodiments of this disclosure also provide support for including pre-roll and in-roll content during the main presentation. Attached Figure Description

[0012] Other features, properties, and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings, in which:

[0013] Figure 1 This is a simplified illustration of a communication system according to an embodiment of this application;

[0014] Figure 2 This is an example illustration of the placement of components according to an embodiment of this application in a streaming environment;

[0015] Figure 3 This is a simplified block diagram of a DASH processing model according to an embodiment of this application;

[0016] Figure 4A This is a simplified illustration of a media source extension (MSE) source buffer implementation of an auxiliary media presentation descriptor (MPD) according to an embodiment of this application.

[0017] Figure 4B This is an exemplary script for an auxiliary MPD element for playing front-facing and center-facing auxiliary media content, according to an embodiment of this application;

[0018] Figure 5 This is an exemplary illustration of a stack based on an auxiliary MPD according to an embodiment of this application;

[0019] Figure 6 This is an exemplary flowchart illustration of an embodiment of the present application for signaling auxiliary media, including pre-roll media content and center-roll media content, in a main media stream;

[0020] Figure 7 This is a simplified diagram of a computer system according to an embodiment of this application. Detailed Implementation

[0021] The proposed features discussed below can be used individually or in any order. Furthermore, implementations can be carried out using a processing circuit system (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored on a non-transitory computer-readable medium.

[0022] Figure 1 A simplified block diagram of a communication system 100 according to an embodiment of the present disclosure is shown. The communication system 100 may include at least two terminals 102 and 103 interconnected via a network 105. For unidirectional data transmission, the first terminal 103 may encode video data at a local location for transmission to the other terminal 102 via the network 105. The second terminal 102 may receive the encoded video data from the other terminal from the network 105, decode the encoded data, and display the recovered video data. Unidirectional data transmission is common in media service applications, etc.

[0023] Figure 1A second pair of terminals 101 and 104 is shown, which are provided to support bidirectional transmission of encoded video, which may occur, for example, during video conferencing. For bidirectional data transmission, each terminal 101 and 104 can encode video data captured at a local location for transmission to the other terminal via network 105. Each terminal 101 and 104 can also receive encoded video data sent by the other terminal, can decode the encoded data, and can display the recovered video data on a local display device.

[0024] exist Figure 1 In this disclosure, terminals 101, 102, 103, and 104 may be shown as servers, personal computers, and smartphones, but the principles of this disclosure are not limited thereto. Embodiments of this disclosure are applicable to laptop computers, tablet computers, media players, and / or dedicated video conferencing equipment. Network 105 refers to any number of networks transmitting encoded video data between terminals 101, 102, 103, and 104, including, for example, wired and / or wireless communication networks. Network 105 may exchange data in circuit-switched and / or packet-switched channels. Representative networks include telecommunications networks, local area networks (LANs), wide area networks (WANs), and / or the Internet. For the purposes of this discussion, unless otherwise stated below, the architecture and topology of network 105 may be irrelevant to the operation of this disclosure.

[0025] Figure 2 The placement of a video encoder and decoder in a streaming environment is illustrated as an example. This implementation can be adapted to other video-enabled applications, including, for example, video conferencing, digital television, and storing compressed video on digital media including CDs, DVDs, Memory Sticks, etc.

[0026] The streaming system may include a capture subsystem 203, which may include a video source 201 (e.g., a digital camera device) that creates, for example, an uncompressed video sample stream 213. The sample stream 213 may be characterized by a high data volume when compared to an encoded video bitstream and may be processed by an encoder 202 coupled to the video source 201. The encoder 202 may include hardware, software, or a combination thereof to implement or enforce aspects of the embodiments described in more detail below. The encoded video bitstream 204 may be characterized by a lower data volume when compared to the sample stream and may be stored on a streaming server 205 for future use. One or more streaming clients 212 and 207 may access the streaming server 205 to retrieve encoded video bitstreams 208 and 206, which may be copies of the encoded video bitstream 204. Client 212 may include video decoder 211, which decodes an incoming copy of the encoded video bitstream 208 and creates an outgoing video sample stream 210 that can be presented on display 209 or another presentation device. In some streaming systems, the encoded video bitstreams 204, 206, and 208 may be encoded according to certain video encoding / compression standards. Examples of these standards have been mentioned above and are further described herein.

[0027] Figure 3 An example DASH processing model 300 is shown, illustrating an example client architecture for handling DASH and CMAF events. In DASH processing model 300, client requests for media segments (e.g., advertising media segments and live media segments) can be based on addresses described in the manifest. The manifest also describes a metadata track from which the client can access segments, parse segments, and send them to application 301.

[0028] The list includes MPD events or events, and the in-band event and "moof" parser 306 can parse MPD event segments or event segments and append the event segments to the event and metadata buffer 330. The in-band event and "moof" parser 306 can also retrieve media segments and append the media segments to the media buffer 340. The event and metadata buffer 330 can send event and metadata information to the event and metadata synchronizer and scheduler 335. The event and metadata synchronizer and scheduler 335 can schedule specific events to DASH player control, selection, and heuristic logic 302, and schedule application-related event and metadata tracks to the application 301.

[0029] According to some implementations, the MSE may include a pipeline comprising a file format parser 350, a media buffer 340, and a media decoder 345. The MSE 320 is a logical buffer for media segments, where media segments can be tracked and ordered based on their presentation time. Media segments may include, but are not limited to, advertising media segments associated with advertising MPDs and live media segments associated with live MPDs. Each media segment can be added to or appended to the media buffer 340 based on its timestamp offset, and the timestamp offset can be used to order the media segments in the media buffer 340.

[0030] Since embodiments of this application may involve constructing a linear media source extension (MSE) buffer from two or more non-linear media sources using MPD links, and the non-linear media sources may be advertising MPDs and live MPDs, a file format parser 350 can be used to process the different media and / or codecs used by the live media segments included in the live MPD. In some embodiments, the file format parser may issue a change type based on the profile, level, and / or codec of the live media segment.

[0031] As long as a media segment exists in media buffer 340, event and metadata buffer 330 maintains the corresponding event segment and metadata. Example DASH processing model 300 may include a timing metadata track parser 325 for maintaining metadata associated with in-band events and MPD events. Figure 3 The MSE 320 includes a file-only format parser 350, a media buffer 340, and a media decoder 345. The event and metadata buffer 330, as well as the event and metadata synchronizer and scheduler 335, are not local to the MSE 320, thus preventing the MSE 320 from processing events locally and sending events to the application.

[0032] Auxiliary presentation

[0033] In this disclosure, an auxiliary media presentation is defined as a media presentation independent of the main media presentation in the MPD (Multi-Media Display). As an example, an advertising media segment or live media segment independent of the main media segment can be an auxiliary presentation. Updates to any auxiliary media presentation or auxiliary media segment do not affect the main media segment. Similarly, updates to the main media segment do not affect the auxiliary media segment. Therefore, an auxiliary media segment (also referred to as an auxiliary media presentation or auxiliary presentation) can be completely independent of the main media segment (also referred to as the main media presentation and media presentation in this disclosure).

[0034] Assisted MPD

[0035] An MPD is a media presentation description that may include a hierarchically organized media presentation. An MPD may include one or more time-segment sequences, where each time-segment may include one or more adapter sets. Each adapter set in an MPD may include one or more representations, each representation including one or more media segments. These one or more media segments carry the actual media data being encoded, decoded, and / or played, along with associated metadata. A secondary MPD may include one or more secondary media segments, and according to one aspect, each secondary media segment included in a secondary MPD may be made available at the start time of the primary MPD's availability. According to another aspect or the same aspect, a secondary MPD must include @mediaPresentationDuration. The media presentation duration may include the presentation duration of one or more secondary media segments. In some implementations, the media presentation duration may include the total media presentation time of all secondary media segments included in the MPD.

[0036] As described above, embodiments of this disclosure define auxiliary MPDs representing supplementary content independent of the main media content. According to one aspect, the main MPD may include references to at least one auxiliary MPD using auxiliary descriptors, or in some embodiments, to each auxiliary MPD. The auxiliary descriptor may have a specific syntax. As an example, the auxiliary descriptor may include an element called EssentialProperty, or it may include an element called SupplementalProperty, which can describe or identify the auxiliary MPD. The auxiliary descriptor may also include a specific generic resource name of the scheme of the auxiliary descriptor. As an example, a specific urn of the auxiliary descriptor may include an @scheme such as "urn:mpeg:dash:auxiliary:2021". The auxiliary descriptor may also include @values, which may include the auxiliary MPD URL, a start time value, and a return time value.

[0037] According to one aspect, the start time value (also referred to herein as start value, start time attribute, and start only) can indicate the moment when the main media presentation stops and the secondary media presentation begins. In some implementations, if start=0 for the secondary media segment, the secondary media presentation can be a pre-positioned secondary media segment. When the secondary media presentation is a pre-positioned secondary media segment, the secondary media segment can be played for the first time before the main media segment.

[0038] According to one approach, the return time (also referred to as "returnTime") value can be a value representing the offset relative to the moment in the main media timeline when the main media presentation stops to play the secondary MPD or one or more secondary media segments. In some implementations, the default value of the returnTime value is the media presentation duration associated with the secondary MPD. A returnTime value of zero will cause the main media presentation to be added at the moment when the main media presentation stops to play the secondary media segment.

[0039] Figure 4B This is an exemplary script 450 that, according to the instructions of the implementation, allows XML code for auxiliary MPD elements used to play front and center auxiliary media content.

[0040] like Figure 4B As shown, an auxiliary MPD described by an EssentialProperty descriptor with @id="e0" can be a front-facing media segment, while an auxiliary MPD described by EssentialProperty descriptors with @id="e1" and @id="e2" can be a center-facing media segment. In some implementations, the auxiliary MPD can be exposed before one or more time periods.

[0041] An implementation of an auxiliary MPD with an MSE source buffer.

[0042] The W3C MSE is a sequential buffer. As long as the MSE source buffer's TimeStamp Offset (TSO) is set appropriately so that the content to be played sequentially is appended to the correct position, the MSE source buffer will play the content correctly. Depending on the implementation, at each moment when the primary presentation switches to secondary presentation or vice versa, the MSE timest offset (TSO) can be adjusted to place the primary and secondary media segments in the correct positions within the MSE buffer.

[0043] Figure 4A Is using Figure 4B The XML code in Figure 400 describes the MSE source buffer.

[0044] As shown in Figure 400, the auxiliary MPD aux e0 can be a front-mounted MPD, and aux e1 and aux e2 can be center-mounted MPDs. The live content p0 can be a main MPD that includes one or more main media segments.

[0045] Because aux e0 is a pre-emptive MPD, one or more secondary media segments in aux e0 can be retrieved and placed before any primary media segment at the beginning of the MSE source buffer. When a pre-emptive secondary media segment in aux e0 is retrieved and appended, the MSE TSO is zero.

[0046] T0 = ​​e0@PTO Equation (1)

[0047] In some implementations, after acquiring and attaching each pre-support media segment, the MSE TSO can be updated based on the start time of the live media p0, the presentation time of the live media p0, and the time shift buffer depth. As an example, in... Figure 4A In this context, T1 can be based on the depth of the time shift buffer, the duration of the pre-support media segment, and the presentation time offset associated with the live media.

[0048] T1 = Max(timeShiftBufferDepth, Reality Edge – P0Start) + e0@duration +p0@PTO - e0@PTO (Equation 2)

[0049] For the center-mounted auxiliary MPDs aux e1 and aux e2:

[0050] T2 = T1 + e1@start + e1@PTO - p0@PTO Formula (3)

[0051] T3 = T2 + e1@returnTime - p0@PTO – e1@PTO Formula (4)

[0052] T4 = T1 + e2@start + e2@PTO - p0@PTO Equation (5)

[0053] T5 = T4 + p0@PTO - e2@PTO Formula (6)

[0054] Where e0@PTO, e1@PTO, and e2@PTO are the presentation time offsets of the first time period in the auxiliary MPD linked in the base descriptor in this order, and ei@starttime and ei@returnTime are the start time and return time values ​​in each corresponding descriptor with id=ei.

[0055] According to the implementation method, the MSE TSO can be updated based on the following:

[0056] The front-end auxiliary media segment can be started from the MSE attachment window start (0). Since the front-end auxiliary media segment is acquired and attached first, the MSE TSO is set to be equal to the PTO of the auxiliary MPD (e.g., see equation (1)).

[0057] In some implementations, a live edge associated with the live media or main media segment can be calculated after the last segment of the attached pre-support media segment. The MSE TSO can be adjusted using the DVR window size, the live edge, the start and duration of the pre-support MPD or media segment, and the PTO of the live content, such that a threshold time range exists between the first media segment attached to the MSE buffer and the last segment of the previously added pre-support MPD. See Equation (2) as an example. In some implementations, the threshold can be less than, equal to, or greater than the DVR window. The DVR window may include the client's ability to scrub back or start playback at an earlier point in the main media stream.

[0058] According to one implementation, for each additional auxiliary MPD or media segment, its offset relative to the live edge is calculated. The MSE TSO is added to this offset and adjusted based on the live time period PTO and the PTO of the auxiliary MPD / media segment. Alternatively, or in another implementation, at the end of each auxiliary MPD / media segment, the MSE TSO is adjusted based on the return time of the auxiliary MPD / media segment, the live time period PTO, and the PTO of the auxiliary MPD / media segment.

[0059] In some implementations, after playback of the pre-amplified MPD / media segment, the client can decide to remove the pre-amplified MPD / media segment time range from the MSE, so the pre-amplified segment can be played only once. In some implementations, the pre-amplified segment may not play if the DVR is accessed. The client can also remove the time range from the MSE additional window buffer based on the value of timeShiftBufferDepth after joining a live session.

[0060] Figure 5 This is an exemplary stack body presentation 500 that illustrates a stack presentation based on auxiliary MPD.

[0061] like Figure 5 As shown, the presentation of the combination can begin with live content (main media) and may also include one or more independent and auxiliary media segments. Based on aux MPD1 referenced in the main MPD, the presentation of the live content (main media segment) can be stopped, and the auxiliary media segments in aux 1 can be presented instead.

[0062] In some implementations, based on aux MPD2 referenced in aux MPD1, the presentation of the secondary media segment in aux 1 is stopped, and instead, the secondary media segment in aux 2 can be played and / or scheduled. At the end of the secondary media segments in aux 1 and aux 2, and at the end of the secondary media segments in aux 2, the presentation can be switched back to live content. Therefore, when the main media stream is independent of the secondary media stream, the secondary media segment (or presentation) can be supported by the MSE source buffer as a pre-channel or center-channel secondary media.

[0063] Therefore, embodiments of this disclosure can provide methods and apparatus for signaling auxiliary media presentations from a main media presentation defined in an MPD by inserting pre-positioned and center-positioned content in a media presentation. Each auxiliary presentation can be represented by a basic or supplementary descriptor in the MPD, which can include a URL to the auxiliary MPD. Additional constraints may exist including stop and join return times for the main media presentation, which can be defined by a start time value and a returnTime value in the same descriptor. In some embodiments, the auxiliary MPD can be used to play back pre-positioned auxiliary content before the playback of the main content or the center-positioning of the auxiliary content. Constraints may also define whether the main media presentation may need to be time-shifted after center-positioning, or whether center-positioning replaces the main content during the duration of the auxiliary MPD. In some embodiments, MPD periods can be updated independently of the auxiliary MPD, and the MSE and EME can have a single MSE buffer for the playback of the main content and auxiliary content, as the buffer can be set up taking into account the characteristics of the main content.

[0064] Embodiments of this disclosure may also provide methods and apparatus for defining the operation of various segments of supplementary pre-supporting content, mid-supporting content, and main content based on an MSE TSO adjusted according to embodiments of this disclosure, and for acquiring and replaying supporting content and main content using an MSE source buffer.

[0065] Implementations of this disclosure may also involve combining presentations via stacking operations of auxiliary MPDs according to any of the embodiments herein, wherein each primary MPD may have a reference to an auxiliary MPD, and an auxiliary MPD may reference another auxiliary MPD, thereby creating a stack of referenced MPDs that results in stacked presentation.

[0066] Figure 6 This is an exemplary flowchart of a process 600 for signaling auxiliary media, including pre-roll and center-roll media content, within the main media stream.

[0067] At operation 610, one or more main MPDs may be received, comprising one or more main media segments and at least one auxiliary descriptor referencing one or more secondary media presentation descriptions (MPDs). As an example, MSE 320 may receive one or more main MPDs, comprising one or more main media segments and at least one auxiliary descriptor referencing one or more secondary media presentation descriptions (MPDs). In some implementations, the secondary MPDs among the one or more secondary MPDs may include one or more secondary media segments independent of the one or more main media segments. The secondary descriptor may include a start time attribute indicating a first time offset at which the main media segments among the one or more main media segments may be stopped to schedule the secondary media segments among the one or more secondary media segments.

[0068] Auxiliary descriptors may include a return time attribute indicating a second time offset that allows the primary media segment in one or more primary media segments to restart after scheduling one or more auxiliary media segments. In some implementations, the default return time attribute value may be the auxiliary media presentation duration associated with the auxiliary media segment. In some implementations, auxiliary descriptors may include a base attribute descriptor or supplementary attribute descriptor at the MPD level, a uniform resource name (URN) of the scheme associated with at least one auxiliary descriptor, or a uniform resource locator associated with one or more auxiliary MPDs.

[0069] At operation 615, at least one auxiliary descriptor can be used to retrieve one or more auxiliary MPDs referenced in one or more main MPDs. As an example, MSE 320 can use at least one auxiliary descriptor to retrieve one or more auxiliary MPDs referenced in one or more main MPDs. The auxiliary MPDs of one or more auxiliary MPDs may include one or more auxiliary media segments and auxiliary media rendering durations.

[0070] At operation 620, one or more secondary media segments and one or more primary media segments can be attached to a media source extension (MSE) source buffer such as MSE 320.

[0071] In some implementations, the auxiliary media segment can be a preceding auxiliary media segment, based on the fact that the auxiliary media segment in one or more auxiliary media segments has a start time attribute equal to zero, and the preceding auxiliary media segment can be scheduled before one or more main media segments.

[0072] At operation 625, one or more auxiliary media segments and one or more main media segments from the MSE source buffer can be scheduled based on the presentation time offset associated with one or more auxiliary media segments and one or more main media segments. As an example, MSE 320 can schedule one or more auxiliary media segments and one or more main media segments based on timing information from timing metadata track parser 325, event and metadata buffer 330, and event and metadata synchronizer and scheduler 335.

[0073] At operation 630, the MSE source buffer timestamp offset can be updated based on the stopping or restarting of one or more main media segments. As an example, MSE 320 can be updated.

[0074] In some implementations, the MSE source buffer timestamp offset may be equal to the rendering time offset associated with the secondary media segment. In some implementations, the MSE source buffer timestamp offset may be updated based on the depth of the time shift buffer, the start time attribute associated with the secondary media segment, the total duration of all preceding media segments, and the rendering time offset associated with the first primary media segment among one or more primary media segments. In the same or different implementations, the MSE source buffer timestamp offset may be updated to include a threshold time range between the last preceding secondary media segment and the first primary media segment. In some implementations, when scheduling one of the one or more secondary media segments in the MSE source buffer, the MSE source buffer timestamp offset may be updated based on the return time attribute and rendering time offset associated with that one of the one or more secondary media segments.

[0075] Although Figure 6 An example block for processing 600 is shown, but in the implementation, it is different from... Figure 6 Compared to the blocks depicted herein, process 600 may include additional blocks, fewer blocks, different blocks, or blocks arranged differently. In an implementation, any blocks of process 600 may be combined or arranged in any number or order as needed. In an implementation, two or more blocks of process 600 may be executed in parallel.

[0076] The techniques described above can be implemented as computer software using computer-readable instructions and physically stored in one or more computer-readable media, or they can be implemented through one or more specially configured hardware processors. For example, Figure 7 A computer system 700 suitable for implementing various implementation methods is shown.

[0077] Computer software can be coded using any suitable machine code or computer language. The machine code or computer language can be subjected to mechanisms such as assembly, compilation, and linking to create code that includes instructions. These instructions can be executed directly by the computer's central processing unit (CPU), graphics processing unit (GPU), or through decoding, microcode execution, etc.

[0078] The instructions can be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, Internet of Things devices, etc.

[0079] Figure 7 The components shown for computer system 700 are exemplary in nature and are not intended to impose any limitation on the scope or functionality of computer software implementing embodiments of this disclosure. The configuration of the components should also not be construed as having any dependency or requirement relating to any component or combination of components shown in the exemplary embodiments of computer system 700.

[0080] Computer system 700 may include a specific human-machine interface (HMI) input device. Such an HMI input device can respond to input from one or more human users via, for example, tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., voice, clapping), visual input (e.g., gestures), or olfactory input. The HMI device can also be used to capture certain media that are not necessarily directly related to conscious human input, such as audio (e.g., speech, music, ambient sounds), images (e.g., scanned images, photographic images obtained from still image capturing devices), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0081] The input human-machine interface device may include one or more of the following (only one of each is depicted in the figures): keyboard 701, mouse 702, touchpad 703, touch screen 710, joystick 705, microphone 706, scanner 708, and camera device 707.

[0082] Computer system 700 may also include specific human-machine interface (HMI) output devices. Such HMI output devices can stimulate the senses of one or more human users through, for example, tactile output, sound, light, and smell / taste. Such HMI output devices may include: tactile output devices (e.g., tactile feedback via touchscreen 710 or joystick 705, but tactile feedback devices that are not used as input devices may also exist), audio output devices (e.g., speakers 709, headphones), visual output devices (e.g., screens 710 including CRT screens, LCD screens, plasma screens, OLED screens, each screen may or may not have touchscreen input capability, each screen may or may not have tactile feedback capability—some of which may be able to output two-dimensional or higher-dimensional visual outputs through means such as stereoscopic image output; virtual reality glasses, holographic displays, and cigarette canisters), and printers.

[0083] The computer system 700 may also include human-accessible storage devices and their associated media, such as optical media including CD / DVD ROM / RW 720 with media such as CD / DVD 711, thumb drives 722, removable hard disk drives or solid-state drives 723, conventional magnetic media such as magnetic tapes and floppy disks, and devices based on dedicated ROM / ASIC / PLD such as security dongles.

[0084] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the presently disclosed subject matter does not include transmission media, carrier waves, or other transient signals.

[0085] Computer system 700 may also include interfaces 799 to one or more communication networks 798. Networks 798 may be, for example, wireless, wired, or optical. Networks 798 may also be local, wide area, metropolitan area, vehicle and industrial, real-time, latency-tolerant, etc. Examples of networks 798 include: local area networks such as Ethernet; wireless LANs; cellular networks including GSM, 3G, 4G, 5G, LTE, etc.; cable or wireless wide area digital networks including cable television, satellite television, and terrestrial broadcast television; vehicle and industrial networks including CANBus, etc. Some networks 798 typically require external network interface adapters attached to certain general-purpose data ports or peripheral buses (750 and 751) (such as, for example, USB ports of computer system 700); other networks are typically integrated into the core of computer system 700 by attaching to system buses as described below (e.g., Ethernet interfaces to PC computer systems or cellular network interfaces to smartphone computer systems). Using any of these networks 798, computer system 700 can communicate with other entities. Such communication can be one-way (receive-only, e.g., broadcast television), one-way (transmit-only, e.g., to a CANbus device), or bidirectional, e.g., using a local area or wide area digital network to another computer system. Certain protocols and protocol stacks can be used on each of these networks and network interfaces as described above.

[0086] The human-machine interface devices, human-accessible storage devices, and network interfaces mentioned above can be attached to the core 740 of the computer system 700.

[0087] The core 740 may include one or more Central Processing Units (CPUs) 741, Graphics Processing Units (GPUs) 742, graphics adapters 717, dedicated programmable processing units in the form of Field Programmable Gate Areas (FPGAs) 743, hardware accelerators 744 for certain tasks, etc. These devices, along with read-only memory (ROM) 745, random access memory 746, and internal mass storage devices 747 such as internal non-user-accessible hard disk drives (SDs), etc., can be connected via a system bus 748. In some computer systems, the system bus 748 may be accessed as one or more physical connectors to allow for expansion by adding CPUs, GPUs, etc. Peripheral devices may be attached directly or via a peripheral bus 751 to the core's system bus 748. Peripheral bus architectures include PCI, USB, etc.

[0088] The CPU 741, GPU 742, FPGA 743, and accelerator 744 can execute specific instructions, and combinations of these instructions can constitute the aforementioned computer code. This computer code can be stored in ROM 745 or RAM 746. Transient data can also be stored in RAM 746, while permanent data can be stored, for example, in an internal mass storage device 747. Fast storage and retrieval of any memory device in the memory device can be achieved by using cache memories, which can be closely associated with one or more CPUs 741, GPUs 742, mass storage devices 747, ROM 745, RAM 746, etc.

[0089] Computer-readable media may have computer code thereon for performing operations of various computer implementations. The media and computer code may be specifically designed and constructed for the purposes of this disclosure, or they may be of a type known and available to those skilled in the art of computer software.

[0090] By way of example and in a non-limiting manner, a computer system 700 having the illustrated architecture, and in particular core 740, can be functionally provided by a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software implemented in one or more tangible computer-readable media. Such computer-readable media can be media associated with user-accessible mass storage devices as described above, and certain storage devices of core 740 with non-transitory characteristics, such as internal mass storage device 747 or ROM 745. Software implementing various embodiments of this disclosure can be stored in such devices and executed by core 740. Depending on specific needs, the computer-readable medium may include one or more memory devices or chips. The software can cause core 740, and in particular its processor (including a CPU, GPU, FPGA, etc.), to execute specific processes or specific portions of specific processes described herein, including defining data structures stored in RAM 746 and modifying such data structures according to software-defined processes. Alternatively or as an alternative, the computer system may provide functionality through hard-wired logic or otherwise embodied in circuitry (e.g., accelerator 744), which may replace or operate with software to perform a particular process or a particular portion of a particular process described herein. Where appropriate, references to software may include logic, and conversely, references to logic may include software. Where appropriate, references to computer-readable media may include circuitry storing software for execution (e.g., integrated circuit (IC)), circuitry implementing logic for execution, or both. This disclosure includes any suitable combination of hardware and software.

[0091] While this disclosure has described several exemplary embodiments, variations, substitutions, and various alternative equivalents fall within the scope of this disclosure. Therefore, it will be appreciated that those skilled in the art will be able to conceive of many systems and methods that, while not expressly shown or described herein, embody the principles of this disclosure and are thus within its spirit and scope.

Claims

1. A method for indicating secondary media in a main DASH media stream, characterized in that, The method includes: Receive one or more main media presentation descriptions (MPDs) including one or more main DASH media segments and at least one auxiliary descriptor, wherein the at least one auxiliary descriptor includes a Uniform Resource Locator (URL) referencing one or more auxiliary MPDs, wherein the auxiliary MPDs among the one or more auxiliary MPDs include one or more auxiliary media segments independent of the one or more main DASH media segments; Retrieve the one or more auxiliary MPDs referenced by the URL in the one or more main MPDs using the at least one auxiliary descriptor, wherein the auxiliary MPD of the one or more auxiliary MPDs includes the one or more auxiliary media segments and auxiliary media presentation duration; Attach the one or more auxiliary media segments and the one or more main DASH media segments to the Media Source Extension MSE source buffer; and Based on the presentation time offset associated with the one or more auxiliary media segments and the one or more main DASH media segments, the one or more auxiliary media segments and the one or more main DASH media segments are scheduled from the MSE source buffer; When one of the one or more auxiliary media segments in the MSE source buffer is scheduled, the MSE source buffer timestamp offset is updated based on the return time attribute and presentation time offset associated with the one or more auxiliary media segments.

2. The method according to claim 1, characterized in that, The at least one auxiliary descriptor includes: A start time attribute, which indicates a first time offset for stopping the primary DASH media segment among the one or more primary DASH media segments to schedule the secondary media segments among the one or more secondary media segments; and Returns a time attribute that indicates a second time offset for restarting the primary DASH media segment in the one or more primary DASH media segments after scheduling the primary DASH media segment in the one or more secondary media segments.

3. The method according to claim 2, characterized in that, The method further includes updating the MSE source buffer timestamp offset based on stopping or restarting the main DASH media segment in one or more main DASH media segments.

4. The method according to claim 2 or 3, characterized in that, The auxiliary media segment is a pre-emptive auxiliary media segment based on the fact that one or more of the auxiliary media segments have a start time attribute equal to zero.

5. The method according to claim 4, characterized in that, Based on the fact that the auxiliary media segment is the front-end auxiliary media segment, scheduling the one or more auxiliary media segments and the one or more main DASH media segments from the MSE source buffer includes scheduling the front-end auxiliary media segment before the one or more main DASH media segments.

6. The method according to claim 4 or 5, characterized in that, Since the auxiliary media segment is the pre-positioned auxiliary media segment, the MSE source buffer timestamp offset is equal to the presentation time offset associated with the auxiliary media segment.

7. The method according to claim 6, characterized in that, The MSE source buffer timestamp offset is updated based on the depth of the time shift buffer, the start time attribute associated with the auxiliary media segment, the total duration of all preceding media segments, and the presentation time offset associated with the first main DASH media segment of the one or more main DASH media segments. The MSE source buffer timestamp offset is updated to include the threshold time range between the last pre-supplementary media segment and the first primary DASH media segment.

8. The method according to claim 2, characterized in that, The default returned time attribute value is the duration of the auxiliary media presentation associated with the auxiliary media segment.

9. The method according to claim 1, characterized in that, The at least one auxiliary descriptor further includes: MPD-level basic attribute descriptors or supplementary attribute descriptors; The Uniform Resource Name (URN) of the scheme associated with the at least one auxiliary descriptor; and Uniform Resource Locator associated with the auxiliary MPD in the one or more auxiliary MPDs.

10. An apparatus for indicating secondary media in a main DASH media stream, characterized in that, The apparatus is used to perform the method according to any one of claims 1 to 9.

11. An electronic device comprising a memory and a processor, characterized in that: The memory stores computer-readable instructions, which, when executed by the processor, cause the electronic device to perform the method according to any one of claims 1 to 9.

12. A non-transitory computer-readable medium storing instructions, the instructions comprising one or more instructions, characterized in that, When executed by one or more processors, the one or more instructions cause the one or more processors to perform the method according to any one of claims 1 to 9.

Images