Method and apparatus for transmitting auxiliary media streams

Abstract

A method and apparatus are provided for signaling the transmission of a secondary media stream during a primary media stream using a Media Source Extended Entity (MSE) buffer. The method includes appending one or more primary media segments to the MSE source buffer based on one or more corresponding primary timestamp offsets, and appending the one or more secondary media segments to the MSE source buffer based on one or more secondary periodic attributes associated with the one or more secondary media segments. The secondary periodic attributes may be periodic-type elements including a start time attribute indicating a first time offset in which the primary media segment can be stopped during a pre-patch or mid-patch insertion to play the secondary media segment. The media segments and secondary media segments can be scheduled.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 253,046, filed October 6, 2021, and U.S. Patent Application No. 17 / 949,684, filed September 21, 2022, the entire contents of which are expressly incorporated herein by reference. Technical Field

[0003] Embodiments of this disclosure relate to streaming media content, and more specifically to streaming media, advertising, and live content based on Dynamic Adaptive Streaming over HTTP (DASH) according to the Moving Picture Experts Group (MPEG) Hypertext Transfer Protocol. Background Technology

[0004] MPEG DASH provides a standard for streaming media content over IP networks. In MPEG DASH, media presentation descriptions (MPDs) and events are used to transmit events related to the media timeline to the client. The ISO / IEC 23009-1 DASH standard allows for streaming of multi-rate content. The DASH standard provides a single linear timeline, where each period is a continuation of the previous one within a single timeline. ISO / IEC 23009-1 also provides a tool for MPD linking, which sends the URL of the next MPD via signaling for playback within the MPD; this can be used for pre-roll ad insertion.

[0005] MPEG DASH provides a standard for streaming multimedia content over IP networks. While this standard addresses linear playback of media content, it fails to handle non-linear operations, such as media segments associated with different, independent timelines. MPD links and pre-roll ad insertions can be used to overcome these shortcomings. However, even MPD links and pre-roll ad insertions fail when the DASH player uses W3C Media Source Extensions (MSEs) because handling such non-linear playback with a single MSE source buffer is very challenging. Furthermore, MPD links and pre-roll ad insertions cannot be used when ads are expected to play before MPD links or pre-roll elements, or when W3C Encrypted Media Extensions (EMEs) are used to play protected live content.

[0006] Therefore, a method is needed for combining supplementary or independent content that differs from the main media content. Specifically, a method and apparatus are needed to combine supplementary content with the main media content for pre-roll or mid-roll playback. Furthermore, a method is needed for using MPD links to process W3C encrypted media extensions for playing back protected live content. Summary of the Invention

[0007] This disclosure addresses one or more technical problems. It includes methods, processes, apparatus, and non-transitory computer-readable media for implementing a novel concept—an auxiliary period (AuxPeriod). The auxiliary period (also referred to as an auxiliary period attribute) allows for the presentation of auxiliary or independent media presentations based on the presentation of the primary media. Furthermore, embodiments of this disclosure provide support for including pre-roll and mid-roll content during the primary presentation.

[0008] Embodiments of this disclosure may provide a method for signaling the transmission of a secondary media stream during a primary HTTP-based Dynamic Adaptive Streaming (DASH) media stream using a Media Source Extension (MSE) buffer. The method may be executed by at least one processor and may include: appending one or more primary media segments to the MSE source buffer based on one or more corresponding primary timestamp offsets associated with each primary media segment in one or more primary media segments; appending one or more secondary media segments to the MSE source buffer based on one or more secondary periodic attributes associated with one or more secondary media segments, wherein the secondary periodic attributes of the one or more secondary periodic attributes may be periodic type elements that may include a start time attribute indicating a first time offset in which, during a pre-patch or mid-patch insertion, a primary media segment in the one or more primary media segments may be stopped to play a secondary media segment in the one or more secondary media segments; and scheduling the one or more primary media segments and the one or more secondary media segments in the MSE source buffer based on the one or more corresponding primary timestamp offsets and the one or more secondary periodic attributes.

[0009] Embodiments of this disclosure may provide an apparatus for signaling a secondary media stream during a primary HTTP-based Dynamic Adaptive Streaming (DASH) media stream using a Media Source Extension (MSE) buffer. The apparatus may include at least one memory configured to store program code; and at least one processor configured to access the program code and operate according to the instructions of the program code. The program code may include first additional code configured to cause the at least one processor to append the one or more main media segments to the MSE source buffer based on one or more corresponding main timestamp offsets associated with each of the one or more main media segments; second additional code configured to cause the at least one processor to append the one or more auxiliary media segments to the MSE source buffer based on one or more auxiliary periodic attributes associated with one or more auxiliary media segments, wherein the auxiliary periodic attributes of the one or more auxiliary periodic attributes are periodic type elements that may include a start time attribute indicating a first time offset in which, during a pre-patch or mid-patch insertion, the main media segments of the one or more main media segments are stopped to play the auxiliary media segments of the one or more auxiliary media segments; and scheduling code configured to cause the at least one processor to schedule the one or more main media segments and the one or more auxiliary media segments in the MSE source buffer based on the one or more corresponding main timestamp offsets and the one or more auxiliary periodic attributes.

[0010] Embodiments of this disclosure may provide a non-transitory computer-readable medium storing instructions. These instructions may include one or more instructions that, when executed by one or more processors of a device for signaling the transmission of secondary media streams during a primary HTTP-based Dynamic Adaptive Streaming (DASH) media stream using a Media Source Extension (MSE) buffer, cause the one or more processors to append one or more primary media segments to the MSE source buffer based on one or more corresponding primary timestamp offsets associated with each of the one or more primary media segments; append one or more secondary media segments to the MSE source buffer based on one or more secondary periodic attributes associated with the one or more secondary media segments, wherein the secondary periodic attributes are periodic type elements that may include a start time attribute indicating a first time offset in which, during a pre-patch or mid-patch insertion, the primary media segments of the one or more primary media segments are stopped to play the secondary media segments of the one or more secondary media segments; and schedule the one or more primary media segments and the one or more secondary media segments in the MSE source buffer based on the one or more corresponding primary timestamp offsets and the one or more secondary periodic attributes. Attached Figure Description

[0011] Further features, properties, and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings, wherein:

[0012] Figure 1 This is a simplified illustration of a communication system according to an embodiment.

[0013] Figure 2 This is an example illustration of the placement of components in a streaming environment according to an embodiment.

[0014] Figure 3 This is a simplified block diagram of a DASH processing model according to an embodiment.

[0015] Figure 4 This is a simplified illustration of a Media Source Extension (MSE) source buffer implementation for auxiliary media presentation according to an embodiment.

[0016] Figure 5 This is an exemplary script for auxiliary MPD elements used to play pre-roll and mid-roll auxiliary media content, according to an embodiment.

[0017] Figure 6 This is an exemplary flowchart illustrating the use of an MSE buffer to send a secondary media stream via signaling during the primary media stream, according to an embodiment.

[0018] Figure 7 This is a simplified diagram of a computer system according to an embodiment. Detailed Implementation

[0019] The features discussed below can be used individually or in any combination in any order. Furthermore, embodiments can be implemented using processing circuitry (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored in a non-transitory computer-readable medium.

[0020] Figure 1 A simplified block diagram of a communication system 100 according to an embodiment of the present disclosure is shown. The communication system 100 may include at least two terminals 102 and 103 interconnected via a network 105. For unidirectional data transmission, the first terminal 103 may encode video data locally for transmission to the other terminal 102 via the network 105. The second terminal 102 may receive the encoded video data from the other terminal from the network 105, decode the encoded data, and display the recovered video data. Unidirectional data transmission is common in media service applications, etc.

[0021] Figure 1 The illustration shows a second pair of terminals, namely terminals 101 and 104, for supporting bidirectional transmission of encoded video that may occur, for example, during a video conference. For bidirectional data transmission, each of terminals 101 and 104 can encode video data captured at a local location for transmission to the other terminal via network 105. Each of terminals 101 and 104 can also receive encoded video data transmitted by the other terminal, decode the encoded data, and display the recovered video data on a local display device.

[0022] exist Figure 1 In this disclosure, terminals 101, 102, 103, and 104 may be illustrated as servers, personal computers, and smartphones, but the principles of this disclosure are not limited thereto. Embodiments of this disclosure are applicable to laptop computers, tablet computers, media players, and / or dedicated video conferencing equipment. Network 105 represents any number of networks, including, for example, wired and / or wireless communication networks, that transmit encoded video data between terminals 101, 102, 103, and 104. Network 105 may exchange data in circuit-switched and / or packet-switched channels. Representative networks include telecommunications networks, local area networks (LANs), wide area networks (WANs), and / or the Internet. For the purposes of this discussion, the architecture and topology of network 105 may be of little importance to the operation of this disclosure unless explained herein.

[0023] As an example, Figure 2The illustration shows the placement of a video encoder and decoder in a streaming environment. This embodiment can be applied to other video-enabled applications, including, for example, video conferencing, digital television, and storing compressed video on digital media including CDs, DVDs, Memory Sticks, etc.

[0024] The streaming system may include a capture subsystem 203, which may include a video source 201, such as a digital camera that creates, for example, an uncompressed video sample stream 213. The sample stream 213 may be emphasized as having a high data volume compared to an encoded video bitstream; and the sample stream may be processed by an encoder 202 coupled to the video source 201. The encoder 202 may include hardware, software, or a combination thereof to enable or implement aspects of the embodiments described in more detail below. An encoded video bitstream 204, which may be emphasized as having a lower data volume compared to the sample stream, may be stored on a streaming server 205 for future use. One or more streaming clients 212 and 207 may access the streaming server 205 to retrieve encoded video bitstreams 208 and 206, which may be copies of the encoded video bitstream 204. Client 212 may include video decoder 211, which decodes the input copy 208 of the encoded video bitstream and creates an output video sample stream 210 that can be rendered on display 209 or other rendering device. In some streaming systems, the encoded video bitstreams 204, 206, and 208 may be encoded according to certain video encoding / compression standards. Examples of these standards have been mentioned above and are described further herein.

[0025] Figure 3 A sample DASH processing model 300 is shown, such as a sample client architecture for processing DASH events and Common Media Application Format (CMAF) events. In DASH processing model 300, client requests for media segments (e.g., advertising media segments and live media segments) can be based on the addresses described in Listing 303. Listing 303 also describes metadata traces from which the client can access metadata trace segments, parse metadata trace segments, and send metadata trace segments to application 301.

[0026] Listing 303 includes MPD events or events, and the in-band event and "moof" parser 306 can parse MPD event segments or event segments and append the event segments to the event and metadata buffer 330. The in-band event and "moof" parser 306 can also extract media segments and append them to the media buffer 340. The event and metadata buffer 330 can send event and metadata information to the event and metadata synchronizer and scheduler 335. The event and metadata synchronizer and scheduler 335 can schedule specific events to DASH player control, selection, and heuristic logic 302, and schedule application-related event and metadata tracks to the application 301.

[0027] According to some embodiments, the MSE may include a pipeline design that includes a file format parser 350, a media buffer 340, and a media decoder 345. The MSE 320 is one (or more) logical buffers for media segments, where media segments can be tracked and ordered based on their presentation time. Media segments may include, but are not limited to, advertising media segments associated with an advertising MPD and live media segments associated with a live MPD. Each media segment can be added to or appended to the media buffer 340 based on its timestamp offset, and the timestamp offset can be used to order the media segments in the media buffer 340.

[0028] Because embodiments of this application can construct a Linear Media Source Extension (MSE) buffer from two or more non-linear media sources using MPD links, and the non-linear media sources can be advertising MPDs and live MPDs, the file format parser 350 can be used to process different media and / or codecs used by live media segments included in the live MPD. In some embodiments, the file format parser can issue change types based on the codec, profile, and / or level of the live media segment.

[0029] As long as the media segment exists in the media buffer 340, the event and metadata buffers maintain the corresponding event segments and metadata. The sample DASH processing model 300 may include a timing metadata track parser 325 to record metadata associated with in-band events and MPD events. Figure 3 The MSE 320 includes only a file format parser 350, a media buffer 340, and a media decoder 345. The event and metadata buffer 330, the event and metadata synchronizer, and the scheduler 335 are not native to the MSE 320, thus preventing the MSE 320 from processing events locally and sending them to the application.

[0030] Auxiliary presentation

[0031] Embodiments of this disclosure define auxiliary media presentation as a media presentation independent of the main media presentation of the MPD. As an example, an advertising media segment or live media segment independent of the main media segment can be an auxiliary presentation. Updates to any auxiliary media presentation or auxiliary media segment do not affect the main media segment. Similarly, updates to the main media segment do not affect the auxiliary media segment. Therefore, an auxiliary media segment (also referred to as an auxiliary media presentation or auxiliary presentation) can be completely independent of the main media segment (also referred to as the main media presentation and media presentation in this disclosure).

[0032] Auxiliary cycle (also known as auxiliary cycle attribute)

[0033] Figure 4 This is an exemplary script 400 that enables an auxiliary cycle for playing pre-roll and mid-roll supplementary media content according to the instructions of the embodiment.

[0034] like Figure 4 As shown, the AuxPeriod described by the EssentialProperty descriptor with @id="e0" can be a front patch media segment, while the AuxPeriod described by the EssentialProperty descriptors with @id="e1" and @id="e2" can be a middle patch media segment. In some embodiments, the auxiliary period may be exposed before one or more periods in the auxiliary MPD.

[0035] Auxiliary cycles are implemented using the MSE source buffer.

[0036] The W3C MSE is a sequential buffer. As long as the MSE source buffer timestamp offset (TSO) is set appropriately so that the content to be played sequentially is appended to the correct position, the MSE source buffer will play the content correctly. According to one embodiment, at each moment when primary rendering is switched to secondary rendering or switched back to primary rendering, the MSE timestamp offset (TSO) can be adjusted to place the primary and secondary media segments in the correct positions within the MSE buffer.

[0037] Figure 5 Is using Figure 4 The XML code in the diagram 500 describes the MSE source buffer.

[0038] As shown in diagram 500, the auxiliary MPD aux e0 can be a front-mount MPD, while aux e1 and aux e2 can be middle-mount MPDs. The live content p0 can be a main MPD that includes one or more main media segments.

[0039] Since AuxPeriod e0 is the pre-patch auxiliary period, one or more auxiliary media segments from aux e0 can be retrieved and placed at the beginning of the MSE source buffer before any main media segment. When the pre-patch auxiliary media segment is retrieved and appended in aux e0, the MSE TSO is zero.

[0040] Equation (1) is T0 = e0@PTO

[0041] In some embodiments, after extracting and appending each prepatch auxiliary media segment, the MSE TSO can be updated based on the start time of the live media p0, the presentation time of the live media p0, and the depth of the time-shift buffer. As an example, in Figure 5 In this context, T1 can be based on the depth of the time-shift buffer, the duration of the preceding supplementary media segment, and the presentation time offset associated with the live media.

[0042] T1=Max(timeShiftBuffeerDepth,Live edge-P0Start)+e0@duration+p0@PTO-e0@PTO Equation (2)

[0043] For the mid-mount auxiliary MPD aux e1 and aux e2:

[0044] Equation (3) is: T2 = T1 + e1@start + e1@PTO - p0@PTO

[0045] T3 = T2 + e1@returnTime - p0@PTO - e1@PTO Equation (4)

[0046] Equation (5) is: T4 = T1 + e2@start + e2@PTO - p0@PTO

[0047] Equation (6) is T5 = T4 + p0@PTO - e2@PTO

[0048] Where e0@PTO, e1@PTO, and e2@PTO are the rendering time offsets of the first cycle in the auxiliary cycles linked in the basic descriptor in this order, and ei@starttime and ei@returnTime are the start time and return time values ​​in each corresponding descriptor with id=ei.

[0049] According to an embodiment, the MSE TSO can be updated based on the following:

[0050] The front patch auxiliary cycle media segment can begin from the MSE append window (0). Since the front patch auxiliary cycle media segment is extracted and appended first, the MSE TSO is set to be equal to the PTO of the auxiliary MPD (e.g., see equation (1)).

[0051] In some embodiments, a live edge associated with the live media or main media segment can be calculated after the last segment of the pre-attach supplementary media segment. Using the DVR window size, the live edge, the start and duration of the pre-attach supplementary period or media segment, and the PTO of the live content, the MSE TSO can be adjusted such that a threshold time range exists between the first media segment attached to the MSE buffer and the last segment of the previously added pre-attach supplementary period. See, for example, equation (2). In some embodiments, the threshold can be less than, equal to, or greater than the DVR window. The DVR window may include the ability for the client to erase or start playback at an earlier point in the main media stream.

[0052] According to one embodiment, for each additional auxiliary period or media segment, its offset to the live edge is calculated. This offset is added to the MSE TSO, and the MSE TSO is adjusted based on the live period PTO and the PTO of the auxiliary period / media segment. Alternatively, in another or similar embodiment, at the end of each auxiliary period / media segment, the MSE TSO is adjusted based on the return time of the auxiliary period / media segment, the live period PTO, and the PTO of the auxiliary period / media segment.

[0053] In some embodiments, when playing back the pre-roll auxiliary period / media segment, the client may decide to remove the pre-roll auxiliary period / media segment time range from the MSE, so the pre-roll may only be played once. In some embodiments, if the DVR is accessed, the pre-roll may not be played. After joining a live session, the client may also remove the time range from the MSE additional window buffer based on the value of the timeshift buffer's depth, timeShiftBufferDepth.

[0054] The advantages of the secondary cycle defined in this article include correctly using the cycle to initialize the MSE / EME with a single MPD, ensuring playback of the main content via the MSE / EME. This single MSE / EME initialization supports seamless content playback during transitions between primary and secondary presentations.

[0055] Since all AuxPeriods are subsets of MPDs, if some periods are removed during an MPD update, the AuxPeriods remain unchanged (useful for pre-roll supplementary content). Furthermore, because supplementary periods can be remote elements, this remote element capability allows for late binding on the client-side mid-roll during server-side supplementary ad insertion.

[0056] Figure 6 This is an exemplary flowchart of a process 600 for sending auxiliary media, including pre-roll media content and mid-roll media content, via signaling in a main media stream.

[0057] At operation 610, the one or more master media segments can be appended to the MSE source buffer based on one or more corresponding master timestamp offsets associated with each of the one or more master media segments.

[0058] At operation 615, one or more auxiliary media segments can be appended to the MSE source buffer based on one or more auxiliary cycles associated with one or more auxiliary media segments. An auxiliary cycle can be a cycle-type element that includes a start time attribute. The start time attribute can indicate a first time offset in which the main media segment of the one or more main media segments can be stopped to play the auxiliary media segment of the one or more auxiliary media segments. In some embodiments, the auxiliary cycle can include a return time attribute. The return time attribute can indicate a second time offset in which the main media segment of the one or more main media segments can be restarted after stopping the main media segment of the one or more main media segments to play the auxiliary media segment of the one or more auxiliary media segments. In some embodiments, the one or more auxiliary cycles can be placed before one or more cycles in the Auxiliary Media Presentation Description (MPD).

[0059] At operation 620, the one or more primary media segments and the one or more secondary media segments in the MSE source buffer can be scheduled based on the one or more corresponding primary timestamp offsets and the one or more secondary periods. In some embodiments, the secondary media segment among the one or more secondary media segments may be a pre-roll media segment to be played before the one or more primary media segments, based on a start time attribute of zero. In the same or different embodiments, the MSE source buffer timestamp offset associated with the secondary media segment may be set or initialized to be equal to the rendering time offset associated with the secondary media segment.

[0060] In some embodiments, the MSE source buffer timestamp offset can be updated based on stopping or restarting the primary media segment. At operation 625, the MSE source buffer timestamp offset can be updated based on the depth of the time-shift buffer, the start time attribute, the total duration of all preceding tile media segments, and the rendering time offset associated with the first primary media segment of the one or more primary media segments. In some embodiments, the update can be based on the last preceding tile auxiliary media segment appended to the MSE source buffer. The updated MSE source buffer timestamp offset may include a threshold time range between the last preceding tile auxiliary media segment and the first primary media segment.

[0061] In some embodiments, when scheduling one of the one or more secondary media segments in the MSE source buffer, the MSE source buffer timestamp offset can be updated based on the return time attribute and the presentation time offset associated with the one of the one or more secondary media segments.

[0062] In the same or another embodiment, the MSE source buffer timestamp offset associated with the secondary media segment may be equal to the rendering time offset associated with the secondary media segment, wherein the secondary media segment is a front-mount secondary media segment based on the one or more secondary media segments.

[0063] although Figure 6 An example block of process 600 is shown, but in embodiments, process 600 may include... Figure 6 Compared to the blocks depicted in the diagram, there may be additional blocks, fewer blocks, different blocks, or blocks with different arrangements. In an embodiment, any block of process 600 can be combined or arranged in any number or order as needed. In an embodiment, two or more blocks of process 600 can be executed in parallel.

[0064] The above-described technologies can be implemented as computer software using computer-readable instructions and physically stored in one or more computer-readable media, or implemented by one or more specifically configured hardware processors. For example, Figure 7 A computer system 700 suitable for implementing various embodiments is shown.

[0065] Computer software can be coded using any suitable machine code or computer language, which can be assembled, compiled, linked or similar mechanisms to create code containing instructions that can be executed directly by a computer's central processing unit (CPU), graphics processing unit (GPU), or through interpretation, microcode execution, etc.

[0066] These instructions can be executed on various types of computers or their components, including, for example, personal computers, tablets, servers, smartphones, gaming devices, and Internet of Things (IoT) devices.

[0067] Figure 7 The components shown for computer system 700 are exemplary in nature and are not intended to impose any limitation on the scope or functionality of computer software implementing embodiments of this disclosure. The configuration of the components should also not be construed as having any dependency or requirement relating to any one or a combination of the components illustrated in the exemplary embodiments of computer system 700.

[0068] Computer system 700 may include certain human-computer interface (HCI) input devices. Such HCI input devices may respond to input from one or more human users through, for example, tactile input (such as keystrokes, swipes, data glove movements), audio input (such as voice, clapping), visual input (such as gestures), and olfactory input. HCI devices may also be used to capture certain media that are not necessarily directly related to conscious human input, such as audio (such as voice, music, ambient sounds), images (such as scanned images, photographic images obtained from still image cameras), and video (such as two-dimensional video, three-dimensional video including stereoscopic video).

[0069] The input human-machine interface device may include one or more of the following (only one of each is depicted in the figure): keyboard 701, mouse 702, touchpad 703, touch screen 710, joystick 705, microphone 706, scanner 708, and camera 707.

[0070] The computer system 700 may also include certain human-machine interface (HMI) output devices. Such HMI output devices can stimulate the senses of one or more human users through, for example, tactile output, sound, light, and smell / taste. These HMI output devices may include tactile output devices (e.g., tactile feedback via a touchscreen 710 or joystick 705, but tactile feedback devices not used as input devices may also exist), audio output devices (such as speakers 709, headphones), visual output devices (such as screens 710, including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touchscreen input capability, each with or without tactile feedback capability—some of which are capable of outputting two-dimensional or more than three-dimensional visual output via devices such as stereoscopic outputs, virtual reality glasses, holographic displays, and smoke canisters), and printers.

[0071] The computer system 700 may also include human-accessible storage devices and their associated media, such as optical media including CD / DVD ROM / RW 720 with CD / DVD 711 or similar media, thumb drives 722, removable hard disk drives or solid-state drives 723, conventional magnetic media such as magnetic tapes and floppy disks, devices based on dedicated ROM / ASIC / PLD such as security dongles, etc.

[0072] Those skilled in the art will also understand that the term "computer-readable medium" as used in connection with the subject matter of this disclosure does not include transmission media, carrier waves, or other transient signals.

[0073] Computer system 700 may also include an interface 799 to one or more communication networks 798. Network 798 may be, for example, wireless, wired, or optical. Network 798 may also be local, wide area, metropolitan, vehicular and industrial, real-time, or latency-tolerant, etc. Examples of network 798 include local area networks such as Ethernet, wireless LAN, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., TV wired or wireless wide area digital networks including cable television, satellite television, and terrestrial broadcast television, vehicular and industrial networks including CANBus, etc. Some networks 798 typically require an external network interface adapter (such as, for example, a USB port of computer system 700) attached to some general-purpose data port or peripheral bus (750 and 751); others are typically integrated into the core of computer system 700 by being attached to the system bus, as described below (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks 798, computer system 700 can communicate with other entities. This communication can be one-way (receive-only, such as broadcast television), one-way (transmit-only, such as CANbus to certain CANbus devices), or two-way (such as to other computer systems using local or wide area digital networks). As mentioned above, certain protocols and protocol stacks can be used on each of these networks and network interfaces.

[0074] The aforementioned human-machine interface device, human-accessible storage device, and network interface can be attached to the kernel 740 of the computer system 700.

[0075] The core 740 may include one or more central processing units (CPUs) 741, graphics processing units (GPUs) 742, graphics adapters 717, dedicated programmable processing units 743 in the form of field-programmable gate arrays (FPGAs), hardware accelerators 744 for certain tasks, etc. These devices, along with read-only memory (ROM) 745, random access memory 746, and internal mass storage 747 such as internal non-user-accessible hard disk drives (HDDs), SSDs, etc., can be connected via a system bus 748. In some computer systems, the system bus 748 may be accessed as one or more physical connectors to allow expansion via additional CPUs, GPUs, etc. Peripheral devices may be directly attached to or connected to the core's system bus 748 via a peripheral bus 751. Peripheral bus architectures include PCI, USB, etc.

[0076] The CPU 741, GPU 742, FPGA 743, and accelerator 744 can execute certain instructions, which, when combined, constitute the aforementioned computer code. This computer code can be stored in ROM 745 or RAM 746. Transient data can also be stored in RAM 746, while permanent data can be stored, for example, in internal mass storage 747. Fast storage and retrieval of any memory device can be achieved by using a cache memory, which can be closely associated with one or more CPUs 741, GPUs 742, mass storage 747, ROM 745, RAM 746, etc.

[0077] Computer-readable media may contain computer code for performing operations of various computer implementations. The media and computer code may be specifically designed and constructed for the purposes of this disclosure, or they may be of a type well known and available to those skilled in the art of computer software.

[0078] By way of example and not limitation, a computer system 700 having the illustrated architecture, particularly a core 740, can provide functionality as a result of one or more processors (including CPUs, GPUs, FPGAs, accelerators, etc.) executing software embodied in one or more tangible computer-readable media. Such a computer-readable medium can be a medium associated with user-accessible mass storage as described above, and with certain memories of the core 740 having non-transitory characteristics (such as internal mass storage 747 or ROM 745). Software implementing various embodiments of this disclosure can be stored in such a device and executed by the core 740. Depending on specific needs, the computer-readable medium may include one or more memory devices or chips. The software can cause the core 740, particularly the processors therein (including CPUs, GPUs, FPGAs, etc.), to execute a specific process or a specific portion of a specific process described herein, including defining data structures stored in RAM 746 and modifying such data structures according to the software-defined process. In addition to or as an alternative, the computer system may provide functionality as a result of hard-wired logic or otherwise embodied in circuitry (e.g., accelerator 744), which may replace or operate with software to perform the specific process or a specific portion of the specific process described herein. Where appropriate, references to software may include logic, and vice versa. Where appropriate, references to computer-readable media may include circuitry (such as integrated circuits (ICs)) storing software for execution, circuitry embodying logic for execution, or both. This disclosure includes any suitable combination of hardware and software.

[0079] While several exemplary embodiments have been described in this disclosure, variations, arrangements, and various alternative equivalents fall within the scope of this disclosure. Therefore, those skilled in the art will be able to design numerous systems and methods that, while not expressly shown or described herein, embody the principles of this disclosure and thus also fall within its spirit and scope.

Claims

1. A method for signaling an auxiliary media stream during a primary Hypertext Transfer Protocol (HTTP)-based dynamically adaptive streaming (DASH) media stream using a Media Source Extended Streaming (MSE) buffer, the method being executed by at least one processor, characterized in that, The method includes: The one or more main media segments are appended to the MSE source buffer based on one or more corresponding main timestamp offsets associated with each of the one or more main media segments; The one or more auxiliary media segments are appended to the MSE source buffer based on one or more auxiliary periodic attributes associated with one or more auxiliary media segments, wherein the auxiliary periodic attribute among the one or more auxiliary periodic attributes is a periodic type element including a start time attribute, the start time attribute indicating a first time offset in which a main media segment among the one or more main media segments is stopped during a pre-patch or mid-patch insertion to play an auxiliary media segment among the one or more auxiliary media segments, and based on the start time attribute being zero, the auxiliary media segment among the one or more auxiliary media segments is a pre-patch media segment to be played before the one or more main media segments; and The one or more primary media segments and the one or more secondary media segments in the MSE source buffer are scheduled based on the one or more corresponding primary timestamp offsets and the one or more secondary periodic attributes. Based on the last pre-patch auxiliary media segment attached to the MSE source buffer, the MSE source buffer timestamp offset is updated based on the depth of the time-shift buffer, the start time attribute, the total duration of all pre-patch media segments, and the rendering time offset associated with the first main media segment of the one or more main media segments, wherein the MSE source buffer timestamp offset is updated to include a threshold time range between the last pre-patch auxiliary media segment and the first main media segment.

2. The method according to claim 1, characterized in that, The auxiliary cycle attribute further includes a return time attribute, wherein the return time attribute indicates a second time offset in which the main media segment of the one or more main media segments is restarted after the main media segment of the one or more main media segments is stopped to play the auxiliary media segment of the one or more auxiliary media segments.

3. The method according to claim 2, characterized in that, The MSE source buffer timestamp offset is updated based on stopping or restarting the main media segment.

4. The method according to claim 1, characterized in that, The one or more auxiliary cycle attributes are placed before one or more cycles in the auxiliary media presentation description MPD.

5. The method according to claim 1, characterized in that, Since the auxiliary media segment in one or more of the auxiliary media segments is a front-mount auxiliary media segment, the MSE source buffer timestamp offset associated with the auxiliary media segment is equal to the rendering time offset associated with the auxiliary media segment.

6. The method according to claim 5, characterized in that, The method further includes: When scheduling one of the one or more auxiliary media segments in the MSE source buffer, the MSE source buffer timestamp offset is updated based on the return time attribute and the presentation time offset associated with the one of the one or more auxiliary media segments.

7. An apparatus for signaling an auxiliary media stream during a primary Hypertext Transfer Protocol (HTTP) based dynamically adaptive streaming (DASH) media stream using a Media Source Extended Stream (MSE) buffer, characterized in that, The device includes: At least one memory is configured to store program code; At least one processor is configured to access the program code and operate in accordance with the instructions of the program code to perform the method of any one of claims 1 to 6.

8. A non-transitory computer-readable medium, characterized in that, The device stores instructions, the instructions including: one or more instructions that, when executed by one or more processors of a device for sending an auxiliary media stream by signaling during a primary Hypertext Transfer Protocol HTTP-based Dynamic Adaptive Streaming (DASH) media stream using a Media Source Extended MSE buffer, cause the one or more processors to perform the method of any one of claims 1 to 6.

9. An apparatus for signaling an auxiliary media stream during a primary Hypertext Transfer Protocol (HTTP) based dynamically adaptive streaming (DASH) media stream using a Media Source Extended Stream (MSE) buffer, characterized in that, The device includes: The first additional module is configured to attach the one or more main media segments to the MSE source buffer based on one or more corresponding main timestamp offsets associated with each of the one or more main media segments. A second additional module is configured to attach the one or more auxiliary media segments to the MSE source buffer based on one or more auxiliary periodic attributes associated with one or more auxiliary media segments, wherein the auxiliary periodic attribute among the one or more auxiliary periodic attributes is a periodic type element including a start time attribute, the start time attribute indicating a first time offset in which a main media segment among the one or more main media segments is stopped during a pre-patch or mid-patch insertion to play an auxiliary media segment among the one or more auxiliary media segments, and based on the start time attribute being zero, the auxiliary media segment among the one or more auxiliary media segments is a pre-patch media segment to be played before the one or more main media segments; and The scheduling module is configured to schedule the one or more primary media segments and the one or more secondary media segments in the MSE source buffer based on the one or more corresponding primary timestamp offsets and the one or more secondary periodic attributes; The update module is configured to update the MSE source buffer timestamp offset based on the depth of the time-shift buffer, the start time attribute, the total duration of all pre-patch media segments, and the rendering time offset associated with the first main media segment of the one or more main media segments, based on the last pre-patch auxiliary media segment attached to the MSE source buffer. The MSE source buffer timestamp offset is updated to include a threshold time range between the last pre-patch auxiliary media segment and the first main media segment.

Images