Method, device, storage medium and computer equipment for processing events in media stream

Abstract

The embodiments of the present application provide a method, apparatus, storage medium and computer equipment for processing events in a media stream. The method includes: obtaining media data; generating at least one event segment from the media data; appending the at least one event segment to a first event processing buffer, wherein the at least one event segment includes an event start time offset of each event segment in the at least one event segment, wherein the event start time offset is determined based on the time when the event segment is appended to the first event processing buffer; appending the at least one event segment to a second event processing buffer, wherein the at least one event segment also includes event dispatching information of each event segment in the at least one event segment; and dispatching the at least one event segment based on the event dispatching information in the first event processing buffer and the second event processing buffer.

Description

[0001] Incorporation by Reference

[0002] This application claims priority to U.S. Provisional Application No. 63 / 176,748, filed on April 19, 2019, and priority to U.S. Application No. 17 / 495,299, filed on October 6, 2021, and the entire contents of which are incorporated by reference into this application. Technical Field

[0003] Embodiments of the present application relate to Internet transmission technology, and more particularly, to a method, apparatus, storage medium, and computer equipment for processing events in a media stream. Background Art

[0004] The Moving Picture Experts Group (MPEG) Hypertext Transfer Protocol (DASH) provides a standard for streaming media content over IP networks. In MPEG DASH, media presentation descriptions (MPD) and inband events are used to deliver events related to the media timeline to the client. DASH provides a mechanism for media segments to carry inband events. Similarly, the Common Media Application Format (CMAF) provides a mechanism for CMAF blocks to carry inband events. A popular DASH event is the inband MPD validity expiration event. Other events include application events such as SCTE-35 ("Digital Program Insertion Hint Message for Cable") events.

[0005] Streaming media players, including DASH players, use Media Source Extensions (MSE), which allow browsers or user agents to process media segments. However, the current MSE specification does not support parsing and processing in-band events embedded in DASH or CMAF media segments. Summary of the invention

[0006] The embodiments of the present application provide a method, apparatus, storage medium, and computer device for processing events in a media stream, so that a browser or user agent using MSE can natively process DASH and CMAF in-band events and send them to corresponding applications.

[0007] The present application provides a method for processing an event in a media stream, including:

[0008] Get media data;

[0009] Generating at least one event segment from the media data;

[0010] appending the at least one event segment to a first event processing buffer, the at least one event segment including an event start time offset for each event segment of the at least one event segment, the event start time offset being determined based on a time at which the event segment is appended to the first event processing buffer;

[0011] appending the at least one event fragment to a second event processing buffer, the at least one event fragment further comprising event dispatch information for each event fragment in the at least one event fragment; and,

[0012] The at least one event fragment is dispatched based on the event dispatching information in the first event processing buffer and the second event processing buffer.

[0013] The embodiment of the present application also provides a device for processing events in a media stream, including:

[0014] A first acquisition module, used to obtain media data;

[0015] A first generating module, configured to generate at least one event segment from the media data;

[0016] A first appending module, configured to append the at least one event segment to a first event processing buffer, the at least one event segment including an event start time offset of each event segment in the at least one event segment, the event start time offset being determined based on a time when the event segment is appended to the first event processing buffer;

[0017] A second appending module is configured to append the at least one event fragment to a second event processing buffer, wherein the at least one event fragment further includes event dispatching information of each event fragment in the at least one event fragment; and

[0018] The first dispatching module is used to dispatch the at least one event fragment based on the event dispatching information in the first event processing buffer and the second event processing buffer.

[0019] An embodiment of the present application also provides a computer device, including a processor and a memory, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the above method.

[0020] The embodiment of the present application also provides a non-temporary computer-readable medium on which instructions are stored. When the instructions are executed by a computer to perform video decoding, the computer executes the above method.

[0021] The embodiment of the present application also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device performs the above method.

[0022] It can be seen from the above technical solution that the method provided by the embodiment of the present invention uses the event start time offset of the event segment, which avoids the need for the application to compare the common time reference when determining the relative start time of the event segment, thereby increasing the application processing efficiency and reducing network overhead (such as bandwidth requirements), server computing overhead and application computing overhead. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Other features, properties and various advantages of the disclosed subject matter will become further apparent from the following detailed description and accompanying drawings, in which:

[0024] Figure 1 A simplified schematic diagram of a communication system according to an embodiment of the present application is shown;

[0025] Figure 2 A schematic diagram showing the placement of components in a flow environment according to an embodiment of the present application is shown;

[0026] Figure 3 A simplified block diagram of a DASH processing model according to an embodiment of the present application is shown;

[0027] Figure 4 A simplified block diagram of a DASH processing model according to another embodiment of the present application is shown;

[0028] Figure 5 A simplified block diagram of a media buffer and an event buffer according to an embodiment of the present application is shown;

[0029] Figure 6 A simplified block diagram of a media buffer and an event buffer according to another embodiment of the present application is shown;

[0030] Figure 7 A simplified block diagram of a media buffer and an event buffer according to another embodiment of the present application is shown;

[0031] Fig. 8A A simplified schematic diagram of a process for processing DASH and CAMF in-band events according to an embodiment of the present application is shown;

[0032] Figure 8B A simplified schematic diagram of a process for processing DASH and CAMF in-band events according to another embodiment of the present application is shown;

[0033] Fig. 9 A simplified schematic diagram of a process for processing DASH and CAMF in-band events according to another embodiment of the present application is shown;

[0034] Fig.10 A schematic diagram of a computer device according to an embodiment is shown. DETAILED DESCRIPTION

[0035] The features proposed in the present application discussed below can be used alone or in any order. Further, the embodiments of the present application can be implemented by a processing circuit (e.g., at least one processor, or at least one integrated circuit). In one example, at least one processor executes a program stored in a non-volatile computer-readable medium.

[0036] Figure 1 1 is a simplified block diagram of a communication system 1000 according to an embodiment disclosed in the present application. The communication system 1000 includes a first terminal 102 and a second terminal 103 interconnected by a network 105. For unidirectional data transmission, the first terminal 103 may encode video data at a local location for transmission to the other terminal 102 via the network 105. The second terminal 102 may receive the encoded video data of the other terminal from the network 105, decode the encoded video data to restore the video data, and display a video picture based on the restored video data. Unidirectional data transmission is more common in applications such as media services.

[0037] Figure 1 Another pair of terminals 101 and 104 is shown to support bidirectional transmission of encoded video data, such as during a video conference. For bidirectional data transmission, each terminal 101 and / or 104 can encode locally acquired video data for transmission to the other terminal via a network 105. Each terminal 101 and / or 104 can also receive encoded video data transmitted by another terminal, and can decode the encoded video data to recover the video data, and can display a video picture on an accessible display device based on the recovered video data.

[0038] exist Figure 1In the embodiment, terminals 101, 102, 103 and / or 104 may be servers, personal computers and smart phones, but the principles disclosed in the present application may not be limited thereto. The embodiments disclosed in the present application are applicable to laptop computers, tablet computers, media players and / or dedicated video conferencing equipment. Network 105 represents any number of networks that transmit encoded video data between terminals 101, 102, 103 and / or 104, including, for example, wired and / or wireless communication networks. Communication network 105 may exchange data in circuit switching and / or packet switching channels. The network may include a telecommunications network, a local area network, a wide area network and / or the Internet. For the purposes of the present application, unless explained below, the architecture and topology of network 105 may be irrelevant to the operations disclosed in the present application.

[0039] As an example, Figure 2 The video encoder and video decoder are shown in a streaming environment. The subject matter disclosed in this application is equally applicable to other video-enabled applications, including, for example, video conferencing, digital TV, storing compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0040] The streaming system may include an acquisition subsystem 203, which may include a video source 201 such as a digital camera, which creates, for example, an uncompressed video sample stream 213. The video sample stream 213 is emphasized as a high-data-volume video picture stream compared to an encoded video bitstream, and may be processed by an encoder 202 coupled to the video source 201. The encoder 202 may include hardware, software, or a combination of hardware and software to implement or implement various aspects of the disclosed subject matter as described in more detail below. The encoded video bitstream 204 is emphasized as a lower-data-volume compared to the sample stream, and may be stored on a streaming server 205 for future use. At least one streaming client 212 and 207 may access the streaming server 205 to retrieve the encoded video bitstream 208 and then 206, both of which are copies of the encoded video bitstream 204. The client 212 includes a video decoder 211. The video decoder 211 decodes the incoming copy of the encoded video stream 208 and produces an output video sample stream 210 that can be presented on a display 209 or another presentation device. In some streaming systems, the encoded video streams 204, 206, 208 may be encoded according to certain video encoding / compression standards. Examples of such standards are those noted above and are further described herein.

[0041] The 5G Media Streaming (5GMS) system is a collection of application functions, application servers, and interfaces that are interconnected with a 5G Media Streaming architecture that supports downlink media streaming services, or uplink media streaming services, or both. A 5GMS application provider may include a party that interacts with the functions of the 5GMS system and provides 5GMS-aware applications that interact with the functions of the 5GMS system. A 5GMS-aware application may refer to an application in a user equipment (UE) provided by a 5GMS application provider, which contains the service logic of the 5GMS application service and interacts with other 5GMS clients and network functions via interfaces and application programming interfaces (APIs) defined in the 5GMS architecture. A 5GMS client may refer to a UE function, including a 5GMS downlink (5GMSd) client or a 5GMS uplink (5GMSu) client or both.

[0042] 5GMSd client may refer to a UE function that includes at least a 5G media streamer and a media session processor for downlink streaming and can be accessed through a well-defined interface / API. 5GMSu client may refer to the initiator of a 5GMSu service, where 5GMSu is accessed through a well-defined interface / API. 5GMSu streamer may refer to a UE function that enables uplink delivery of streaming media content to the application server (AS) function of a 5GMS application provider and is able to interact not only with 5GMSu-aware applications for media capture and subsequent streaming, but also with the media session processor for media session control.

[0043] Dynamic policy refers to dynamic policy and charging control (PCC) rules for application flows in uplink or downlink during a media session. An egest session refers to an uplink media flow session from a 5GMS AS to a 5GMSu application provider. An ingest session refers to a session that uploads media content to a 5GMSd AS. A policy template refers to a set of (semi-static) policy or control function (PCF) / network exposure function (NEF) API parameters, where the NEF API parameters are specific to the 5GMS application provider and the resulting PCC rules. A policy template ID can identify the desired policy template, which is used by the 5GMSd application function (AF) to select the appropriate PCF / NEF API for the 5G system so that the PCF can compile the desired PCC rules. A media player entry refers to a document or an indicator pointing to a document that defines a media presentation (e.g., a media presentation description (MPD) for DASH or a uniform resource locator (URL) for a video clip file). A Media Streamer Entry is an indicator (eg, in the form of a URL) that defines the entry point of an uplink media streaming session. A Presentation Entry is a document or an indicator pointing to a document that is used to define an application presentation, such as an HTML5 document.

[0044] Provisioning Session refers to a data structure provided at an interface (M1d) by a 5GMSd application provider that configures 5GMSd features associated with a set of 5GMSd-aware applications. 5GMSd Media Player refers to a UE function that enables playback and presentation of media presentations based on media playback entries, and exposes some basic controls such as play, pause, search, stop, etc. to 5GMSd-aware applications. Server Access Information refers to a set of parameters and addresses (including 5GMSd AF and 5GMSd AS addresses) required to activate a receiving streaming session. Service and Content Discovery refers to functions and procedures provided by a 5GMSd application provider to a 5GMS-aware application that enables an end user to discover available streaming services and content settings, and select specific services or content items to access. Service Announcement refers to a procedure executed between a 5GMS-aware application and a 5GMS application provider that enables a 5GMS-aware application to obtain 5GMS service access information directly or in the form of reference information.

[0045] A third-party player is an application that uses the API to exercise selected 5GMSd functions to play back media content. A third-party uplink streaming player is an application that uses the API to exercise selected 5GMSu functions to capture and stream media content.

[0046] Figure 3 A sample DASH processing model 300 is shown, such as a sample client architecture for processing DASH and CMAF events. In the DASH processing model 300, the client's request for media segments can be based on the addresses described in the manifest parser 303. The manifest parser 303 also describes the metadata track from which the client can access the segments of the metadata track, parse these segments, and send them to the application 301.

[0047] The manifest parser 303 includes MPD events or in-band events, and the in-band event and 'moof' parser 306 can parse the MPD event fragments or in-band event fragments and append the event fragments to the event and metadata buffer 330. The in-band event and 'moof' parser 306 can also extract media fragments and append them to the media buffer 340. The event and metadata buffer 330 can send event and metadata information to the event and metadata synchronizer and dispatcher 335. The event and metadata synchronizer and dispatcher 335 can dispatch specific events to the DASH player control, selection and heuristic logic (HeuristicLogic) 302, and dispatch application-related events and metadata tracks to the application 301.

[0048] According to some embodiments, the MSE may include a media buffer 340 and a media decoder 345. The MSE 320 is one or more logical buffers of media segments that can confirm and sort the media segments based on the presentation time of the media segments. Each media segment can be added or appended to the media buffer 340 based on the timestamp offset of the media segment, and the timestamp offset can be used to sort the media segments in the media buffer 340.

[0049] As long as the media segments exist in the media buffer 340, the event and metadata buffer 330 maintains the corresponding event segments and metadata. Figure 3 , MSE 320 only includes a media buffer 340 and a media decoder 345. Event and metadata buffer 330 and event and metadata synchronizer and dispatcher 335 are not native to MSE 320, prohibiting MSE 320 from natively processing events and sending them to applications.

[0050] Figure 4A sample DASH processing model 400, such as a sample client architecture for processing DASH and CMAF events, is shown. In the DASH processing model 400, client requests for media segments can be based on addresses described in a manifest 403. The manifest 403 also describes a metadata track from which the client can access segments of the metadata track, parse the segments, and send them to the application 401.

[0051] The manifest 403 includes MPD events or in-band events, and the in-band event and 'moof' parser 406 can parse the MPD event segments or in-band event segments and append the event segments to the event purge buffer 430. Based on the media data or media stream and the information in the manifest 403, the in-band event and 'moof' parser 406 can extract the media segments and append them to the media buffer 440. The event purge buffer 430 can send the event and metadata information to the event synchronizer and dispatcher 435. The event synchronizer and dispatcher 435 can dispatch specific events to the DASH player control, selection and heuristic logic 402, and dispatch the event and metadata tracks related to the application to the application 401.

[0052] According to an exemplary embodiment, Figure 4 In the embodiment, the MSE is extended to include: a media buffer 340, a media decoder 345, an event clearing buffer 430, an event synchronizer and dispatcher 435, and an already-dispatched event table 450. The already-dispatched event table 450 can record events that have been dispatched to the application 401. The extension of the event clearing buffer 430 and the event synchronizer and dispatcher 435 enables the MSE to locally process in-band events and create the already-dispatched event table 450, so that the already-dispatched event messages can be recorded and detected.

[0053] In an exemplary embodiment, the MSE 420 or a component thereof may dispatch events based on event specific offsets of event segments. Examples include event start time offsets. The MSE 420 or a component thereof may determine event start time offsets of event segments, wherein the start time offset is determined by reference to the event presentation time, and the event start time offset is used to sort and dispatch events in the event clear buffer 430 or the event synchronizer and dispatcher 435. In an exemplary embodiment, the event start time offset and the timestamp offset may be equivalent, wherein the event start time offset refers to the event segment in the event clear buffer 430 or the event synchronizer and dispatcher 435, and the timestamp offset refers to the media segment in the media buffer 440.

[0054] In an exemplary embodiment, MSE 420 can handle event segments and media segment clearing and rewriting for associated events and media. If the media segments associated with the event are retained in media buffer 440, MSE 420 can retain the event in event clearing buffer 430 or event synchronizer and dispatcher 435. MSE 420 can manage the timing and duration of event segments based on the timing and duration of the associated media segments.

[0055] In an exemplary embodiment, the application 401 may set a timestamp offset or an event timestamp offset. The application 401 may also set a scheme URI / value and a dispatch mode for the scheme URI / value. The MSE 420 may dispatch an event based on the scheme URI / value, the event ID, an event association ID indicating a media segment associated with the event segment, a presentation time of the event segment, a start time offset of the event, and an event duration.

[0056] Figure 5 An example media and event buffer group 500 is shown that handles media segments and event segments. Each media stream 510 can include media segments (S0, S1, S2, S3) and event segments (E0, E1, E2). The media segments can be appended to a media source buffer or media buffer 520.

[0057] The in-band event and 'moof' parser 406 can parse the media stream and generate media segments and event segments. Each media segment can be appended to a media source buffer or media buffer 520 based on a timestamp offset. Each media segment appended to the media source buffer or media buffer 520 can have a unique event association ID. Using the event association ID of the corresponding media segment, the generated event segments (E0, E1, E2) can be associated with a specific media segment (S0, S1, S2, S3). In an exemplary embodiment, multiple event segments can be associated with one media segment.

[0058] In an exemplary embodiment, the event segments (E0, E1, E2) may be appended to the event clear buffer 530 by aligning the event start time and event duration of the event segments in the event clear buffer 530 with the start time and duration of the associated media segments (S0, S1, S2, S3) in the media buffer 520. As an example, the presentation time or event start time offset and duration in the event clear buffer 530 of one or more event segments may be aligned with the presentation time and duration of the associated media segments in the media buffer 620. In some exemplary embodiments, if more than one event segment is associated with the same media segment, the event start time offset of each event segment may be adjusted based on the presentation time of the associated media segments.

[0059] In some exemplary embodiments, the event fragments may be appended to the event dispatch buffer 540 by copying the event start time and duration of the corresponding event fragment in the event clear buffer 530. In some embodiments, the order, presentation time, and duration of the event fragments may be equivalent in both the event clear buffer 530 and the event dispatch buffer 540. As an example, if the application has set the scheme to "on_receive", the presentation time or event start time offset of the event fragment and the duration in the event dispatch buffer 540 may be aligned with the presentation time and duration of the associated media fragment, i.e., the event dispatch buffer 540 may be a copy of the event clear buffer 530.

[0060] Figure 6 An example media and event buffer group that handles media segments and event segments is shown. Each media stream 610 can include media segments (S0, S1, S2, S3) and event segments (E0, E1, E2). The media segments can be appended to a media source buffer or media buffer 620.

[0061] The in-band event and 'moof' parser 406 can parse the media stream and generate media segments and event segments. Each media segment can be appended to a media source buffer or media buffer 620 based on a timestamp offset. Each media segment appended to the media source buffer or media buffer 620 can have a unique event association ID. Using the event association ID of the corresponding media segment, the generated event segments (E0, E1, E2) can be associated with a specific media segment (S0, S1, S2, S3). In an exemplary embodiment, multiple event segments can be associated with one media segment.

[0062] In an exemplary embodiment, the event segments (E0, E1, E2) may be appended to the event clear buffer 630 by aligning the event start time and event duration of the event segments in the event clear buffer 630 with the start time and duration of the associated media segments (S0, S1, S2, S3) in the media buffer 620. As an example, the presentation time or event start time offset and duration in the event clear buffer 630 of one or more event segments may be aligned with the presentation time and duration of the associated media segments in the media buffer 620. In some exemplary embodiments, if more than one event segment is associated with the same media segment, the event start time offset of each event segment should be adjusted based on the presentation time of the associated media segments.

[0063] In an exemplary embodiment, an event fragment may be appended to the event dispatch buffer 640 by adding the event fragment so that the event start time offset of the event fragment is the presentation time of the event in the event dispatch buffer 640, and the duration of the event fragment is the duration of the event in the event dispatch buffer 640. As an example, if the application has set the scheme to "on_start", the presentation time of the event fragment and the duration in the event dispatch buffer 640 may reflect the event start time offset and duration of the event fragment. In some embodiments, the event dispatch buffer 640 may reflect the event start time and duration of the event fragment, that is, the event fragment may be appended so that the event fragment is in the event dispatch buffer 640 from the beginning of the event fragment to the end of the event fragment. Therefore, in some embodiments, the event clearing buffer 630 and the event dispatch buffer 640 may not be equivalent.

[0064] Figure 7 An example media and event buffer set 700 for processing media segments and event segments is shown. Each media stream 710 may include media segments (S0, S1, S2, S3) and event segments (E0, E1, E2). The media segments may be appended to a media source buffer or media buffer 720.

[0065] In an exemplary embodiment, the event segments (E0, E1, E2) in the event clear buffer 730 and the event dispatch buffer 740 may be removed, cleared, or rewritten. In some embodiments, the event segments may be partially removed, cleared, or rewritten. In order to remove, clear, or rewrite the event segments, the event segments may be removed from the event clear buffer 730 and the event dispatch buffer 740. As an example, once a media segment is rewritten in the media buffer 720, the corresponding event segments associated with the rewritten media segment in the event clear buffer 730 and the event dispatch buffer 740 may also be rewritten.

[0066] In some embodiments, in order to partially remove or rewrite event fragments, the event fragments may be divided into smaller sections in the event clear buffer 730 and the event dispatch buffer 740, and only the sections of the event fragments that overlap with the new event fragment may be removed, cleared, or rewritten, and then the sections of the new event fragment may be added. As an example, if the new event fragment is smaller than the current event fragment in the event clear buffer 730 or the event dispatch buffer 740, one or more current event fragments in the event clear buffer 730 may be divided into smaller sections to match the new event fragment. The sections of the event fragment in the event clear buffer 730 that overlap with the new smaller event fragment may be removed, cleared, or rewritten.

[0067] In some embodiments, only event segments in the event clear buffer 730 that overlap with the new event segment may be removed, cleared, or rewritten. In some embodiments, only event segments in the event dispatch buffer 740 that overlap with the new event segment may be removed, cleared, or rewritten. The event clear buffer 730 may be rewritten, and then the event dispatch buffer 740 may be rewritten, so that the event clear buffer 730 and the event dispatch buffer 740 may be equivalent. As an example, if the application has set the scheme to "on_receive", event segments or individual segments of event segments that overlap with the new event segment in both the event clear buffer 730 and the event dispatch buffer 740 may be rewritten.

[0068] In some exemplary embodiments, the removal, clearing or overwriting of an event segment requires a determination that no other event segments associated with the same media segment exist in the event clear buffer 730. In some exemplary embodiments, if no other event segments associated with the same media segment exist in the event clear buffer 730, the corresponding event in the event dispatch buffer 740 may be removed, cleared or overwritten. As an example, if more than one event segment overlaps with a new event segment in the event clear buffer 730, overwriting the overlapping event segments requires that all event segments associated with the same media segment be dispatched before overwriting the overlapping event segments in the event clear buffer 730. In some embodiments, if no event segments associated with the same media segment exist in the event clear buffer 730, the overlapping event segments may be overwritten in the event clear buffer 730. As an example, if the application has set the scheme to "on_start", overwriting the overlapping events in the event clear buffer 730 requires that all event segments associated with the same media segment be dispatched before overwriting the overlapping event segments in the event clear buffer 730. In some embodiments, once an event segment is removed, cleared, or overwritten in the event clear buffer 730, the corresponding event segment may be removed, cleared, or overwritten in the event dispatch buffer 740. As an example, if the application has set the scheme to "on_start", the overlapping segments to be overwritten in the event dispatch buffer 740 may be overwritten after one or more overlapping event segments in the event clear buffer 730 are overwritten.

[0069] Fig. 8A An exemplary flow chart illustrating a process 800 for processing DASH and CMAF in-band events is shown, wherein, according to an exemplary embodiment, event fragments may be appended to an event clearing buffer and an event dispatching buffer. At 810, media data is obtained. The media data may be any type of media data. As an example, the media data may be audio data, video data, a document, or any combination thereof. The obtained media data may also include event data.

[0070] At 815, at least one event segment may be generated from the obtained media data. In some embodiments, the MSE or a parser component of the MSE may generate an event segment from the obtained media data. The event segment may include an MPD and an in-band event to deliver events related to the media timeline to a client or application. As an example, the in-band event and 'moof' parser (306 or 406) may generate an event segment from the obtained media data. In some embodiments, the parser may generate both a media segment and an event segment corresponding to the media segment. As an example, the in-band event and 'moof' parser (306 or 406) may generate a media segment and an associated event segment based on the media data. The event segment may be associated with the event segment using an event association ID, wherein the event association ID may be a unique ID associated with the media segment. In some embodiments, multiple event segments may be associated with the same media segment. Event segments associated with the same media segment may have an event association ID.

[0071] At 820, the event start time offset of the event fragment is determined. In some embodiments, the MSE or a component thereof determines the event start time offset of each event fragment to be attached to the event clear buffer or the event dispatch buffer. In some embodiments, the event start time offset of the event fragment can be determined based on the time when the event fragment is attached to the event clear buffer. In some embodiments, the event start time offset of the event fragment can be determined based on the time when the event fragment is attached to the event dispatch buffer. In some embodiments, the event start time offset of the event fragment can be determined based on the presentation time of the event fragment or media fragment in the MSE API. The start time offset of the event fragment indicates the start time of the event starting from the dispatch moment. Therefore, when the event fragment is sent to the application, the application can determine the relative start of the event from the dispatch moment, rather than comparing a common time reference.

[0072] The traditional approach is to use the event start time in MSE and process the event in the application. However, this traditional approach requires maintaining a common time reference.

[0073] According to this embodiment, an alternative time synchronization method is provided, which uses the event start time offset of the event segment determined based on the time in the MSE API, thereby avoiding the need for the application to compare a common time reference when determining the relative start time of the event segment, increasing the application processing efficiency and reducing bandwidth requirements.

[0074] At 825, the event segments can be appended to an event clear buffer based on the event start time offset of the event segments. The event clear buffer can be used to determine the order of the event segments, their location, and dispatch timing. In some embodiments, appending the event segments to the clear buffer can include aligning the presentation time and duration of the event segments with the corresponding presentation time and duration of the associated media segments. As an example, the event segments associated with the media segments can be placed in the event clear buffer so that each event segment and the associated media segment have the same presentation time and duration.

[0075] At 830, the event fragment may be attached to the event dispatch buffer based on the dispatch information of the event fragment. In some embodiments, the dispatch information may include information indicating at least one of event initialization, event attachment, event clearing, event duration, event override, or attributes set by the application. Some attributes set by the application, or some attributes set by the application based on the attribute set, include a scheme URI / value, an id, an event start time offset of the event fragment, a start presentation increment of the event fragment, and an event fragment duration.

[0076] In some embodiments, appending the event fragments to the event dispatch buffer may include repeatedly appending the event fragments to the event clearing buffer, or duplicating such appending operation. Thus, the order, presentation time, and duration of the event fragments may be the same in the event clearing buffer and the event dispatch buffer. As an example, if the application has set the scheme to "on_receive", the presentation time or event start time offset of one or more event fragments and the duration in the event dispatch buffer may be aligned with the presentation time and duration of the associated media fragments, i.e., the event dispatch buffer may be a copy of the event clearing buffer.

[0077] In some embodiments, appending an event fragment to an event dispatch buffer may be accomplished by adding the event fragment as follows: the event start time offset of the event fragment is the event presentation time in the event dispatch buffer, and the duration of the event fragment is the event duration in the event dispatch buffer. As an example, if the application has set the scheme to "on_start", the presentation time and duration of the event fragment in the event dispatch buffer may reflect the event start time offset and duration of the event fragment. In some embodiments, the event dispatch buffer may reflect the event start time and duration of the event fragment, i.e., the event fragment may be appended so that the event fragment is in the buffer from the beginning of the event fragment until the end of the event fragment. Therefore, in some embodiments, the event clearing buffer and the event dispatch buffer may not be equivalent.

[0078] At 835, the event fragment can be dispatched to the application based on the dispatch information of the corresponding event fragment. The dispatch information can include information about the following items: event fragment initialization, event correlation ID, event start time offset, event fragment duration, application scheme URI / value, and event clear, event append, event override properties.

[0079] Figure 8B An exemplary flow chart illustrating a process 850 for processing DASH and CMAF in-band events is shown, wherein event fragments may be dispatched from an event purge buffer and an event dispatch buffer according to an exemplary embodiment.

[0080] At 860, an event segment in an event dispatch buffer for a particular presentation time is identified. In some embodiments, the MSE or a component thereof may identify an event segment in an event dispatch buffer for a particular presentation time. In some embodiments, the presentation time may be determined based on an MSE API event presentation time. In other embodiments, the presentation time may be determined based on a presentation time of a media segment. Multiple event segments may be associated with an event association ID, i.e., multiple event segments may be associated with the same media segment.

[0081] At 865, determine whether the event fragment for a specific presentation time identified in the event dispatch buffer is included in the dispatch event table. In some embodiments, the MSE or its components can determine whether the event fragment is included in the dispatch event table. The dispatch event table can save records of event fragments or event messages that have been dispatched to the application. Therefore, if the event fragment is included in the dispatch event table, the event fragment may have been dispatched to the application. If the event fragment in the event dispatch buffer at the presentation time is not included in the dispatch event table, the event fragment can be dispatched to the application at 870 because the event fragment may not have been dispatched to the application. As an example, if the event ID of the event fragment in the event dispatch buffer at the presentation time does not exist in the dispatch event table, the event fragment is dispatched to the application. As another example, if the event ID of the event fragment in the event dispatch buffer at the presentation time is included in the dispatch event table, the event fragment may have been dispatched to the application.

[0082] At 875, the event fragment dispatched to the application can be added to the dispatch event table. In some embodiments, the event fragment that is dispatched and not included in the dispatch event table is added to the dispatch event table. As an example, if the event fragment is dispatched to the application, the event ID of the event fragment can be added to the dispatch event table to maintain a record that the event fragment is dispatched to the application.

[0083] Fig. 9An exemplary flow chart illustrating a process 900 for processing DASH and CMAF in-band events is shown, wherein, according to an exemplary embodiment, event segments may be removed, cleared, or overwritten from an event clear buffer and an event dispatch buffer. At 910, new event segments from the new media data are generated. In some embodiments, the MSE or a component thereof may generate one or more new event segments from the new media data. In some exemplary embodiments, new event segments may be generated for new media segments appended to the media buffer. As an example, when new media segments are parsed out and appended to the media buffer, new event segments corresponding to the new media segments may be appended to the event clear buffer and the event dispatch buffer.

[0084] At 915, the event fragments that overlap with the new event fragments in the event clear buffer can be split to match the event duration, event start time offset, or presentation time of the new event fragment. If the new fragment is smaller than the event fragment in the event clear buffer, the overlapping event fragments in the event clear buffer can be split to match the event duration of the new event fragment. As an example, if the new event fragment that overlaps with the event fragment in the event clear buffer has a shorter event duration, the overlapping event fragments in the event clear buffer can be split into two or more segments corresponding to the shorter event duration of the new event fragment. In some embodiments, the overlapping event fragment segments in the event clear buffer are removed, cleared, or rewritten based on the duration of the new event fragment.

[0085] At 920, the event fragments overlapping with the new event fragments in the event dispatch buffer can be split to match the event duration, event start time offset, or presentation time of the new event fragment. If the new fragment is smaller than or shorter than the event fragments in the event dispatch buffer, the overlapping event fragments in the event dispatch buffer can be split to match the event duration, event start time offset, or presentation time of the new event fragment. As an example, if the new event fragments overlapping with the event fragments in the event dispatch buffer have a shorter event duration, the overlapping event fragments in the event dispatch buffer can be split into two or more segments corresponding to the shorter event duration of the new event fragment. In some embodiments, the overlapping event fragment segments in the event dispatch buffer are removed, cleared, or rewritten based on the duration of the new event fragments.

[0086] At 925, it is determined whether the event segments from the event clear buffer and the event dispatch buffer can be removed, cleared, or rewritten. Whether the event segments from the event clear buffer or the event dispatch buffer can be removed, cleared, or rewritten depends on whether all event segments associated with the media segment being rewritten have been dispatched. If the event segments from the event clear buffer that overlap with the new event segments are not associated with the same media segment, then at 930, the event segments from the event clear buffer that overlap with the new event segments can be deleted from the event dispatch buffer. At 935, the event segments from the event clear buffer that overlap with the new event segments can be deleted from the event clear buffer. As an example, if the application has set the scheme to "on_start", and if the event segments from the event clear buffer that overlap with the new event segments are not associated with the same media segment, the event segments that overlap with the new event segments can be deleted from the event clear buffer, and the event segments that overlap with the new event segments can also be deleted from the event dispatch buffer because there are other events associated with the same media segment that may have been dispatched.

[0087] In some embodiments, the step of determining whether overlapping event segments in the event clear buffer are not associated with the same media segment may be optionally performed. As an example, if the application has set the scheme to "on_receive", the step of determining whether overlapping event segments in the event clear buffer are not associated with the same media segment may not be performed.

[0088] At 930, event segments that overlap with the new event segments may be deleted from the event clear buffer. At 935, event segments that overlap with the new event segments may be deleted from the event dispatch buffer. At 940, according to an embodiment of the present disclosure, the new event segments may be appended to the event clear buffer and the event dispatch buffer.

[0089] Although Fig. 8A , Figure 8B and Fig. 9 Example blocks of processors 800, 850, 900 are shown, but in some implementations, processors 800, 850, 900 may include Fig. 8A , Figure 8B and Fig. 9 In one embodiment, the processors 800, 850, 900 may include more blocks, fewer blocks, different blocks, or differently arranged blocks than those depicted in the flowchart. In an embodiment, any blocks of the processors 800, 850, 900 may be combined or arranged in any quantity or order as desired. In an embodiment, at least two blocks of the processors 800, 850, 900 may be executed in parallel.

[0090] The above techniques may be implemented as computer software via computer-readable instructions and physically stored in one or at least two computer-readable media. For example, Fig.10 A computer device 1000 is shown that is suitable for implementing certain embodiments of the disclosed subject matter.

[0091] The computer software may be encoded in any suitable machine code or computer language, and may be assembled, compiled, linked, or other mechanisms to create a code comprising instructions, which may be directly executed by one or at least two computer central processing units (CPUs), graphics processing units (GPUs), etc., or executed by decoding, microcode, or the like.

[0092] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablets, servers, smartphones, gaming devices, IoT devices, etc.

[0093] Fig.10 The components shown for computer device 1000 are exemplary in nature and are not intended to limit the scope of use or functionality of computer software implementing embodiments of the present application. The configuration of components should not be interpreted as having any dependency or requirement on any component or combination of components shown in the exemplary embodiment of computer device 1000.

[0094] The computer device 1000 may include certain human-computer interface input devices. Such human-computer interface input devices may respond to input from one or at least two human users through tactile input (e.g., keyboard input, sliding, data glove movement), audio input (e.g., sound, applause), visual input (e.g., gestures), and olfactory input (not shown). The human-computer interface device may also be used to capture certain media that are not necessarily directly related to conscious human input, such as audio (e.g., speech, music, ambient sound), images (e.g., scanned images, photographic images obtained from a still image camera), and videos (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0095] The human-machine interface input device may include one or at least two of the following (only one of which is depicted): keyboard 1001 , mouse 1002 , touch pad 1003 , touch screen 1010 , joystick 1005 , microphone 1006 , scanner 1008 , camera 1007 .

[0096] The computer device 1000 may also include certain human-computer interface output devices. Such human-computer interface output devices may stimulate the senses of one or at least two human users through, for example, tactile output, sound, light, and smell / taste. Such human-computer interface output devices may include tactile output devices (e.g., tactile feedback through a touch screen 1010 or a joystick 1005, but there may also be tactile feedback devices that are not used as input devices), audio output devices (e.g., speakers 1009, headphones), visual output devices (e.g., screens 1010 including cathode ray tube screens, liquid crystal screens, plasma screens, organic light emitting diode screens, each of which has or does not have a touch screen input function, each of which has or does not have a tactile feedback function - some of which may output two-dimensional visual outputs or outputs of more than three dimensions by means such as stereoscopic picture output; virtual reality glasses (not shown), holographic displays, and smoke boxes) and printers.

[0097] The computer device 1000 may also include human-accessible storage devices and their associated media, such as optical media including high-density read-only / rewritable optical disks (CD / DVD ROM / RW) 1020 with CD / DVD or similar media 1011, a thumb drive 1022, a removable hard disk drive or solid state drive 1023, traditional magnetic media such as tapes and floppy disks, special-purpose devices based on ROM / ASIC / PLD such as security software protectors, and the like.

[0098] Those skilled in the art should also understand that the term "computer-readable media" used in connection with the disclosed subject matter does not include transmission media, carrier waves, or other transient signals.

[0099] The computer device 1000 may also include an interface 1099 to one or at least two communication networks 1098. For example, the network 1098 may be wireless, wired, or optical. The network 1098 may also be a local area network, a wide area network, a metropolitan area network, an in-vehicle network, an industrial network, a real-time network, a delay-tolerant network, and the like. The network 1098 also includes local area networks such as Ethernet, wireless local area networks, cellular networks (GSM, 3G, 4G, 5G, LTE, etc.), television wired or wireless wide area digital networks (including cable television, satellite television, and terrestrial broadcast television), in-vehicle and industrial networks (including CANBus), and the like. Some networks 1098 typically require an external network interface adapter for connecting to some universal data ports or peripheral buses (1050 and 1051) (e.g., a USB port of the computer device 1000); other systems are typically integrated into the core of the computer device 1000 by connecting to the system bus described below (e.g., an Ethernet interface integrated into a PC computer device or a cellular network interface integrated into a smart phone computer device). By using any of these networks 1098, the computer device 1000 can communicate with other entities. The communication can be one-way, for reception only (e.g., wireless television), one-way, for transmission only (e.g., CAN bus to certain CAN bus devices), or two-way, such as to other computer devices via a local or wide area digital network. Each of the above networks and network interfaces can use certain protocols and protocol stacks.

[0100] The above-mentioned human-machine interface devices, human-accessible storage devices, and network interfaces may be connected to the core 1040 of the computer device 1000 .

[0101] The core 1040 may include one or at least two central processing units (CPUs) 1041, a graphics processing unit (GPU) 1042, a dedicated programmable processing unit in the form of a field programmable gate array (FPGA) 1043, a hardware accelerator for a specific task 1044, etc. These devices, as well as a read-only memory (ROM) 1045, a random access memory 1046, an internal mass storage (e.g., an internal non-user accessible hard disk drive, a solid state drive, etc.) 1047, etc., may be connected via a system bus 1048. In some computer devices, the system bus 1048 may be accessed in the form of one or at least two physical plugs so that it may be expanded by additional central processing units, graphics processing units, etc. Peripheral devices may be directly attached to the system bus 1048 of the core, or connected via a peripheral bus 1049. The architecture of the peripheral bus includes a peripheral controller interface PCI, a universal serial bus USB, etc.

[0102] The CPU 1041, GPU 1042, FPGA 1043, and accelerator 1044 may execute certain instructions, which in combination may constitute the above-mentioned computer code. The computer code may be stored in ROM 1045 or RAM 1046. Transitional data may also be stored in RAM 1046, while permanent data may be stored in, for example, internal mass storage 1047. Fast storage and retrieval of any memory device may be achieved by using a cache memory, which may be closely associated with one or at least two CPUs 1041, GPU 1042, mass storage 1047, ROM 1045, RAM 1046, etc.

[0103] The computer readable medium may have computer code thereon for performing various computer-implemented operations. The medium and computer code may be specially designed and constructed for the purpose of this application, or may be medium and code well known and available to those skilled in the art of computer software.

[0104] As an embodiment and not limitation, the computer device 1000 having the structure shown, in particular the core 1040, can be used as a processor (including a CPU, a GPU, an FPGA, an accelerator, etc.) to provide the function of executing the software contained in one or at least two tangible computer-readable media. Such a computer-readable medium can be a medium associated with the above-mentioned user-accessible mass storage, and a specific memory of the core 1040 having non-volatility, such as a core internal mass storage 1047 or a ROM 1045. The software implementing the various embodiments of the present application can be stored in such a device and executed by the core 1040. According to specific needs, the computer-readable medium may include one or more storage devices or chips. The software can enable the core 1040, in particular the processor therein (including a CPU, a GPU, an FPGA, etc.) to perform a specific process or a specific part of a specific process described herein, including defining a data structure stored in the RAM 1046 and modifying such a data structure according to a software-defined process. Additionally or alternatively, a computer device may provide functionality hardwired in logic or otherwise contained in a circuit (e.g., accelerator 1044) that may replace software or run in conjunction with software to perform a particular process or a particular portion of a particular process described herein. Where appropriate, references to software may include logic and vice versa. Where appropriate, references to computer-readable media may include circuits (such as integrated circuits (ICs)) storing execution software, circuits containing execution logic, or both. The present application includes any suitable combination of hardware and software.

[0105] Although at least two exemplary embodiments have been described in this application, various changes, arrangements and various equivalent substitutions of the embodiments are within the scope of this application. Therefore, it should be understood that those skilled in the art can design a variety of systems and methods, which, although not explicitly shown or described herein, embody the principles of this application and are therefore within the spirit and scope of this application.

Claims

1. A method for processing events in a media stream, characterized in that: include: Get media data; Generating at least one event segment from the media data; appending the at least one event segment to a first event processing buffer, the at least one event segment including an event start time offset for each event segment of the at least one event segment, the event start time offset being determined based on a time at which the event segment is appended to the first event processing buffer; appending the at least one event fragment to a second event processing buffer, the at least one event fragment further comprising event dispatch information for each event fragment in the at least one event fragment; and, Dispatching the at least one event fragment based on the event dispatching information in the first event processing buffer and the second event processing buffer; The method further comprises: generating at least one new event segment; splitting the at least one event segment in the first event processing buffer and the second event processing buffer based on the event duration of the at least one new event segment; rewriting the at least one event fragment in the first event processing buffer and the second event processing buffer based on the event duration of the at least one new event fragment; Wherein, rewriting the at least one event segment in the first event processing buffer and the second event processing buffer based on the event duration of the at least one new event segment comprises: determining that the at least one event segment in the first event processing buffer is not associated with the same media segment; deleting the at least one event fragment in the second event processing buffer; deleting the at least one event fragment in the first event processing buffer; The at least one new event fragment is appended to the first event processing buffer and the second event processing buffer.

2. The method according to claim 1, characterized in that The appending the at least one event fragment to the first event processing buffer comprises: A presentation time and a duration of each of the at least one event fragment are aligned with a presentation time and a duration of at least one associated media fragment in the media buffer.

3. The method according to claim 1, characterized in that The appending the at least one event fragment to the second event processing buffer comprises: The additional operations of the at least one event fragment are copied to the first event processing buffer.

4. The method according to claim 1, characterized in that: The appending the at least one event fragment to the second event processing buffer comprises: Based on the event dispatching information of each event fragment in the at least one event fragment, the at least one event fragment is appended to the second event processing buffer.

5. The method according to claim 1, characterized in that The dispatching of the at least one event fragment based on the event dispatching information in the first event processing buffer and the second event processing buffer comprises: determining whether the at least one event fragment is included in a dispatch event table; When it is determined that the at least one event fragment is not included in the dispatch event table, the at least one event fragment is dispatched to the application.

6. A device for processing events in a media stream, characterized in that: include: A first acquisition module, used to obtain media data; A first generating module, configured to generate at least one event segment from the media data; A first appending module, configured to append the at least one event segment to a first event processing buffer, the at least one event segment comprising an event start time offset of each event segment in the at least one event segment, the event start time offset being determined based on a time when the event segment is appended to the first event processing buffer; A second appending module, configured to append the at least one event fragment to a second event processing buffer, the at least one event fragment further comprising event dispatching information of each event fragment in the at least one event fragment; and, A first dispatching module, configured to dispatch the at least one event fragment based on the event dispatching information in the first event processing buffer and the second event processing buffer; A second generating module, configured to generate at least one new event segment; A first splitting module, configured to split the at least one event segment in the first event processing buffer and the second event processing buffer based on an event duration of the at least one new event segment; A first rewriting module, configured to rewrite the at least one event segment in the first event processing buffer and the second event processing buffer based on the event duration of the at least one new event segment; Wherein, the first rewriting module includes: A second determination module, configured to determine that the at least one event segment in the first event processing buffer is not associated with the same media segment; A first deleting module, configured to delete the at least one event fragment in the second event processing buffer; A second deleting module, configured to delete the at least one event segment in the first event processing buffer; A third appending module is used to append the at least one new event fragment to the first event processing buffer and the second event processing buffer.

7. The device according to claim 6, characterized in that The first additional module is used to align the presentation time and duration of each event segment of the at least one event segment with the presentation time and duration of at least one associated media segment in the media buffer.

8. The device according to claim 6, characterized in that The second additional module is used for copying the additional operation of the at least one event fragment to the first event processing buffer.

9. The device according to claim 6, characterized in that The second appending module is used to append the at least one event fragment to the second event processing buffer based on the event dispatching information of each event fragment in the at least one event fragment.

10. The device according to claim 6, characterized in that The first distribution module includes: A first determination module, configured to determine whether the at least one event fragment is included in a dispatch event table; The second dispatching module is configured to dispatch the at least one event fragment to the application when it is determined that the at least one event fragment is not included in the dispatching event table.

11. A non-transitory computer-readable medium, characterized in that A program is stored thereon, and when the program is executed by at least one processor, the method according to any one of claims 1 to 5 is implemented.

12. A computer device, characterized in that: The method comprises a processor and a memory, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the method according to any one of claims 1 to 5.

Images