Server-side initiation and playback of primary and secondary media content

Abstract

The server receives a request for a media segment of a media stream to be delivered to the client. Using the server-based timeline of the media stream, it determines whether the requested media segment is intended to contain primary or secondary media content. The timeline defines one or more locations within the media stream for containing secondary media content. In response to determining that the requested media segment corresponds to one of the one or more locations defined in the timeline, the server retrieves the secondary media content item. To provide the requested media segment to the client, the server adds the media segment of the secondary media content item to the media stream.

Description

Technical Field

[0001] Aspects and embodiments of the present disclosure relate to streaming media, and more particularly, to the server-side initiation and playback of primary and secondary media content.

Background Art

[0002] Streaming refers to the process of continuously delivering media content (e.g., video, audio) to a recipient while the recipient consumes the media content. In an exemplary example, a server device may stream video to a client device by dividing the video data into segments and sequentially transmitting those segments to the client device via a network. The client device may present the received segments to the user as soon as the segments arrive at the client device, or the client may store the segments in a buffer for later presentation.

[0003] A streaming platform can support a variety of client devices and network conditions. For example, a streaming platform can support different streaming formats (e.g., H.264 / AVC, VP9, ​​H.265 / HEVC, AV1, ACC, MP3) and adaptive bitrate (ABR) streaming. ABR technology allows the server to detect the current bandwidth conditions of the network or client device and increase or decrease the stream's bitrate to match the current bandwidth. The stream's bitrate can be modified, for example, by changing the quality of the media content (e.g., resolution, frame rate). Technologies such as ABR can reduce the user's perception of buffering on the client device, as a bitrate exceeding the current bandwidth can cause the stream to lag behind the client-side presentation of media content, resulting in a pause in the user's viewing experience to replenish the client-side buffer. A streaming platform may further use technologies such as digital rights management (DRM) and encryption to manage access to media content. [Overview of the project]

[0004] The following summary is a simplified overview of the Disclosure to provide a basic understanding of some aspects of the Disclosure. This summary is not a comprehensive overview of the Disclosure. It is not intended to identify any major or significant elements of the Disclosure, nor to clarify any scope of any particular embodiment of the Disclosure or any scope of any claim. Its sole purpose is to present some concepts of the Disclosure in a simplified form as a prelude to more detailed explanations presented later.

[0005] In some embodiments, systems and methods for initiating and playing secondary media content for a streaming platform are disclosed. In one embodiment, the method includes a server receiving a request for a media segment of a media stream to be delivered to a client. The method further includes using a server-based timeline of the media stream to determine whether the requested media segment is for primary media content or for secondary media content. The timeline defines one or more locations in the media stream for including secondary media content. In response to the method determining that the requested media segment corresponds to one of the one or more locations defined in the timeline, the server further includes retrieving a secondary media content item. The method further includes adding the media segment of the secondary media content item to the media stream in order to provide the requested media segment to the client. In some embodiments, the server-based timeline is synchronized across multiple timeline servers.

[0006] In some embodiments, the method further includes sending metadata associated with a secondary media content item to a client, the metadata corresponding to one or more graphical user interface (GUI) elements presented to the client by the secondary media content item.

[0007] In some embodiments, the method further includes sending a timeline of a media stream to a client, the timeline of the media stream showing one or more permitted seek actions available to the client's users.

[0008] In some embodiments, the method further includes the server receiving indications of authorized seek actions performed by a client user. The method further includes updating a server-based timeline of the media stream based on the indications of authorized seek actions performed by a client user. The method further includes the server receiving a request for a second media segment of the media stream to be delivered to the client. The method further includes using the updated server-based timeline of the media stream to determine whether the requested second media segment is for primary media content or for secondary media content.

[0009] In some embodiments, the method further includes the server receiving an indication of an unauthorized seek action performed by a client user. The method further includes the server receiving a request for a second media segment of a media stream to be delivered to the client. Regardless of the indication of an unauthorized seek action performed by a client user, the method further includes using a server-based timeline of the media stream to determine whether the requested second media segment is intended to contain primary media content or secondary media content.

[0010] In some embodiments, obtaining a secondary media content item includes the server requesting a secondary media content server to select one of several secondary media content items, and receiving the secondary media content item from the secondary media content server.

[0011] In some embodiments, the media segment preceding the media stream is associated with a first format. The media segment of the secondary media content item is associated with a second format. Adding the media segment of the secondary media content item to the media stream includes prepending an initialization segment to the media segment of the secondary media content item. The initialization segment causes the client to reinitialize its decoder to decode the second format.

[0012] In some embodiments, a computer-readable storage medium (which may be a non-temporary computer-readable storage medium, but is not limited thereto) stores, when executed, instructions causing a processing device to perform an operation including a method according to any embodiment or aspect described herein.

[0013] In some embodiments, the system comprises a memory and a processing device operably coupled to the memory for performing operations including methods according to any embodiment or aspect described herein.

[0014] The aspects and embodiments of this disclosure will be better understood from the detailed descriptions of various aspects and embodiments of this disclosure below, as well as from the accompanying drawings, but these are for illustrative and illustrative purposes only and should not be construed as limiting this disclosure to any particular aspect or embodiment. [Brief explanation of the drawing]

[0015] [Figure 1] This disclosure illustrates an exemplary streaming platform system architecture that provides server-side initiation and playback of primary and secondary media content, according to at least one embodiment of this disclosure. [Figure 2] An exemplary client device for a streaming platform, according to at least one embodiment, is shown. [Figure 3A]An exemplary media stream timeline for initiating primary and secondary media content on the server side, according to at least one embodiment, is shown. [Figure 3B] An exemplary media stream timeline for initiating primary and secondary media content on the server side, according to at least one embodiment, is shown. [Figure 4A] This document shows an exemplary media stream for playing primary and secondary media content on the server side, according to at least one embodiment. [Figure 4B] This document shows an exemplary media stream for playing primary and secondary media content on the server side, according to at least one embodiment. [Figure 5A] A flowchart shows an exemplary method for providing server-side initiation and playback of primary and secondary media content on a streaming platform, according to at least one embodiment. [Figure 5B] A flowchart shows an exemplary method for providing server-side initiation and playback of primary and secondary media content on a streaming platform, according to at least one embodiment. [Figure 5C] A flowchart shows an exemplary method for providing server-side initiation and playback of primary and secondary media content on a streaming platform, according to at least one embodiment. [Figure 6] A block diagram showing an exemplary computer system according to at least one embodiment of the present disclosure. [Modes for carrying out the invention]

[0016] This disclosure relates to the server-side initiation and playback of primary and secondary media content. A streaming experience may include primary and secondary media content. Examples of primary media content include live video or audio (e.g., live TV, music concerts, etc.) and media on demand (e.g., video on demand or VOD, recorded music, video or music playlists, etc.). Primary content may range in length from a few seconds (e.g., short videos) to continuous or endless content (e.g., natural streams). Secondary media content may be placed at the beginning or end of primary content (e.g., pre-roll or post-roll), or within primary media content as a break or interruption (e.g., mid-roll). Examples of secondary media content include advertisements, information segments or announcements, surveys, and links to other media streams. Secondary media content may also vary in length, and multiple secondary media content items may be placed within or near a single primary media content item (e.g., multiple commercials within a single live TV program). Both primary and secondary media content may be interactive. For example, playback controls (e.g., play, pause, fast forward / rewind, volume control) may be located inside or outside the visible content area. In another example, secondary media content may include survey responses that the user can select (e.g., by clicking or tapping). Secondary media content items for a given media stream may be predetermined (e.g., before streaming begins) or determined immediately before each secondary media content item is presented. In the latter case, the media stream may include fixed-length or variable-length placeholders that are filled with secondary media content items at the appropriate time. For example, an ad auction may take place immediately before ad slots are streamed to determine which ads will fill the slots.

[0017] In conventional systems, primary and secondary media content may be streamed to the client device as separate media streams, which are then combined (e.g., stitched) into a single, continuous viewing experience for the user through migration from primary to secondary media content (or vice versa). For example, live TV content may be streamed from a live TV server on a streaming platform, and advertisements may be streamed from an ad server on the streaming platform or a third-party ad platform. These separate streams may be encoded in different formats, encrypted with different DRM keys, or have other compatibility issues that prevent the client device from simply appending data from one stream to the end of data from the other in order to perform the migration from primary to secondary media content. Therefore, in some situations, the client device may initialize two media players (one for each format or DRM key) and swap them out during content migration, or the client device may reinitialize the media player during content migration to decrypt different formats. These actions may result in significant latency for the client device during content migration, which can negatively impact the user experience. Furthermore, supporting the transition from primary to secondary media content on the client side can be a burden for streaming platform developers. Streaming platform developers often have to develop and maintain custom implementations of the transition logic for each client device their platform supports (e.g., smartphones, tablets, smart TVs, etc.). This can increase the likelihood of software or hardware bugs in some implementations, potentially negatively impacting the user experience.These challenges can lead to unnecessary consumption of computing resources on both the client and server sides, excessive use of network bandwidth, and an excessive burden on developers' time and resources, all of which could lead to increased spending on streaming platforms.

[0018] Another challenge in traditional systems relates to deciding which secondary media content items (or more) to present to the client device. The client device often controls this process in traditional systems. The client device may maintain a local timeline of the media stream, which can define where secondary media content should be included. Using this local timeline, the client device can request secondary media content items from the server at the appropriate time to fill the defined positions. For example, the client device might initiate an ad auction or request an interactive survey. In some systems, the client may make multiple requests in sequence to retrieve secondary media content items, consuming additional network bandwidth for more round trips and exacerbating migration latency issues. For example, the client might first contact one server to initiate an ad auction and receive an identifier for a selected ad, and then the client might contact a second server to begin streaming the identified ad. Client-side systems like the example above can lead to unnecessary use of computing resources and network bandwidth, excessive latency, and a degraded user experience, as well as challenges associated with developing custom implementations of the aforementioned client-side systems (e.g., increased developer time / cost, increased potential for bugs). Furthermore, client-side implementations may be susceptible to undesirable manipulation by users or other client-side software or hardware. For example, software may manipulate the client-local timeline to delete secondary content or block requests to secondary content servers. These actions can affect the normal functionality of the streaming system, placing an additional load on the streaming platform and negatively impacting the user experience.

[0019] Aspects of this disclosure address the above and other deficiencies by providing server-side initiation and playback of primary and secondary media content. Streaming platforms and other streaming systems or services utilizing the techniques described herein may include a server-side media stream timeline for determining whether to stream primary or secondary media content or to initiate selection of media content. A streaming platform may also include a streaming server for server-side playback of the media stream, which may generate a client media stream by providing server-side stitching or encapsulation of primary and / or secondary media content. The streaming platform may define the boundaries between primary and secondary media content items in the server-stitched media stream and further generate metadata including graphical user interface (GUI) elements for the secondary media content items.

[0020] In at least one embodiment, a server-side media stream timeline is provided. The server-side media stream timeline can be created and updated by a streaming platform and can define the positions of primary and secondary media content within the timeline. The media stream timeline can further indicate the current playback position (e.g., with respect to the current media content item or the entire media stream) for various media content items, and permitted or non-permitted user actions (e.g., seek actions). The streaming platform can query the server-side timeline to determine the next content item to be streamed in response to requests from the client for additional media segments of the media stream. Next, the streaming platform can select and obtain the next media content item. The streaming platform can update the server-side timeline based on permitted user seek actions and can ignore non-permitted user seek actions to provide a final timeline for the media stream. The server-side media stream timeline can further be shared among multiple users, and the content items of the media stream timeline can be customized for each user based on the characteristics of the user or client device.

[0021] In at least one embodiment, server-side stitching and encapsulation of a media stream are provided. Server-side stitching of a media stream may include combining primary and secondary media content items encoded in the same format such that the media segment of one content item is appended to the media segment of the other content item, creating a single media stream in one format. Server-side encapsulation of a media stream may include combining primary and secondary media content items encoded in different formats such that the media segment of one content item is preceded only by an initialization segment, with the content item having the initialization segment appended to the media segment of the other content item, creating a single media stream in two formats. The initialization segment causes the client to reinitialize a decoder or media player to switch from decoding the first format to decoding the second format. In both stitched and encapsulated streams, only a single stream containing all primary and secondary media content items is streamed to the client.

[0022] In at least one embodiment, media stream metadata is provided. The media stream metadata may include a copy of the server-side media stream timeline that is sent to the client. The media stream metadata may also include an indication of whether the server is currently streaming primary media content or secondary media content, and may further include GUI elements presented to the user in association with the currently presented media content item. The metadata may cause the client device to update a local progress bar indicating the current playback time of the current media content item, present a "skip" button or link, or perform other local updates. The media stream metadata may also instruct the client device to perform certain activities, such as logging data related to engagement with secondary media content.

[0023] Therefore, a streaming platform using the techniques described herein can provide an enhanced streaming experience, including primary and secondary media content. Server-side initiation and selection of secondary media content can reduce client requests and round trips to the server, as well as interference from other client-side software. A shared server-side timeline can reduce the computational load on the streaming platform and can adapt to user interests or client device limitations. Server-side playback of media streams can also reduce client network connectivity and the computational load on client devices. Both server-side initiation and playback of media streams can reduce the amount of software that streaming platform developers must create and maintain by reducing the software footprint on client devices. Thus, a streaming platform may experience reduced latency (e.g., network latency and transition latency), reduced use of computing resources, improved developer efficiency, and lower operating costs, benefiting both users and the streaming platform.

[0024] Figure 1 shows an exemplary streaming platform system architecture 100 that provides server-side initiation and playback of primary and secondary media content. The system architecture 100 (also referred to herein as the “System” or “Streaming Platform”) includes one or more server machines 110-140, a data store 106, and client devices 102A-102N connected to a network 104. In some implementations, the network 104 may include a public network (e.g., the Internet), a private network (e.g., a local area network (LAN) or a wide area network (WAN)), a wired network (e.g., an Ethernet network), a wireless network (e.g., an 802.11 network or a Wi-Fi network), a cellular network (e.g., a Long-Term Evolution (LTE) network), routers, hubs, switches, server computers, and / or combinations thereof. For example, network 104 may include a private enterprise network connecting datastore 106 to one or more of server machines 110-140, and the private enterprise network may similarly be connected to client devices 102A-102N via the internet. In some embodiments, network 104 may be a physical or virtual interconnection within a single server providing all of one or more components of server machines 110-140. For example, network 104 may be a PCIe bus, a messaging system, or an API.

[0025] The datastore 106 is persistent storage capable of storing streaming platform content, such as primary and secondary media content items, user profiles and preferences, system configurations and settings, and log data. The datastore 106 may be hosted by one or more storage devices, such as main memory, disks based on magnetic or optical storage, tape or hard drives, NAS, or SAN. In some embodiments, the datastore 106 may be a network-attached file server. In some embodiments, the datastore 106 may be several other types of persistent storage, such as object-oriented databases or relational databases. In some embodiments, the datastore 106 may be hosted on one or more of the server machines 110-140, or may be a component thereof. In some embodiments, the datastore 106 may be provided by a third-party service, such as a cloud platform provider.

[0026] Client devices 102A-102N may be personal computers (PCs), laptops, notebook computers, mobile phones, smartphones, tablet computers, digital assistants, network-connected televisions (e.g., smart TVs), or any other computing devices. The computer system in Figure 6 may be an example of a client device. In some embodiments, client devices 102A-102N may also be referred to as “user devices”. Client devices 102A-102N may run an operating system (OS) that manages the hardware and software of client devices 102A-102N. Client devices 102A-102N may further include a web browser, application, or other software for streaming media content. Client devices 102A-102N may be used by users such as viewers of a streaming platform. Exemplary client devices will be further described with respect to Figure 2.

[0027] Each of the server machines 110-140 could be a rack-mount server, router computer, personal computer, portable digital assistant, mobile phone, laptop computer, tablet computer, netbook, desktop computer, virtual machine (VM), or any combination thereof. The computer system in Figure 6 may be an example of a server machine. In some embodiments, one or more of the server machines 110-140 may be combined into a single server machine that provides all the components of the individual server machines shown in Figure 1. In some embodiments, each of the server machines 110-140 could be several computing devices, such as multiple rack-mount servers in a data center or multiple VMs in a cloud platform.

[0028] The server machine 110 includes a streaming server 112 that can provide streaming functionality to the streaming platform. Streaming functionality may include receiving client requests to start a media stream or stream a media segment, querying a server-side timeline, determining the type of media content and selecting media content items to stream, retrieving media content items from local or remote storage (e.g., datastore 106 or secondary content server 132), adding media segments to the media stream (e.g., by stitching or encapsulation), adding DRM protection to the media stream, and various other activities. The streaming server 112 may manage multiple active media streams for multiple clients. In some embodiments, a single media stream managed by the streaming server 112 may be associated with multiple clients (e.g., live TV programs).

[0029] The streaming server 112 may include one or more media content items, such as media content item 114. Media content item 114 may be a primary media content item (e.g., live TV program, VOD, music track) or a secondary media content item (e.g., advertisement, announcement, survey). Media content item 114 can be consumed via the Internet or via a mobile device application, such as a media player 206 on a client device 200 as described in Figure 2. In some embodiments, media content item 114 may correspond to a media file (e.g., video file, audio file, video stream, audio stream, etc.). In other or similar embodiments, media content item 114 may correspond to a portion of a media file (e.g., a portion or chunk of a video file, audio file, etc.). As used herein, “media,” “media item,” “multimedia item,” “online media item,” “digital media,” “digital media item,” “content,” “multimedia content,” and “content item” may include electronic files that can be executed or loaded using software, firmware, or hardware configured to present digital media items to an entity. In at least one embodiment, the streaming server 112 may use the data store 106 to store media content items 114 or references to media content items 114. In other embodiments, the streaming server 112 may use the data store 106 to store media content items 114 or fingerprints as electronic files in one or more formats (e.g., H.264 / AVC, VP9, ​​H.265 / HEVC, AV1, ACC, MP3).The streaming server 112 can provide the media content item 114 to a user associated with one of the client devices 102A to 102N by enabling access to the media content item 114 (for example, via a streaming platform application), by sending the media content item 114 to the client device and / or allowing the presentation or presentation of the media content item 114 via the client device. Streaming to the client device will be further described with reference to Figures 3 and 4.

[0030] In some embodiments, the media content item 114 may be a video item. A video item refers to a set of consecutive video frames (e.g., image frames) representing a scene in motion. For example, a series of consecutive video frames may be continuously filmed or later reconstructed to create an animation. Video items may be provided in a variety of formats, including but not limited to analog, digital, two-dimensional, and three-dimensional video. Furthermore, a video item may include a film, a video clip, a video stream, or any set of images displayed in sequence (e.g., animated images, non-animated images, etc.). In some embodiments, a video item may be stored (e.g., in the data store 106) as a video file containing video and audio components. The video component may include video data corresponding to one or more consecutive video frames of the video item. The audio component may include audio data corresponding to the video data.

[0031] In some embodiments, a media content item 114 may be associated with metadata. The metadata may include the title, creator, channel, caption, and comments from other users related to the media content item 114. The metadata may also include timeline-related information, such as the current playback position, the most viewed or most interesting time range, and the location of secondary media content within the media content item 114. The metadata may also include GUI elements or identifiers of GUI elements presented to the user. For example, the metadata may cause the client device to present a skip button, a list of exploration options, or a link to a user associated with the media content item 114. The metadata may also include instructions to the client device to perform one or more actions, such as logging events related to the secondary media content item (e.g., user interaction or user engagement).

[0032] A streaming platform can include multiple channels (e.g., channels A through Z). A channel can include one or more media content items 114 available from a common source, or media content items 114 that share a common topic, theme, or content. Media content items 114 can be user-selected digital content, user-made digital content, user-uploaded digital content, content selected by a content provider, content selected by a broadcaster, etc. For example, channel X can include videos Y and Z. A channel can be associated with an owner, who is a user who can take actions on the channel. Different activities can be associated with a channel based on the owner's actions, such as the owner making digital content available on the channel, the owner selecting digital content associated with another channel (e.g., liking it), or the owner commenting on digital content associated with another channel. Activities associated with a channel can be collected in an activity feed for that channel. Users other than the channel owner can subscribe to one or more channels that interest them. The concept of "subscribing" is sometimes referred to as "liking," "following," or "becoming friends."

[0033] In some embodiments, the system 100 may include one or more third-party platforms (not shown). In some embodiments, the third-party platforms may provide other services associated with the media content items 114. For example, the third-party platform may include an advertising platform that can provide video and / or audio advertisements, which may be uploaded to the secondary content server 132 described below. In other examples, the third-party platform may be a video streaming service provider that produces media streaming services via a communication application for users to play videos, TV shows, video clips, audio, audio clips, and movies on client devices 102A-102N.

[0034] The server machine 120 includes a timeline server 122 that can provide server-side timeline management for the streaming platform. The timeline server 122 includes a media stream timeline 124, which defines the order and relative positions of primary and secondary media content items within the media stream, the current playback location of the timeline, and other characteristics related to the media stream. The timeline will be further described with reference to Figures 3A and 3B. The timeline server 122 may include multiple media stream timelines 124, for example, one for each client device 102A to 102N, or one for each active stream managed by the streaming server 112. In some embodiments, a single media stream timeline 124 may be associated with multiple client devices 102A to 102N. For example, a live TV program (e.g., a sporting event or a music concert) may have a timeline that all users follow.

[0035] The timeline server 122 may communicate with the streaming server 112 and / or client devices 102A-102N to provide information related to the media stream timeline 124 or to update the media stream timeline 124. For example, the streaming server 112 may query the timeline server 122 to determine the current playback position or the next media content item to fetch from the data store 106. In another example, the streaming server 112 may send a request to the timeline server 122 to return to the current playback position in response to a seek action. In yet another example, the timeline server 122 may send a copy of the media stream timeline 124 to a client device (e.g., client devices 102A-102N) in response to a request from a client device. In some embodiments, the streaming server 112 may function as a relay between client devices 102A-102N and the timeline server 122 (for example, to copy the timeline to the client in the above example), while the timeline server 122 may have direct access to client devices 102A-102N in other embodiments. In some embodiments, the timeline server 122 and / or the media stream timeline 124 may define permitted actions with respect to the timeline and / or block prohibited actions. For example, a user seek action may be permitted when the current playback position is within a primary content item, but a user seek action may not be permitted when the current playback position is within a secondary content item or when the seek action skips a secondary content item. In some embodiments, the timeline server 122 may communicate with other timeline servers (not shown) to duplicate and synchronize the media stream timeline 124 so that any timeline server can provide the media stream timeline 124 and associated data or metadata to the streaming server 112 and / or client devices 102A-102N.Timeline duplication may be beneficial in providing redundancy to maintain the integrity of the timeline, as well as in preventing tampering with the timeline by client devices 102A-102N and / or third parties.

[0036] The server machine 130 includes a secondary content server 132 that can select and / or provide secondary media content items for the streaming platform. The secondary content server 132 may contain one or more media content items 114 that may be secondary content items such as advertisements, announcements, and surveys. In some embodiments, the secondary content server 132 may select secondary media content items for a media stream in response to a request from the streaming server 112 or client devices 102A-102N. For example, the secondary content server 132 may conduct an advertisement auction or select a survey from a list of currently available surveys. The request may include specifications or limitations on the secondary media content item. For example, the request may specify the playback length or encoding format required for the advertisement. In some embodiments, the secondary content server 132 may supply secondary content items to the streaming server 112 or client devices 102A-102N. For example, the secondary content server 132 may supply the streaming server 112 to stream a particular advertisement to a client in response to a request for that advertisement from the streaming server 112. In some embodiments, the secondary content server 132 may be provided by a streaming platform, a third-party secondary content provider, or a combination thereof.

[0037] Server machine 140 includes a content delivery network 142 (which is also a content distribution network, i.e., a CDN) that can provide content distribution services to the streaming platform. In some embodiments, the CDN 142 may include multiple geographically distributed nodes (e.g., multiple server machines 140, multiple data centers) that are close to users such as client devices 102A-102N. These nodes may function as proxies or relays to a centralized server for the streaming service (e.g., server machines 110-130) and may provide functions such as caching media content items, timelines, and other data for the streaming service. The CDN 142 may offer benefits such as reduced latency experienced by users, improved network bandwidth utilization, reduced load on the centralized server, and redundancy in case of network interruptions or failure / overload of the centralized server or CDN nodes. For example, CDN142 may cache media content items streamed from streaming server 112 to a first user, and subsequent requests for the same media content items by other local users may be served by CDN142 instead of streaming server 112. In some embodiments, CDN142 may be transparent to users and / or streaming platforms, so that functions described herein as being performed by server machines 110-130 may also be performed by CDN142. For example, server machines 110-130 (or their state) may be replicated on one or more nodes of CDN142. In an additional example, a media stream timeline 124 may be replicated and synchronized between replica timeline servers 122 on one or more nodes of CDN142, thereby allowing the media stream to migrate between nodes (for example, when a node fails or a client moves to a new geographical location) while maintaining the integrity of the associated media stream timeline 124. Similarly, other aspects of the streaming platform may be replicated and synchronized across CDN142.In some embodiments, CDN142 may be provided by a streaming platform, a third-party CDN provider, or a combination thereof. In some embodiments, CDN142 may not be present, and all communication from client devices 102A-102N is directed to a centralized server (e.g., server machines 110-130).

[0038] In general, the functions described in the embodiments as being performed by the streaming platform and / or server machines 110-140 can also be performed by client devices 102A-102N in other embodiments. Furthermore, functionality assigned to a particular component can be performed by different components or multiple components working together. The streaming platform can also be accessed as a service provided to other systems or devices through a suitable application programming interface, and is therefore not limited to use in a website.

[0039] While the implementations of this disclosure have been described from the perspective of a streaming platform and users of a streaming platform accessing electronic documents, the implementations as a whole can be applied to any type of document or file. The implementations of this disclosure are not limited to electronic document platforms that provide users with tools for creating, editing, and / or viewing documents. Furthermore, embodiments of this disclosure are not limited to text objects or drawing objects and can be applied to other types of objects.

[0040] In embodiments of this disclosure, “User” may represent a single individual. However, in other embodiments of this disclosure, “User” may encompass a set of users and / or entities controlled by an automated source. For example, a set of individual users integrated as a community within a social network may be considered a “User.” In another example, an automated consumer may be an automated ingestion pipeline of a streaming platform.

[0041] In addition to the above description, users may be provided with controls that allow them to make choices regarding both whether and when the systems, programs, or functions described herein may enable the collection of user information (e.g., information about the user's social networks, social behavior, or activities, occupation, user preferences, or current location), and whether the user receives content or communications from the servers. Furthermore, data may be processed in one or more ways before it is stored or used so that personally identifiable information is removed. For example, a user's identity may be processed so that the user's personally identifiable information cannot be determined, or if location information is obtained (e.g., down to the city, zip code, or state level), the user's geographical location may be generalized so that the user's specific location cannot be determined. Thus, users may have control over what information about them is collected, how that information is used, and what information is provided to them.

[0042] Figure 2 shows an exemplary client device 200 according to at least one embodiment. Client device 200 may correspond to one or more of the client devices 102A to 102N in Figure 1. Client device 200 includes a media decoder 202, a media buffer 204, a media player 206 associated with a media player GUI 208, and media metadata 210 associated with a metadata GUI element 212. In some embodiments, client device 200 may include more, fewer, or different components than those shown in Figure 2. For example, client device 200 may include a processor, memory, storage, peripherals, and other components described in relation to Figure 6.

[0043] The media decoder 202 may be a hardware component, a software component, or a mixture of hardware and software components that decode media streams in various formats such as H.264 / AVC, VP9, ​​H.265 / HEVC, AV1, ACC, and MP3. In some embodiments, the media decoder 202 may support decoding a single format, and in other embodiments, the media decoder 202 may support decoding multiple formats. In some embodiments, the media decoder 202 may provide initialization or reset capability to change the active decoding format from a first format (e.g., AVC) to a second format (e.g., HEVC). In some embodiments, the media decoder 202 may support decoding multiple streams in parallel (potentially in different formats). For example, the media decoder 202 may support simultaneous decoding of video and audio streams. In some embodiments, the client device 200 may include multiple media decoders 202 for decoding multiple streams in parallel. In some embodiments, the media decoder 202 may provide decryption or other capabilities associated with removing DRM protection from a media stream, for example, using a key provided by a streaming platform. In some embodiments, DRM decryption may be provided by other components of the client device 200 (not shown in Figure 2).

[0044] The media buffer 204 may be a hardware buffer, a section of memory (e.g., physical memory or virtual memory), or other storage area for storing media stream data before presenting it to the user. In some embodiments, the media buffer 204 may store encoded stream data before it is decoded by the media decoder 202. In some embodiments, the media buffer 204 may store decoded stream data after it has been decoded by the media decoder 202. In some embodiments, the media buffer 204 may store data from multiple media streams. In some embodiments, the client device 200 may include multiple media buffers 204 for storing data from multiple media streams before presenting the media streams to the user.

[0045] The media player 206 may be a software component that manages the manner in which media streams are retrieved and presented. The media player 206 may communicate with a streaming platform (e.g., a streaming server 112 in Figure 1) to initiate streaming or request subsequent media segments. The media player 206 may also communicate with, or configure, a media decoder 202 and a media buffer 204 to decode and temporarily store streamed media segments before presenting them to the user. The media player 206 includes a media player GUI 208 for providing the user with playback of media streams and receiving user input for interacting with the media streams. The media player may also provide a non-graphical user interface, such as playback of audio streams and receiving user audio via an audio input device (e.g., a microphone) of the client device 200. In some embodiments, the media player 206 may be received from the streaming platform in response to the client device 200 initiating a request for a new media stream. In some embodiments, the client device 200 may include multiple media players 206. Each media player 206 may be associated with a different media decoder 202 or media buffer 204, and the media player GUI 208 may be displayed or hidden for each media player 206. For example, the first media player may be associated with a media buffer and media decoder for an AVC-encoded media stream, and the second media player may be associated with a media buffer and media decoder for an HEVC-encoded media stream. While an AVC-encoded media stream is being played, the first media player GUI may be displayed and the second media player GUI may be hidden, and the opposite may be true while an HEVC-encoded media stream is being played. The transition between media streams in similar or different formats will be further explained with reference to Figures 4A and 4B.

[0046] In some embodiments, a client device 200 can send a request to a streaming platform (e.g., a streaming server 112) to access a media content item (e.g., media content item 114). The request may be, for example, a request to start a media stream of the media content item, or a request for a media segment of the media stream. The streaming platform may identify the media content item of the request (e.g., in a data store 106) and provide access to the media content item via a media player GUI 208 of a media player 206 (which may be provided by the streaming platform). In some embodiments, the requested media content item may be generated by another client device 200 connected to the streaming platform. For example, a client device 200 may generate a video item (e.g., via an audiovisual component of the client device 200, such as a camera) for other users of the platform to access, and provide the generated video item to the streaming platform (e.g., over a network). In other or similar embodiments, the requested media content item may be generated using another device (e.g., separate from or different from client device 200) and transmitted to client device 200 (e.g., via a network, via a bus, etc.). As described above, client device 200 can provide the video item to the streaming platform (e.g., via a network) so that other users of the platform can access it. Other client devices can send requests to the streaming platform (e.g., via a network) to access the video item provided by client device 200, according to the examples described above.

[0047] In some embodiments, the media player GUI 208 of a media player 206 (which may be provided by a streaming platform) may include one or more GUI elements that allow a user to control the playback of a media stream via the media player 206. For example, the media player GUI 208 may include one or more GUI elements that allow a user to start playback of a media stream (e.g., a “Play” button) and / or pause playback of a media stream (e.g., a “Pause” button). In other examples, the media player GUI 208 may include one or more GUI elements (e.g., a “Next” button) that allow a user to start streaming other media content items related to the requested media content item. The media player GUI 208 may further include one or more GUI elements that show a timeline of the media stream and / or the progress of playback of media content items within the timeline. In some embodiments, the timeline shown by the GUI elements may correspond to a local copy of a server-based timeline associated with the media stream (e.g., media stream timeline 124 in Figure 1). Exemplary timelines are described further with reference to Figures 3A and 3B.

[0048] In some embodiments, media metadata 210 may be received from a streaming platform (e.g., from a streaming server 112) and may be associated with one or more media streams or media segments. Media metadata 210 may include information related to the media stream, such as the title, producer, and channel of media content items within the media stream. Media metadata 210 may also include timeline-related information, which the media player 206 may use to display the progress of playback of the media stream or media content items. Media metadata 210 may also include information about stream quality, such as changes in current network bandwidth or encoding quality. Media metadata 210 may also include instructions to cause the client device 200 to take actions related to the media stream, such as logging data or events associated with the media stream. Media metadata 210 may include various other data. In some embodiments, media metadata 210 may indicate to the client device 200 that the current media segment is part of a primary media content item or a secondary media content item. This may be useful when, for example, the client device 200 cannot distinguish between primary and secondary media content by server-side stitching and encapsulation techniques described with reference to Figures 4A and 4B. In some embodiments, the client may retrieve one or more metadata GUI elements 212 from local or remote storage and present them in association with primary or secondary media content as specified in the media metadata 210. For example, the GUI element(s) may be a “Skip Ad” button, a set of radio buttons for a survey, or one or more hyperlinks linking to other media content. In some embodiments, the media metadata 210 may include one or more metadata GUI elements 212 presented to the client.For example, media metadata 210 may include HTML, CSS, XML, JavaScript, etc., which are inserted into the GUI of the client device 200.

[0049] Figures 3A and 3B show exemplary media stream timelines 300 and 320 for initiating primary and secondary media content on the server side, according to at least one embodiment. The exemplary media stream timelines 300 and 320 may be server-side timelines maintained on a streaming server (e.g., streaming server 112 in Figure 1), a timeline server (e.g., timeline server 122), a CDN (e.g., CDN 142), or elsewhere within the streaming platform.

[0050] Referring to Figure 3A, the exemplary media stream timeline 300 represents primary media content items 302A-302B. In some embodiments, the media stream timeline may represent a single primary media content item 302A-302B that can be divided into two or more parts. For example, 302A may be the first part of a primary media content item before an advertisement, and 302B may be the second part following the advertisement. In some embodiments, the media stream timeline may include multiple primary media content items 302A-302B. For example, a live TV channel may have a first program (302A) before a commercial break and a second program (302B) after a commercial break. In an additional example, a playlist may include multiple videos, short videos, songs, etc., represented by two or more primary media content items 302A-302B. The media stream timeline 300 further includes secondary media content positions, namely secondary media content pre-roll 304, secondary media content mid-roll 306, and secondary media content post-roll 308. Each of the secondary media content positions 304-308 may include one or more secondary media content items (or the positions of those items), as described with respect to Figure 3B. In some embodiments, the media stream timeline may have fewer than the three illustrated secondary media content positions, such as one or two of the pre-roll, mid-roll, and post-roll media content positions. In some embodiments, the media stream timeline may have more than the three illustrated secondary media content positions, such as an additional secondary media content mid-roll position 306 that divides a primary media content item(s) into three or more parts. The media stream timeline 300 may indicate transitions between primary and secondary media content in transitions 310A-310D, etc. Transitions between primary and secondary media content relate to server-side playback of the media stream, as described with respect to Figures 4A-4B.

[0051] The media stream timeline 300 may indicate the current playback position 312 of the stream with respect to the stream global playback time 314 and / or content local playback time 316. Playback position, playback time, transition time (e.g., 310A-310D), and other information may be stored as data or metadata associated with the media stream timeline 300. In some embodiments, this information may be provided to a client device and used to render a global progress bar or a local progress bar, as described with respect to Figure 2. The media stream timeline 300 may further describe permitted or prohibited user / client actions with respect to the timeline, such as seeking or skipping within media content (e.g., in 318). In some embodiments, this information may be provided to a client device and used to render available user interactions. The streaming platform may also use this information in response to client requests, such as updating the timeline or providing media segments from different playback positions on the timeline, to determine whether such requests are permitted or not. In some embodiments, the streaming platform may automatically update the media stream timeline 300 when there is no client activity. For example, secondary media content (e.g., pre-roll 304) may be automatically removed from the media stream timeline 300 after the secondary media content has been fully played, preventing the secondary media content from being played again if the user seeks back to the start time of the secondary media content or a time earlier than the start time.

[0052] Referring to Figure 3B, the exemplary media stream timeline 320 includes a primary media content item 322 and a mid-roll 324 of secondary media content. The media stream timeline may include multiple secondary media content items consecutively at pre-roll, mid-roll, or post-roll positions, etc. The mid-roll 324 clarifies various timeline characteristics related to the secondary media content. In some embodiments, the mid-roll and other secondary media content positions may contain more or less secondary media content than shown in Figure 3B. In some embodiments, the example of the mid-roll 324 may be applied to primary media content.

[0053] The mid-roll 324 includes a skipped secondary media content item 326. The secondary media content item may be skipped, for example, by an authorized user seek action on the client device. In some embodiments, metadata associated with the media stream (e.g., media metadata 210 in Figure 2) may include a “skip” GUI element that allows the user to skip secondary media content items. In some embodiments, in response to a secondary media content item being skipped, the streaming platform may update the timeline by moving the current playback position to the next primary or secondary media content item (e.g., the current playback position 334 within a fixed-length secondary media content item 328). The streaming platform may remove the skipped secondary media content item 326 from the timeline to prevent the skipped secondary media content item 326 from being repeated when the user later seeks to a position in the timeline prior to the skipped secondary media content item 326. In some embodiments, in response to a secondary media content item being skipped, the streaming platform may replace the skipped secondary media content item 326 with another secondary media content item and stream the replacement instead (not shown in Figure 3B). For example, a live TV stream may specify a fixed length of time for commercials that include one or more advertisements. If a user skips an advertisement, the streaming platform may replace the skipped advertisement with another advertisement to maintain the fixed length of the commercial time.

[0054] Mid-roll 324 includes fixed-length secondary media content items 328. Examples of fixed-length secondary media content items include fixed-length advertisements and announcements, as well as surveys, links, and other static content that may have time limits. Seek actions may or may not be permitted in fixed-length secondary media content items.

[0055] The mid-roll 324 includes a secondary media content position 330. In some embodiments, the media stream timeline may specify one or more positions to be filled with fixed-length or variable-length secondary media content items without first selecting a secondary media content item. In some embodiments, the secondary content media positions may be filled with media content items selected by the streaming platform. The selection may occur during the time period between the initialization of the media stream timeline and the presentation of the secondary media content positions. For example, the streaming platform may query an ad server (e.g., secondary content server 132 in Figure 1) and initiate an ad auction to select an ad. The ad auction and / or selection may occur when the current playback position is close enough to a secondary media content position (e.g., when the current playback position 334 is within a previous secondary media content item, a fixed-length secondary media content item 328). In some embodiments, the secondary media content positions may be filled based on the characteristics of the user or client device. For example, a live TV media stream viewed by many users may specify secondary media content locations (e.g., fixed-length commercial breaks) that all users share, and these locations may be filled with secondary media content items unique to each user based on their interests. In an additional example, secondary media content locations may be filled based on the decoding capabilities of the client device. For instance, a client device may not support decoding a first format used by a first secondary media content item, so the stream may select an alternative secondary media content item encoded in a second format supported by the client device.

[0056] Mid-roll 324 includes variable-length secondary media content items 332. Examples of variable-length secondary media content items include surveys, links to other media content, and static content that may have no time limit or static content whose presentation time may change based on user interaction. Seek actions may or may not be permitted for variable-length secondary media content items.

[0057] Using a server-side media stream timeline to initiate primary and secondary media content items on the server side can offer several advantages. By initiating ad auctions, selecting primary and secondary media content items, retrieving media content items, and performing other initiation activities on the server side, client devices may be able to reduce the demands placed on the streaming platform. This can reduce latency experienced by the user (e.g., fewer round-trip communication cycles), reduce the computational load on the client device, and improve network bandwidth utilization. Similarly, the streaming platform may reduce the load on secondary content servers by reusing secondary media content items or their selections from previous calls to the secondary content server (e.g., by serving the same advertisement to multiple viewers of a live TV program). Furthermore, performing initiation activities on the server side can improve the user experience because it reduces the ability of ad blockers and other software on client devices to interfere with the normal functionality of the streaming platform. Streaming platform developers may also experience increased efficiency and spend less time maintaining and debugging multiple device-specific implementations of client-side initiation activities. For example, a streaming platform developer could focus on maintaining a single server-side implementation of the ad auction rather than separate implementations for web browsers, smartphone applications, and smart TVs. Improved developer efficiency could then translate to benefits in user experience, computing efficiency, and network efficiency.

[0058] Figures 4A and 4B show exemplary media streams 400 and 420 for playing primary and secondary media content on the server side, according to at least one embodiment. The exemplary media streams 400 and 420 explicitly combine primary and secondary media content items on a streaming platform (e.g., streaming server 112 in Figure 1). This may correspond to sections of the media stream timeline where primary media content items transition to secondary media content items (e.g., as shown in 310B in Figure 3A). The exemplary media streams 400 and 420 may also be applied to other media content transitions, such as primary to primary, secondary to primary, and secondary to secondary. In some embodiments, a complete media stream may include multiple transitions, which may be a mixture of transitions as shown in Figures 4A and 4B.

[0059] Referring to Figure 4A and the exemplary media stream 400, media segments 402 and 404 can be joined at transition 406. Media segment 402 may be a media segment of a primary media content item and may be in a first format (e.g., AVC). Media segment 404 may be a media segment of a secondary media content item and may also be in the first format. The streaming platform may join at transition 406 by appending the data of media segment 404 to the media stream after the data of media segment 402. In some embodiments, this may be called server-side stitching of the media stream. Thus, the media stream may appear to both the server and the client as a continuous stream of media segments in the first format. From the client's perspective (e.g., client device 200 in Figure 2), media segments 402 and 404 are decoded as if they originated from the same content item, without the need to reset the media decoder or switch media players between media segments. In some embodiments, as illustrated with reference to Figure 2, the client may rely on metadata to determine which media content item (e.g., primary or secondary) each media segment is associated with. In some embodiments, the client may not need or allow associating media segments with media content items, and therefore, that information may be stored only within the server.

[0060] Referring to Figure 4B and the exemplary media stream 420, media segments 422 and 424 may be joined at transition 428. Media segment 422 may be the media segment of a primary media content item and may be in a first format (e.g., AVC). Media segment 424 may be the media segment of a secondary media content item and may be in a second format (e.g., HEVC). The streaming platform may join at transition 428 by appending an initialization segment 426 in 430 before media segment 424 and appending the data of media segment 424 and initialization segment 426 to the media stream after the data of media segment 422. In some embodiments, this may be called server-side encapsulation of the media stream. In some embodiments, initialization segment 426 may contain instructions or data to decode the media segment in a second format instead of a first format, such that the media decoder of a client device (e.g., media decoder 202 in Figure 2) is reset or reinitialized. In some embodiments, initialization segment 426 may include instructions or data to decode the media segment in a second format instead of a first format, such that the client device's media player (e.g., media player 206 in Figure 2) is reset or reinitialized. In some embodiments, initialization segment 426 may include instructions to switch the active decoder and / or media player of the client device. For example, initialization segment may cause the client device to move the media player for the first format to the background and the media player for the second format to the foreground. As described with respect to Figure 4A, the client device may use metadata or other information to determine which media content item each media segment corresponds to, if such information is permitted and / or useful to the client.

[0061] In some embodiments, the streaming platform may apply DRM protection to the exemplary media streams 400 and 420 before or after the media segments (and, if applicable, the initialization segments) are combined. For example, the streaming platform may generate a DRM encryption key and encrypt the combined media streams before streaming them to the client. The streaming platform may send a complementary DRM decryption key to the client to remove the DRM protection before presenting the media streams to the user (for example, in the media decoder 202 in Figure 2).

[0062] As described above, a streaming platform may provide live media streams and / or on-demand media streams, and the methods for combining primary and secondary media content items, as described with respect to exemplary media streams 400 and 420, may offer different advantages to live and on-demand media streams. A live media stream (e.g., a live sports broadcast or a music concert) may have many concurrently present clients (e.g., thousands or millions of viewers), all of whom may follow the same or similar media stream timeline. For example, clients may consume secondary media content simultaneously, and may consume the same or different secondary media content items (e.g., secondary media content items may be customized for each client). Furthermore, a live media stream may be encoded in a single format streamed to all clients. Therefore, it may be beneficial for a streaming platform to identify all potential secondary media content items (in various formats) that may be streamed during a live stream and to pre-transcode the content items into the same format as the media stream. A streaming platform may use the method shown in Figure 4A for server-side playback of live streams and associated secondary media content items, which can offer the benefits of reduced client-side latency and decreased server and network load. Furthermore, a streaming platform may use a CDN (e.g., CDN142 in Figure 1) to cache the same media stream shared among many clients, thereby further reducing server and network load.

[0063] In contrast to live media streams, on-demand media streams may have fewer simultaneous viewers, and on-demand content items may vary significantly in format. As illustrated with respect to Figure 2, on-demand content items may be uploaded to the streaming platform by various clients. On-demand content collections may be developed over long periods (e.g., several years), and may include older content items using older encoding formats as well as newer content items using the latest encoding formats. Transcoding all on-demand and / or secondary media content items for all media streams to maintain a consistent format within the media stream may not be practical, as the computational and storage requirements can be exorbitant. Therefore, in some embodiments, the streaming platform may use a combination of server-side playback methods, as described with respect to exemplary media streams 400 and 420, to optimize client latency, server storage and computation, and network load. For example, the streaming platform may transcode some secondary media content items to support server-side stitching for some on-demand media streams and fall back to server-side encapsulation for other on-demand media streams. In some embodiments, server-side stitching, server-side encapsulation, or a combination of these methods may be useful for other types of media streams besides live and on-demand streams and may offer different advantages.

[0064] Server-side playback of primary and secondary media content can offer further advantages to streaming platform developers. By migrating content on the server rather than to each client, developers may only need to create and maintain migration logic for the server, rather than for each of the various client devices supported by the streaming platform. Consequently, developers may experience increased productivity due to reduced software maintenance requirements, and the user experience may be improved by reducing the proliferation of software buses and minimizing inefficiencies caused by client-specific implementations in traditional systems.

[0065] Figure 5A is a flowchart of an exemplary method 500 for providing server-side initiation and playback of primary and secondary media content on a streaming platform, according to at least one embodiment. Method 500 can be implemented by processing logic that may include hardware (circuitry, dedicated logic, etc.), software (e.g., instructions executed on a processing device), or a combination thereof. In at least one implementation, some or all of the operations of Method 500 can be performed by one or more components of the streaming platform system architecture 100 shown in Figure 1.

[0066] For the sake of simplicity, the methods disclosed herein are shown and described as a series of actions. However, the actions relating to this disclosure may occur in various orders and / or simultaneously, and in conjunction with other actions not shown and described herein. Furthermore, not all illustrated actions are necessary to carry out the methods in accordance with the disclosed subject matter. Furthermore, those skilled in the art will understand and recognize that the methods may be alternatively represented as a series of interrelated states, for example, by a state diagram. Furthermore, it should be understood that the methods disclosed herein are storeable in a product to facilitate the transmission and transfer of such methods to computing devices. The term "product," as used herein, is intended to encompass computer programs accessible from any computer-readable device or storage medium.

[0067] In block 502, the processing logic receives a request for a media segment of a media stream to be delivered to a client on the server. In some embodiments, the request for a media segment may be received by the streaming server 112, the CDN 142, or a combination of the server machines 110-140 in Figure 1. The request may also be received from a client device, such as one of the client devices 102A-102N. The request for a media segment may be independent of the type of media content being delivered (e.g., primary or secondary) or may specify the type of media content. The request for a media segment may include timeline-related data such as the start and end times (or length) of the media segment to be delivered, the end time of the last media segment in the local buffer, and indications of user or non-user seek activity (e.g., skipping secondary media content items). The request may include other data and metadata related to, for example, secondary media streams (e.g., audio or captions), client capabilities (e.g., supported codecs), network or device status, etc. In some embodiments, the request may be a first request for a media segment. The request may specify the primary media content item to be streamed, causing the streaming platform to retrieve the primary media content item (e.g., from datastore 106) and create a server-side timeline (e.g., on timeline server 122). In some embodiments, the request may be a subsequent request for a media segment, following a previous request for a media segment.

[0068] In block 504, the processing logic uses a server-based timeline of the media stream to determine whether the requested media segment is intended to contain primary or secondary media content, and the timeline defines one or more locations within the media stream to contain secondary media content. In some embodiments, the server-based timeline may be the media stream timeline 124 of the timeline server 122 in Figure 1, or one of the media stream timelines 300 and 320 in Figures 3A and 3B. The streaming server 112 or the timeline server 122 may query the timeline to determine the current playback position and whether the media content at that position is primary or secondary media content. The media content at the current playback position may be a media content item or a placeholder for a media content item that has not yet been determined. Furthermore, the timeline may include one or more placeholders for one or more locations within the media stream to contain secondary media content. In some embodiments, additional data from the request in block 502 may be used to query the timeline. For example, a request may specify a new current playback position or a user seek action, which can be used to update the timeline before determining whether the media content at the current playback position is primary or secondary media content. In some embodiments, a server-based timeline is synchronized across multiple timeline servers, such as timeline servers replicated on one or more CDN nodes.

[0069] In block 506, in response to determining that the requested media segment corresponds to one of one or more locations defined in the timeline, the processing logic retrieves a secondary media content item from the server. In some embodiments, the server (e.g., streaming server 112) may retrieve the secondary media content item from datastore 106. In some embodiments, the server may retrieve the secondary media content item from a secondary media content server, such as secondary content server 132 in Figure 1. The server may request the secondary media content server to select one of several secondary media content items. For example, the server may request the secondary content server to perform an ad auction to select an ad from a collection of ads, or the server may request the secondary content server to select a survey from a list of currently available surveys. In some embodiments, the server may receive an indication (e.g., content ID, pointer to content) of the selected secondary media content item from the secondary media content server. The server may then retrieve the selected secondary media content item from datastore 106 or another location. In some embodiments, the server may receive the secondary media content item from the secondary media content server. In some embodiments, the server can reuse selected secondary media content items or their indications from previous requests to the secondary media content server, thereby reducing the load on the secondary media content server.

[0070] In block 508, the processing logic adds the media segment of the secondary media content item to the media stream in order to provide the requested media segment to the client. In some embodiments, the added media segment may be multiple bytes, packets (e.g., network packets), or other types of data. In some embodiments, the added media segment may be encoded in a streaming-compatible format such as H.264 / AVC, VP9, ​​H.265 / HEVC, AV1, ACC, or MP3.

[0071] In some embodiments, the media segment preceding the media stream is associated with a first format, and the media segment of a secondary media content item added to the media stream is also associated with the first format. Adding the media segment of a secondary media content item to the media stream may further include (as described with respect to Figure 4A, for example) appending the media segment of the secondary media content item to the preceding media segment to form a series of media segments in the first format, and a client decoder (e.g., media decoder 202 in Figure 2) configured to decode the preceding media segment in the first format will also decode the media segment of the secondary media content item in the first format without reinitializing the decoder.

[0072] In some embodiments, the media segment preceding the media stream is associated with a first format, and the media segment of a secondary media content item added to the media stream is associated with a second format. Adding the media segment of a secondary media content item to the media stream may further include prepending an initialization segment before the media segment of the secondary media content item. The media segment of the secondary media content item having a previously prepended initialization segment may be prepended to the preceding media segment (as described, for example, with respect to Figure 4B) to form a series of consecutive media segments. In some embodiments, the initialization segment causes the client to reinitialize the client's decoder (e.g., media decoder 202 in Figure 2) to decode the second format, which was previously configured to decode the first format. In some embodiments, the initialization segment causes the client to start the client's second decoder to decode the second format. In some embodiments, the initialization segment causes the client to reinitialize the media player, or initialize a second media player (e.g., media player 206 in Figure 2) to play the media segment of the secondary media content item in the second format.

[0073] In some embodiments, the processing logic may apply DRM protection to a media segment or media stream. For example, the processing logic may generate a DRM encryption key and encrypt the media stream before providing the requested media segment to the client. The processing logic may send a complementary DRM decryption key to the client to remove the DRM protection before presenting the media stream to the user.

[0074] In block 510, the processing logic sends metadata associated with a secondary media content item to the client, which corresponds to one or more graphical user interface (GUI) elements presented to the client by the secondary media content item. In some embodiments, since the client may not be able to distinguish between primary and secondary media content in a media stream, the metadata may indicate to the client that the media segment provided in block 508 is a segment of a secondary media content item. The client may retrieve one or more GUI elements from local or remote storage and present them in association with the secondary media content. For example, the GUI elements may include a "skip ad" button, a set of radio buttons for a survey, or one or more hyperlinks linking to other media content. In some embodiments, the metadata may include the GUI elements presented to the client. For example, the metadata may include HTML, CSS, XML, JavaScript, etc., which are inserted into the GUI of the client device. In some embodiments, the metadata may cause the client to take actions such as logging events in association with the secondary media content item.

[0075] In block 512, the processing logic sends a media stream timeline to the client, which indicates one or more permitted seek actions available to the client's user. In some embodiments, as described with reference to Figure 1, the timeline may be a media stream timeline 124, and the timeline may be sent from a streaming server 112 or a timeline server 122. The timeline sent to the client may be a copy of the media stream timeline 124, or may contain more or less information. In some embodiments, the timeline may indicate one or more permitted seek actions or other actions (e.g., skip, pause, forward seek, backward seek, double speed) with respect to each primary and secondary media content item in the timeline. Permitted seek actions may also relate to other content items in the timeline. For example, a permitted seek action may be to seek forward, but only to the next secondary media content item. In some embodiments, the timeline may indicate one or more unauthorized seek actions, such as seeking forward during a secondary media content item.

[0076] Figure 5B is a flowchart of an exemplary method 520 for providing server-side initiation and playback of primary and secondary media content in a streaming platform, according to at least one embodiment. Method 520 can be carried out by processing logic that may include hardware (circuitry, dedicated logic, etc.), software (e.g., instructions executed on a processing device), or a combination thereof. In at least one implementation, some or all of the operations of Method 500 can be carried out by one or more components of the streaming platform system architecture 100 of Figure 1. In some embodiments, Method 520 may be carried out in response to having performed block 512 in Figure 5A.

[0077] In block 522, the processing logic receives an indication of an authorized seek action performed by the client user on the server. For example, the indication might specify that the user skipped a secondary media content item or jumped forward for a certain period of time. In some embodiments, the indication of an authorized seek action may be received along with a request for a media segment (for example, by block 526 below).

[0078] In block 524, the processing logic updates the server-based timeline of the media stream based on indications of authorized seek actions performed by the client user. For example, the processing logic may move the current playback position of the server-based timeline forward or backward to reflect the seek action. In another example, the timeline may allocate a fixed time to secondary media content at a certain position, and an authorized action may skip the first selected secondary media content item at that position. Therefore, updating the server-based timeline may involve selecting a new secondary media content item to fill that position rather than seeking beyond it.

[0079] In block 526, the processing logic receives a request on the server for a secondary media segment of the media stream to be delivered to the client. The request received in block 526 may have similar characteristics to those described for the request received in block 502. As described for block 522, the request may also indicate an authorized seek action that may require a subsequent update to the server-based timeline.

[0080] In block 528, the processing logic uses an updated server-based timeline of the media stream to determine whether the requested second media segment is intended to contain primary media content or secondary media content. The decision in block 528 may be similar to the decisions described in relation to block 504, but using the updated timeline. Method 520 may proceed to retrieve the secondary media content and add the media segment to the media stream, as described in relation to blocks 506 and 508, and may further receive subsequent requests for additional media segments.

[0081] Figure 5C is a flowchart of an exemplary method 540 for providing server-side initiation and playback of primary and secondary media content in a streaming platform, according to at least one embodiment. Method 540 can be carried out by processing logic that may include hardware (circuitry, dedicated logic, etc.), software (e.g., instructions executed on a processing device), or a combination thereof. In at least one implementation, some or all of the operations of Method 500 can be carried out by one or more components of the streaming platform system architecture 100 of Figure 1. In some embodiments, Method 540 may be carried out in response to having performed block 512 in Figure 5A.

[0082] In block 542, the processing logic receives an indication on the server of an unauthorized seek action performed by the client user. For example, the indication might specify that the user attempted to skip a secondary media content item or jumped forward beyond an permitted time range. In some embodiments, the indication of an unauthorized seek action may be received along with a request for a media segment (for example, by block 544 below).

[0083] In block 544, the processing logic receives a request on the server for a secondary media segment of the media stream to be delivered to the client. The request received in block 544 may have similar characteristics to those described with respect to the request received in block 502. As described with respect to block 542, the request may also indicate an unauthorized seek action.

[0084] In block 546, the processing logic uses the server-based timeline of the media stream to determine whether the requested second media segment is intended to contain primary media content or secondary media content, regardless of any indication of unauthorized seek actions performed by the client user. The decision in block 546 may be similar to the decision described in block 504, since unauthorized seek actions do not result in updates to the server-based timeline. Method 540 may proceed to retrieve the secondary media content and add the media segment to the media stream, as described in blocks 506 and 508, and may further receive subsequent requests for additional media segments.

[0085] Figure 6 is a block diagram showing an exemplary computer system 600 according to an embodiment of the present disclosure. The computer system 600 may correspond to client devices 102A-102N and / or server machines 110-140 as described with respect to Figure 1. The computer system 600 may also correspond to client device 200 as described with respect to Figure 2. The computer system 600 may operate within the capacity of a server or endpoint machine in an endpoint server network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. A machine may be a television, personal computer (PC), tablet PC, set-top box (STB), personal digital assistant (PDA), mobile phone, web appliance, server, network router, switch or bridge, or any machine capable of executing (sequentially or otherwise) a set of instructions that specify the actions that the machine takes. Furthermore, although only a single machine is shown, the term “machine” shall also be interpreted as including any collection of machines that individually or collectively execute one or more sets of instructions to perform any one or more methodologies of the methodologies described herein.

[0086] The illustrated computer system 600 includes a processing device (processor) 602, main memory 604 (e.g., read-only memory (ROM), flash memory, synchronous DRAM (SDRAM), double data rate (DDR SDRAM), or dynamic random access memory (DRAM) such as DRAM (RDRAM)), static memory 606 (e.g., flash memory, static random access memory (SRAM), etc.), and a data storage device 618, which communicate with each other via a bus 640.

[0087] The processor (processing device) 602 represents one or more general-purpose processing devices, such as a microprocessor or a central processing unit. More specifically, the processor 602 may be a composite instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, or a processor implementing another instruction set, or a processor implementing a combination of instruction sets. The processor 602 may also be one or more dedicated processing devices, such as an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital signal processor (DSP), or a network processor. The processor 602 is configured to execute instructions 605 for performing the operations described herein (for example, for providing an automated rule generation and data-driven detection engineering system).

[0088] The computer system 600 may further include a network interface device 608. The computer system 600 may also include a video display unit 610 (e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)), an input device 612 (e.g., a keyboard, an alphanumeric keyboard, a motion-sensing input device, a touchscreen), a cursor control device 614 (e.g., a mouse), and a signal generation device 620 (e.g., a speaker). In some embodiments, the computer system 600 may not include the video display unit 610, the input device 612, and / or the cursor control device 614 (e.g., in a headless server configuration).

[0089] The data storage device 618 may include a non-temporary machine-readable storage medium 624 (which is also a computer-readable storage medium) that stores one or more sets of instructions 605 that embody any one or more of the methodologies or functions described herein (e.g., for providing automatic rule generation and data-driven detection engineering systems). The instructions may reside entirely or at least partially in the main memory 604 and / or the processor 602 while being executed by the computer system 600, and the main memory 604 and the processor 602 also constitute a machine-readable storage medium. The instructions may be further transmitted or received via the network 630 through the network interface device 608.

[0090] In one embodiment, instruction 605 includes instructions for providing an automated rule generation and data-driven detection engineering system. Although the computer-readable storage medium 624 (machine-readable storage medium) is shown as a single medium in the exemplary embodiment, the terms “computer-readable storage medium” and “machine-readable storage medium” should be interpreted to include a single or multiple mediums (e.g., a centralized or distributed database, and / or associated caches and servers) that store one or more instruction sets. The terms “computer-readable storage medium” and “machine-readable storage medium” should also be interpreted to include any medium that has the ability to store, encode, or execute instruction sets for machine execution, causing a machine to perform one or more of the methods of the Disclosure. Accordingly, the terms “computer-readable storage medium” and “machine-readable storage medium” should be interpreted to include, but are not limited to, solid-state memory, optical media, and magnetic media.

[0091] Throughout this specification, any reference to “one implementation,” “one embodiment,” “an implementation,” or “an embodiment” means that any particular feature, structure, or characteristic described in relation to an embodiment is included in at least one embodiment and / or embodiment. Therefore, while the phrases “in one implementation” or “in an implementation” appear in various places throughout this specification, they may, depending on the context, refer to the same embodiment, but not necessarily. Furthermore, any particular feature, structure, or characteristic can be combined in any preferred manner in one or more embodiments.

[0092] To the extent that the terms “includes,” “including,” “possess,” “contain,” their variations, and other similar terms are used in either embodiments or claims to describe the invention, these terms are intended to be inclusive in the same manner as the term “equip,” without excluding any additional or other elements.

[0093] As used in this application, terms such as “component,” “module,” and “system” are intended to generally refer to computer-related entities, either hardware (e.g., circuits) or software, a combination of hardware and software, or entities relating to an operable machine with one or more specific functionalities. For example, a component may be, but is not limited to, a process running on a processor (e.g., a digital signal processor), a processor, an object, an executable file, an execution thread, a program, and / or a computer. As an example, an application running on a controller, and both the application and the controller, may be components. One or more components may reside in an execution process and / or thread, and components may be localized on one computer and / or distributed across two or more computers. Furthermore, a “device” may be provided in the form of specially designed hardware, general-purpose hardware specialized by the execution of software that enables the hardware to perform a specific function (e.g., generating points of interest and / or descriptors), software on a computer-readable medium, or a combination thereof.

[0094] The aforementioned systems, circuits, modules, etc., are described in relation to interactions between multiple components and / or blocks. Such systems, circuits, components, blocks, etc., may include their components or designated subcomponents, parts of designated components or subcomponents, and / or additional components, and it can be recognized that they follow the various permutations and combinations described above. Furthermore, subcomponents may be implemented not as being contained within a parent component (hierarchical), but as components communicatively coupled with other components. It should also be noted that one or more components may be combined into a single component providing an integrated function, or divided into several individual subcomponents, and communicatively coupled with such subcomponents to provide integrated functionality by providing one or more intermediate layers, such as a management layer. Additionally, any component described herein may interact with one or more other components known to those skilled in the art, but not specifically described herein.

[0095] Furthermore, the terms “example” or “exemplary” are used herein in the sense of serving as an example, case, or illustration. No aspect or design described herein as “exemplary” is necessarily construed to be preferable or advantageous to another aspect or design. Rather, the use of the terms “example” or “exemplary” is intended to concretely present a concept. As used in this application, the term “or” is intended to mean inclusive “or” rather than exclusive “or.” That is, unless otherwise specified or it is clear from the context, “X adopts A or B” is intended to mean either of the natural inclusive substitutions. That is, if X adopts A, if X adopts B, or if X adopts both A and B, “X adopts A or B” satisfies the condition of any of the aforementioned examples. Furthermore, the articles “a” and “an” as used in this application should generally be construed to mean “one or plural” unless otherwise specified or it is clear from the context that they refer to a singular form.

[0096] Finally, the embodiments described herein include the collection of data describing the user and / or the user's activities. In one embodiment, such data is collected only if the user consents to this data collection. In some embodiments, the user is explicitly prompted to allow data collection. Furthermore, the user can opt in or opt out of participating in such data collection activities. In one embodiment, the collected data is anonymized before any analysis is performed to obtain any statistical patterns, so that the user's specific information cannot be determined from the collected data.

Claims

1. On the server, receiving requests for media segments of media streams to be delivered to clients, Using a server-based timeline of the media stream, determine whether the requested media segment is intended to contain primary media content or secondary media content, wherein the timeline defines and determines one or more locations within the media stream for containing the secondary media content. In response to determining that the requested media segment corresponds to one of the one or more locations defined on the timeline, the server retrieves a secondary media content item. A method comprising adding the media segment of the secondary media content item to the media stream in order to provide the requested media segment to the client.

2. The method according to claim 1, further comprising transmitting metadata associated with the secondary media content item to the client, wherein the metadata corresponds to one or more graphical user interface (GUI) elements presented to the client by the secondary media content item.

3. The method according to claim 1 or 2, further comprising transmitting the timeline of the media stream to the client, wherein the timeline of the media stream indicates one or more permitted seek actions available to the client's users.

4. The server receives an indication of an authorized seek action performed by the user of the client. Updating the server-based timeline of the media stream based on the indication of the authorized seek action performed by the user of the client, The server receives a request for a second media segment of the media stream to be delivered to the client, The method according to claim 3, further comprising using the updated server-based timeline of the media stream to determine whether the requested second media segment is for primary media content or for secondary media content.

5. The server receives an indication of an unauthorized seek action performed by the user of the client, The server receives a request for a second media segment of the media stream to be delivered to the client, The method according to claim 3 or 4, further comprising using the server-based timeline of the media stream to determine whether the requested second media segment is for primary media content or secondary media content, regardless of the indication of the unauthorized seek action performed by the user of the client.

6. Acquiring the aforementioned secondary media content item means The aforementioned server requests the secondary media content server to select one of several secondary media content items, The method according to any one of claims 1 to 5, further comprising receiving the secondary media content item from the secondary media content server.

7. The method according to any one of claims 1 to 6, wherein the media segment preceding the media stream is associated with a first format, the media segment of the secondary media content item is associated with a second format, and adding the media segment of the secondary media content item to the media stream further includes adding an initialization segment before the media segment of the secondary media content item, the initialization segment causing the client to reinitialize the client's decoder to decode the second format.

8. The method according to any one of claims 1 to 7, wherein the server-based timeline is synchronized with a plurality of timeline servers.

9. Memory devices and, A system comprising a processing device coupled to the memory device, wherein the processing device performs the method according to any one of claims 1 to 8.

10. A non-temporary computer-readable storage medium, which, when executed by a processing device, includes an instruction causing the processing device to perform the method according to any one of claims 1 to 8.

Images