Event progress detection in media items

Abstract

The invention relates to event progress detection in media items. One or more frames sampled from media items of events simultaneously streamed to one or more users of a content delivery platform are received. The one or more frames are analyzed to identify one or more candidate event clocks within the one or more frames. For each frame of the one or more frames, it is determined whether a candidate event clock of the one or more candidate event clocks satisfies one or more conditions. In response to determining that the candidate event clock of the one or more candidate event clocks satisfies the one or more conditions, the candidate event clock is identified as an actual event clock for timing the event, and generate mapping data that maps the actual event clock to timestamps associated with respective ones of the one or more frames of the media item.

Description

[0001] Description of the case

[0002] This application is a divisional application of Chinese invention patent application No. 201980087636.5, with an application date of August 5, 2019. Technical Field

[0003] Aspects and implementations of the present disclosure relate to content delivery platforms and, more particularly but not by way of limitation, to event progress detection in media items such as live streaming media items. Background Art

[0004] Content delivery platforms connected via the Internet enable users to connect with each other and share information. Many content delivery platforms include content sharing aspects that enable users to upload, view and share content such as video items, image items, audio items, etc. Other users of the content delivery platform can comment on the shared content, discover new content, locate updates, share content, and otherwise interact with the provided content. Shared content can include content from professional content creators, such as movie clips, television clips, and music video items, as well as content from amateur content creators, such as video blogs and original short video items. Summary of the invention

[0005] The following is a brief overview of the disclosure in order to provide a basic understanding of some aspects of the disclosure. This overview is not a broad overview of the disclosure. It is not intended to identify key or critical elements of the disclosure, nor to limit any scope of a particular implementation of the disclosure or any scope of the claims. Its sole purpose is to present some of the concepts of the disclosure in a simplified form as a preface to a more detailed description presented later.

[0006] One aspect of the present disclosure provides a method comprising: receiving one or more frames sampled from a media item of an event that is simultaneously streamed to one or more users of a content delivery platform; analyzing the one or more frames to identify one or more candidate event clocks within the one or more frames; for each of the one or more frames, determining whether a candidate event clock among the one or more candidate event clocks satisfies one or more conditions; and in response to determining that the candidate event clock among the one or more candidate event clocks satisfies the one or more conditions, identifying the candidate event clock as an actual event clock for timing the event, and generating mapping data that maps the actual event clock to a timestamp associated with a corresponding frame among the one or more frames of the media item.

[0007] Another aspect of the present invention provides a system, the system comprising: a memory; and a processing device coupled to the memory, the processing device being used to perform a method according to any aspect or embodiment described herein. Other aspects of the present invention provide a computer program product (such as a tangible computer-readable medium or a software product that can be downloaded without having to be stored in a non-transitory manner) including instructions, the instructions being responsive to execution of a processing device, causing the processing device to perform operations including the method according to any aspect or embodiment described herein. Other aspects of the present disclosure provide a computer program, the computer program comprising instructions that, when executed by a processing device, cause the processing device to perform operations according to any aspect or embodiment described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The various aspects and implementations of the present disclosure will be more fully understood based on the detailed description given below and the accompanying drawings of various aspects and implementations of the present disclosure. However, these aspects or implementations should not be regarded as limiting the present disclosure to specific aspects or implementations, but are for illustration and understanding.

[0009] Figure 1 An example system architecture according to an embodiment of the present disclosure is illustrated.

[0010] Figure 2A A flowchart of a method for identifying a candidate event clock as an actual event clock and mapping the actual event clock to a timestamp associated with a corresponding frame of a media item according to an embodiment of the present disclosure is illustrated.

[0011] Figure 2B A flowchart of a method for determining whether a candidate event clock satisfies one or more conditions according to an embodiment of the present disclosure is illustrated.

[0012] Figure 3A is an example pipeline of operations for mapping an actual event clock to timestamps associated with frames of a live streaming media item in accordance with an embodiment of the present disclosure.

[0013] Figure 3B is an example pipeline of operations for mapping actual event periods to timestamps associated with frames of a live streaming media item in accordance with an embodiment of the present disclosure.

[0014] Figure 4 is an example block diagram illustrating a scoring operation for generating scores for candidate event clocks according to an embodiment of the present disclosure.

[0015] Figure 5 is an example illustrating presentation of a live streaming media item with annotated content at a client device in accordance with an embodiment of the present disclosure.

[0016] Figure 6 is a block diagram illustrating an exemplary computer system according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0017] The content delivery platform can provide content such as video items, audio items or game items to users via user devices. Users can log in to a user account associated with the content delivery platform to access the content delivery platform and upload and / or consume content. The content delivery platform can stream media items such as live streaming video items to one or more client devices for user consumption. Live streaming media items can refer to live broadcasts or transmissions of live events, wherein media items are at least partially transmitted simultaneously when the event occurs, and wherein the media items are not all available when the media items begin to be transmitted. Media items typically include multiple frames, wherein each frame includes video data (and optionally, audio data) that can be rendered at an instantaneous time. Each frame is associated with a timestamp. Each timestamp is indexed for the corresponding frames of the media items. For example, each timestamp can indicate a specific instantaneous time when its associated frame is rendered. Timestamps can be represented as a time (e.g., milliseconds) relative to the first frame of the media item or relative to any other suitable reference point. Timestamps can be stored in metadata associated with their corresponding frames, or stored in any other suitable manner. The content delivery platform may receive live streaming media items from source entities such as content owners or content distributors, and buffer the media items for a short amount of time (e.g., 30 seconds) before transmitting the media items to one or more client devices (e.g., user devices of viewers). During the buffering period, the content delivery platform may add annotation content to be presented with the media items at one (or more) client devices. Annotation content may refer to additional content that is presented or provided simultaneously with media items such as live streaming media items or pre-recorded media items. Annotation content may include, but is not limited to, video, text, images, analysis, video overlays, and the like. In some systems, in order to determine the relevant annotation content of the media items, analysis such as video analysis may be performed on the media items to identify specific features or characteristics of the media items. Examples of such features or characteristics may include people or things presented within the media items, actions being performed by people, and the like. The annotation content of the media items may be generated using the specific features or characteristics. The analysis operation may consume a large amount of computing resources (e.g., processing and memory resources) and may be performed for multiple media items of the same event.

[0018] For example, multiple content distributors may stream different live streaming media items of the same event. Even if the live streaming media items are of the same event, the timestamps associated with the same scene may be different for the live streaming media items. This typically occurs when the timestamps of different media items are represented relative to different reference points. For example, annotation content at a specific timestamp of live streaming media item A may not apply to another live streaming media item B at the same timestamp. In this way, computationally intensive analysis is performed on each live streaming media item of the same event, so that the same annotation content can be added to each live streaming media item at the same scene (but at different corresponding timestamps).

[0019] Various aspects of the present disclosure address the other challenges mentioned above by analyzing one or more media items of the same event to identify reference points such as event clocks and / or event periods within the frames of the media items, and generating mapping data that maps reference points such as event clocks and / or event periods to timestamps associated with corresponding frames of the one or more media items. In some embodiments, reference points can be used so that annotation content generated for one media item of the event can be applied to other media items of the same event without performing analysis operations on all media items of the same event.

[0020] In some embodiments, the reference point is a common feature shared by the media items. For example, the event can be a sporting event using an event clock and / or an event period. The event clock can be used to time the event, and the event period can be used to divide the time of the event. For example, a football match is divided into two periods (e.g., halftime) of 45 minutes each. The event clock of the football match can start from 0 and run until the end of the second half (e.g., 90 minutes). Although different live streaming media items of the football match can be transmitted by different content distributors and have different timestamps for the same scene (e.g., match) of the football match, different live streaming media items all share the same event clock and period, which can be used as a reference point.

[0021] In some embodiments, one or more frames of a media item of an event are received. The media item may be simultaneously live streamed to one or more users of a content delivery platform. One or more frames of the media item are analyzed to identify one or more candidate event clocks (or event periods) within the frame. For the one or more frames, one or more candidate event clocks (or event periods) are compared with one or more conditions (e.g., whether the total score of one or more candidate event clocks within the frame exceeds a threshold, whether the candidate event clock is presented within the frame, etc.) to determine whether the candidate event clock satisfies the one or more conditions. In response to determining that the candidate event clock (or event period) satisfies the one or more conditions, the candidate event clock is identified as the actual event clock of the event (and / or the candidate event period is identified as the actual event period). Mapping data is generated that maps the actual event clock (e.g., the time displayed on the actual event clock) and / or the event period to a timestamp associated with a corresponding frame in one or more frames (e.g., a frame in which the actual event clock appears). In some embodiments, mapping data may be generated in a similar manner for each of multiple media items of the same event.

[0022] In some embodiments, annotation content associated with a media item is identified. The media item is annotated with the annotation content based on the mapping data. The media item with the annotation content is provided for presentation at a client device.

[0023] In some embodiments, the annotation content can correspond to a specific timestamp of the media item that has been analyzed to derive the annotation content. For example, the annotation content can be related to an important scene or segment (e.g., a goal) of a sporting event. Mapping data can be used to identify a specific event clock (e.g., a specific time of an event clock) and / or an event period associated with the timestamp of the analyzed media item. Utilizing the specific event clock and / or event period, mapping data can also be used to identify the corresponding timestamps of other media items with the same specific event clock and / or event period. The same or similar annotation content can be applied to other media items using corresponding timestamps, without analyzing other media items to identify the annotation content.

[0024] As noted, the technical problem solved by the embodiments of the present disclosure is the large amount of computing resources used to perform analysis operations on multiple media items of the same event. For example, performing feature recognition on multiple media items consumes a large amount of computing resources. In addition, since media items such as live streaming media items will be provided to end users with little or no delay, analysis operations on multiple media items of the same event are often performed simultaneously, which increases the computing resources required at any given time.

[0025] A technical solution to the technical problem identified above may include: receiving one or more frames sampled from a media stream of an event that is simultaneously streamed live to one or more users of a content delivery platform; analyzing the one or more frames to identify one or more candidate event clocks within the one or more frames; for each of the one or more frames, determining whether a candidate event clock of the one or more candidate event clocks satisfies one or more conditions; and in response to determining that the candidate event clock of the one or more candidate event clocks satisfies the one or more conditions, identifying the candidate event clock as an actual event clock for timing the event, and generating mapping data that maps the actual event clock to a timestamp associated with a corresponding frame of one or more frames of the media item.

[0026] Therefore, technical effects of the present disclosure may include reducing computing resources used when generating annotation content for media items of the same event and providing annotation content for media items of the same event.

[0027] As noted above, a live streaming item may be a live broadcast or transmission of a live event. It may also be noted that, unless otherwise noted, a "live streaming item" or "currently presented live streaming item" refers to a media item that is being live streamed (e.g., the media item is being simultaneously streamed while the event is occurring). Following the end of the live stream of a live streaming item, the complete live streaming item may be obtained and stored, and may be referred to herein as a "previously presented live streaming item" or "archived live streaming item."

[0028] Figure 1 An example system architecture 100 according to one embodiment of the present disclosure is illustrated. The system architecture 100 (also referred to herein as the "system") includes one or more of a content delivery platform 114 (also referred to herein as the "content distribution platform") connected to a network 105, data stores 104 and 109, client devices 103A-103Z (generally referred to herein as the "client device(s) 103"), an annotation content generation server 108, a content distribution server 107, and a scheduling server 106.

[0029] In embodiments, the network 105 may include a public network (e.g., the Internet), a private network (e.g., a local area network (LAN) or a wide area network (WAN)), a wired network (e.g., Ethernet), a wireless network (e.g., an 802.11 network or a Wi-Fi network), a cellular network (e.g., a Long Term Evolution (LTE) network), routers, hubs, switches, server computers, and / or combinations thereof.

[0030] In an embodiment, data stores 104 and 109 are persistent storages capable of storing content items (such as media items) and data structures for tagging, organizing, and indexing content items. Data stores 104 and 109 may be hosted by one or more storage devices such as main memory, magnetic or optical storage-based disks, tapes, or hard drives, network attached storage (NAS), storage area networks (SAN), and the like. In some embodiments, data stores 104 and 109 may be network attached file servers, while in other embodiments, data stores 104 and 109 may be some other type of persistent storage such as an object-oriented database, a relational database, and the like, which may be hosted by content delivery platform 114 or one or more different machines coupled to content delivery platform 114 via network 105.

[0031] In some embodiments, client devices 103A-103Z may each include a computing device such as a personal computer (PC), a laptop computer, a mobile phone, a smart phone, a tablet computer, a netbook computer, a network-connected television, etc. In some embodiments, client devices 103A-103Z may also be referred to as "user devices". In an embodiment, each client device includes a media viewer 102. In one embodiment, the media viewer 102 may be an application that enables a user to play back, view, or upload content such as images, video items, web pages, documents, audio items, etc. For example, the media viewer 102 may be a web browser that can access, retrieve, present, or navigate content served by a network server (e.g., web pages such as hypertext markup language (HTML) pages, digital media items, etc.). The media viewer 102 may render, display, or present content (e.g., web pages, media viewers) to the user. The media viewer 102 may also include a built-in media player (e.g., a FLASH® player or an HTML5 player) built into a web page (e.g., a web page that may provide information about products sold by an online merchant). In another example, the media viewer 102 may be a standalone application (e.g., a mobile application or a native application) that enables a user to play back digital media items (e.g., digital video items, digital images, electronic books, etc.). According to aspects of the present disclosure, the media viewer 102 may be a content delivery platform application for a user to record, edit, and / or upload content shared on the content delivery platform 114. As such, the media viewer 102 may be provided to the client devices 103A-103Z by the content delivery platform 114. For example, the media viewer 102 may be a built-in media player built into a web page provided by the content delivery platform 114. In another example, the media viewer 102 may be an application downloaded from the content delivery platform 114.

[0032] In some embodiments, one or more of the content delivery platform 114, the scheduling server 106, the content distribution server 107, or the annotation content generation server 108 may be one or more computing devices (such as rack servers, router computers, server computers, personal computers, mainframe computers, laptop computers, tablet computers, desktop computers, etc.), data storage (e.g., hard disks, memories, databases), networks, software components, or hardware components that may be used to enable users to access media items or provide media items to users. For example, the content delivery platform 114 may enable users to consume, upload, search, approve (“like”), disapprove (“dislike”), or comment on media items. The content delivery platform 114 may also include a website (e.g., a web page) or application backend software that may be used to enable users to access media items.

[0033] In embodiments of the present disclosure, a "user" may be represented as a single individual. However, other embodiments of the present disclosure encompass an entity that is controlled by a group of users and / or automated sources. For example, a group of individual users united as a community in a social network may be considered a "user." In another example, an automated consumer may be an automated ingestion pipeline such as a topic channel of a content delivery platform 114.

[0034] In some embodiments, the content delivery platform 114 may include multiple channels (e.g., channels A to Z, Figure 1 , only channel A is shown therein). A channel may be data content available from a common source or data content having a common theme, subject or substance. The data content may be digital content selected by a user, made available to a user, uploaded by a user, selected by a content provider, selected by a broadcaster, etc. For example, channel X may include videos Y and Z. A channel may be associated with an owner, who is a user who may perform actions on the channel. Different activities may be associated with a channel based on the actions of the owner, such as the owner making digital content available on the channel, the owner selecting (e.g., liking) digital content associated with another channel, the owner commenting on digital content associated with another channel, etc. Activities associated with a channel may be collected into an activity feed for the channel. Users other than channel owners may subscribe to one or more channels of interest to them. The concept of "subscription" may also be referred to as "liking", "following", "friending", etc.

[0035] Once a user subscribes to a channel, information from the channel activity feed can be presented to the user. If a user subscribes to multiple channels, the activity feeds of the channels to which the user subscribes can be combined into a joint activity feed. Information from the joint activity feed can be presented to the user. Channels can have their own feeds. For example, when navigating to the homepage of a channel on the content delivery platform 114, the feed items generated by the channel can be shown on the channel homepage. A user can have a joint feed, which is a feed that includes at least a subset of the content items from all channels to which the user subscribes. The joint feed can also include content items from channels to which the user is not subscribed. For example, the content delivery platform 114 or other social network can insert the recommended content items into the user's joint feed, or can insert the content items associated with the user's relevant connections into the joint feed.

[0036] Each channel may include one or more media items 111. Examples of media items 111 may include, without limitation, digital videos, digital movies, digital photos, digital music, audio content, melodies, website content, social media updates, electronic books (e-books), electronic magazines, digital newspapers, digital audio books, electronic journals, web blogs, Real Simple Syndication (RSS) feeds, electronic comic books, software applications, etc. In some embodiments, media items 111 are also referred to as content or content items.

[0037] For the sake of brevity and simplicity, and not limitation, video items, audio items, or game items are used throughout this document as examples of media items 111. As used herein, "media," "media items," "online media items," "digital media," "digital media items," "content," and "content items" may include electronic files that may be executed or loaded using software, firmware, or hardware configured to present digital media items to an entity. In one embodiment, content delivery platform 114 may use data store 104 to store media items 111. In another embodiment, content delivery platform 114 may use data stores 104 and 109 to store video items or fingerprints as electronic files in one or more formats.

[0038] In one embodiment, media item 111 is a video item. A video item is a set of continuous video frames (e.g., image frames) representing a scene in motion. For example, a series of sequential video frames can be captured continuously or subsequently reconstructed to generate an animation. Video items can be presented in various formats, including but not limited to analog, digital, two-dimensional, and three-dimensional video. In addition, video items can include movies, video clips, or any set of animated images to be displayed in sequence. In addition, video items can be stored as video files including video components and audio components. Video components can refer to video data in a video encoding format or an image encoding format (e.g., H.264 (MPEG-4 AVC), H.264 MPEG-4 Part 2, Graphics Interchange Format (GIF), WebP, etc.). Audio components can refer to audio data in an audio encoding format (e.g., Advanced Audio Coding (AAC), MP3, etc.). It can be noted that GIF can be saved as an image file (e.g., a .gif file), or saved as a series of images in an animated GIF (e.g., GIF89a format). It may be noted that H.264 may be a video coding format, ie, a video compression standard based on block-oriented motion compensation, for example, for recording, compressing or distributing video contents.

[0039] In some embodiments, media item 111 may be streamed to one or more of client devices 103A-103Z, such as in a live stream. It may be noted that "being streamed" or "streaming" refers to the transmission or broadcast of content, such as a media item, wherein some portions of a received media item may be played back by a receiving device immediately upon receipt, or while other portions of the media content are delivered without the receiving device receiving the entire media item (within technical limitations). "Stream" may refer to content, such as a media item, being streamed or being streamed. A live streaming media item may refer to a live broadcast or transmission of a live event, wherein the media item is transmitted to a receiving device at least partially simultaneously as the event occurs, and wherein the media item is not available in its entirety at least when the transmission of the media item begins.

[0040] In an embodiment, content delivery platform 114 can allow users to create, share, view or use a playlist (e.g., playlist AZ containing media items 111) containing media items. A playlist refers to a collection of media items that are configured to be played in a particular order in succession without any user interaction. In an embodiment, content delivery platform 114 can maintain a playlist on behalf of a user. In an embodiment, the playlist feature of content delivery platform 114 allows users to group their favorite media items together in a single location for playback. In an embodiment, content delivery platform 114 can send the media items on the playlist to client device 103 for playback or display. For example, media viewer 102 can be used to play the media items on the playlist in the order in which the media items are listed on the playlist. In another example, a user can switch between the media items on the playlist. In yet another example, a user can wait for the next media item on the playlist, or can select a specific media item in the playlist for playback.

[0041] In an embodiment, a user may access the content delivery platform 114 through a user account. A user may access (e.g., log in to) a user account by providing user account information (e.g., a username and password) via an application on a client device 103A (e.g., a media viewer 102). In some embodiments, a user account may be associated with a single user. In other embodiments, a user account may be a shared account (e.g., a family account shared by multiple users) (also referred to herein as a "shared user account"). A shared account may have multiple user profiles, each associated with a different user. Multiple users may log in to a shared account using the same account information or different account information. In some embodiments, multiple users of a shared account may be distinguished based on different user profiles of the shared account.

[0042] In some embodiments, the content delivery platform 114 may broadcast one or more live streaming media items to the client devices 103A-103Z. In some embodiments, when the live streaming media items are streamed from the content distribution server 107, the content delivery platform 114 may receive portions of the live streaming media items, such as the live streaming media item 120. In some embodiments, the portions of the live streaming media items may correspond to a sequence of frames of the live streaming media item 120. For example, the first portion of the live streaming media item streamed to the content delivery platform 114 may be the first 30 seconds (0-30 seconds) of the live streaming media item, the second portion of the live streaming media item may be the second 30 seconds (31-60 seconds) of the live streaming media item, and so on. In some embodiments, the content delivery platform 114 may buffer portions of the live streaming media items before streaming them to the client devices 103A-103Z. For example, the content delivery platform 114 may buffer the live streaming item (eg, portions thereof) for 30 seconds after receiving the live streaming item 120 from the content distribution server 107 and before streaming the live streaming item 120 to the client devices 103A-103Z.

[0043] In some embodiments, the content delivery platform 114 can add annotation content to the presentation of the live streaming media item at the client devices 103A-103Z. In some embodiments, the content delivery platform 114 can determine one or more reference points such as an event clock or an event period associated with the live streaming media item, which can be used as a reference for adding annotation content to the presentation of the live streaming media item.

[0044] In some embodiments, the content delivery platform 114 may sample frames (e.g., video frames) of portions of the live streaming media item 120. The content delivery platform 114 may analyze one or more sampled frames to determine one or more candidate event clocks within the sampled frames of the live streaming media item 120. For each frame in the sampled frames, the content delivery platform 114 may determine whether any of the one or more candidate event clocks satisfies one or more conditions. In response to determining that the candidate event clock satisfies one or more conditions, the content delivery platform 114 may identify that the candidate event clock is an actual event clock for timing events. The content delivery platform 114 may map the actual event clock to a timestamp associated with a corresponding frame in which the actual event clock is presented. In some embodiments, the content delivery platform 114 may determine the actual event period within some or all sampled frames of the live streaming media item 120 in a similar manner as described above. The actual event clock and the actual event period may be mapped to a timestamp associated with a corresponding frame in which the actual event clock and / or the actual event period are presented. Relative to Figure 3A Further description of the identification of the actual event clock and relative to Figure 3B The identification of the actual event period is further described.

[0045] In some embodiments, the system architecture 100 includes a scheduling server 106. The scheduling server 106 can provide information such as scheduling information about the transmission or broadcast of media items such as live streaming media items. The scheduling information can include information such as the event type of the live streaming media item, the start time of the transmission or broadcast of the media item, the end time of the transmission or broadcast of the media item, the title of the media item, and / or other metadata associated with the media item. In some embodiments, the scheduling information can indicate a specific content distributor that will transmit or broadcast the live streaming media item. For example, the scheduling information can indicate that the content distributor of the content distribution server 107 will transmit the live streaming media item 120 at 4:00 p.m. Pacific Standard Time on July 18. The scheduling information can also indicate that the live streaming transmission is a basketball sports event between the Santa Clara Broncos and the Colorado Tigers. In some embodiments, the scheduling server 106 is a third-party service provider. In some embodiments, the scheduling server 106 can be a part of the content delivery platform 114 or the content distribution server 107.

[0046] In some embodiments, the content delivery platform 114 (e.g., mapping module 110) can query the scheduling server 106 to retrieve scheduling information corresponding to the broadcast or transmission of media items such as live streaming media items. In some embodiments, the content delivery platform 114 can communicate with the scheduling server 106 via the network 105 to determine any live streaming media items to be broadcast in the future and the event type of the live streaming media items. In some embodiments, for example, the content delivery platform 114 can select live streaming media items of a particular event type of interest, such as a sporting event or any event using an event clock.

[0047] In some embodiments, the system architecture 100 includes one or more content distribution servers such as content distribution server 107. Content distribution server 107 can be part of a content distributor of broadcast or transmission media items such as live streaming media items. In some embodiments, content distribution server 107 can transmit one or more live streaming media items to content distribution platform 114. Content delivery platform 114 can annotate live streaming media items with annotation content and transmit the annotated live streaming media items to one or more client devices 103A-103Z. In some embodiments, content distribution server 107 is part of content delivery platform 114. In other embodiments, content distribution server 107 is a third-party platform that distributes content to content delivery platform 114.

[0048] In some embodiments, the system architecture 100 includes an annotation content generation server 108. In some embodiments, the annotation content generation server 108 may provide an annotation service that identifies or generates annotation content that is provided to the content delivery platform 114 for inclusion in the presentation of a live streaming media item.

[0049] In some embodiments, the annotation content generation server 108 may analyze the live streaming media item (e.g., some portions of the live streaming media item) to identify the content of interest within or associated with the live streaming media item. In some embodiments, the annotation content generation server 108 may use the machine learning model 112 to identify the content of interest within or associated with the live streaming media item. For example, the content delivery platform 114 may send some portions of the live streaming media item to the annotation content generation server 108. The annotation content generation server 108 may use a portion of the live streaming media item as an input (e.g., feature) to the machine learning model 112 (before any preprocessing or after preprocessing some portions of the live streaming media item), and receive the output of the machine learning model 112, which gives the probability that a specific timestamp range of the live streaming media item contains certain content of interest. In some embodiments, the annotation server may generate specific annotation content for the live streaming media item based on the output of the machine learning model 112. The machine learning model 112 may be trained using training data (e.g., exemplary media items such as live streaming media items may be used as training inputs, and interesting content within media items such as short clips within media items may be used as target outputs).

[0050] For example, the annotation content generation server 108 may identify relevant segments (e.g., important plays or key plays) of a basketball game relative to the live streaming item within a timestamp range of 1 minute 30 seconds to 2 minutes. The annotation content generation server 108 may also generate annotation content including a highlight clip (e.g., a short video) showing a video of the relevant segment along with some information about the relevant segment, such as the players involved, the timestamp range when the important play occurred relative to the live streaming item, the game score when the important play occurred, etc. The generated annotations may be provided to the content delivery platform 114 for inclusion in the presentation of the live streaming item.

[0051] In some embodiments, the annotation content generation server 108 may provide the annotation information to the content delivery platform 114 , where the content delivery platform 114 uses the annotation information to generate the annotation content.

[0052] For example, the annotation content generation server 108 may send annotation information such as a timestamp range when important plays occurred relative to the live streaming item to the content delivery platform 114. The content delivery platform 114 may generate a highlight clip of the important plays based on the timestamp range relative to the live streaming item.

[0053] In some embodiments, the annotation content generation server 108 may be part of the content delivery platform 114. In some embodiments, the annotation content generation server 108 may be provided by a third-party service that communicates with the content delivery platform 114 via an application programming interface (API). Figure 2A and Figure 3A , further describing the annotation content generated by the annotation content generation server 108.

[0054] In some embodiments, the system architecture 100 includes a data store 109. In some embodiments, the data store 109 can store relevant content related to the live streaming media item. For example, the relevant content can include event information that can be used to generate annotation content included in the presentation of the live streaming media item. In some embodiments, the annotation content generation server 108 or the content delivery platform 114 can use the data store 109 to generate annotation content for the live streaming media item. For example, the data store 109 can include event information 113 related to a specific event that is being streamed as a live streaming media item. For example, the event information can include live statistics about the event, statistics or other information about the event participants, players of the event, scoring information about the event, team information about the (multiple) teams participating in the event, historical information about the event, participants or teams, or any other type of event related information. The data store 109 can be queried to retrieve event information. In some embodiments, the information of the data store 109 can be obtained based on the actual event clock (and / or the actual event period) of the event.

[0055] In some embodiments, one or more of the operations described herein may be performed by a mapping module 110 as part of a content delivery platform 114. In some embodiments, the mapping module 110 may be Figure 1 For example, the mapping module 110 can be hosted by an external service such as a third-party service.

[0056] Figure 2AA flowchart of a method 200 for identifying a candidate event clock as an actual event clock and mapping the actual event clock to a timestamp associated with a corresponding frame of a media item according to an embodiment of the present disclosure is illustrated. The method 200 is performed by processing logic that may include hardware (circuitry, dedicated logic, etc.), software (e.g., instructions running on a processing device), or a combination thereof. In some embodiments, some or all of the operations of the method 200 may be performed by Figure 1 In some embodiments, one or more operations of method 200 may be performed by mapping module 110 of content delivery platform 114, such as with respect to Figure 1 It can be noted that the Figure 1 The components described are used to illustrate Figure 2A all aspects.

[0057] At block 202, processing logic implementing method 200 receives one or more frames sampled from a media item, such as a live streaming item, of an event. In some embodiments, the event is simultaneously live streamed to one or more users of content delivery platform 114. For example, the one or more frames may be from a portion of a media item received at content delivery platform 114, where other portions of the media item have not yet been recorded (because the event has not yet been completed) or transmitted to content delivery platform 114.

[0058] At block 204, processing logic analyzes one or more frames to identify one or more candidate event clocks within the one or more frames. The term "candidate event clock" refers to a portion of a frame that may or may not be an actual event clock. For example, a candidate event clock may include text (e.g., a number) visibly displayed in a frame. The candidate event clock may be analyzed to determine whether it is an actual event clock for timing an event. In some embodiments, analyzing one or more frames to identify one or more candidate event clocks within one or more frames includes performing an optical character recognition operation on the one or more frames to identify text indicating the one or more candidate event clocks and position information describing the position of the one or more candidate event clocks relative to the one or more frames. Some aspects of analyzing one or more frames to identify one or more candidate event clocks within one or more frames are further described with respect to FIG. 3.

[0059] At block 206, processing logic determines whether a candidate event clock in the one or more candidate event clocks satisfies one or more conditions. In an embodiment, determining whether a candidate event clock in the one or more candidate event clocks satisfies one or more conditions may be performed for each frame in the one or more frames. Figures 2B to 3ASome aspects of determining whether a candidate event clock satisfies one or more conditions are further described. For each frame in which processing logic determines a candidate event clock that satisfies one or more conditions, processing logic may proceed to block 208. If processing logic determines that the candidate event clock of any of the one or more frames does not satisfy the one or more conditions, processing logic may return to block 202 to perform operations of method 200 on additional sample frames of other portions of the media item (e.g., most recently received).

[0060] At block 208, processing logic identifies the candidate event clock as the actual event clock for timing the event. In some embodiments, the candidate event clock is identified as the actual event clock in each of one or more frames, for each of the one or more frames, the candidate event clock is determined to satisfy one or more conditions.

[0061] At block 210, processing logic generates mapping data that maps an actual event clock to a timestamp associated with a corresponding frame in one or more frames of the media item. In some embodiments, in response to determining that a candidate event clock in one or more candidate event clocks satisfies one or more conditions, generating mapping data that maps the actual event clock to a timestamp associated with a corresponding frame in one or more frames of the media item. In some embodiments, generating mapping data that maps the actual event clock to a timestamp associated with a corresponding frame in one or more frames of the media item is performed for each frame in the one or more frames in which the actual event clock is identified. FIG. 3A to FIG. 3B Some aspects of generating mapping data are further described.

[0062] In some embodiments, processing logic can identify the annotation content related to the media item. In some embodiments, processing logic can annotate the media item with the annotation content based on mapping data. For example, when a given annotation content is rendered at a specific time defined with reference to an actual event clock, processing logic can use mapping data to identify the timestamp corresponding to the specific time. Then, processing logic can annotate the media item by associating the annotation content with the frame with the identified timestamp. For example, processing logic can annotate the media item by adding the annotation content to the metadata of the frame. In some embodiments, processing logic can provide the media item with annotation content for presentation at the client device.

[0063] In some embodiments, processing logic may analyze one or more frames to identify one or more candidate event periods within the one or more frames. The term "candidate event period" refers to a portion of a frame that may or may not represent an actual event period. For example, a candidate event clock may include text (e.g., one or more letters and / or numbers) that is visibly displayed in a frame. The candidate event period may be analyzed to determine whether it is an actual event period for dividing the time of an event. In some embodiments, analyzing one or more frames to identify one or more candidate event periods within one or more frames includes performing an optical character recognition operation on the one or more frames to identify text indicating one or more candidate event periods and position information describing the position of the one or more candidate event periods relative to the one or more frames.

[0064] In some embodiments, the processing logic may determine whether a candidate event period in one or more candidate event periods satisfies one or more conditions. In some embodiments, the determination of whether a candidate event period in one or more candidate event periods satisfies one or more conditions may be performed for each of one or more frames or some of one or more frames.

[0065] In some embodiments, in response to determining that a candidate event period in one or more candidate event periods satisfies one or more conditions, processing logic identifies the candidate event period as an actual event period for dividing the event time. In some embodiments, processing logic generates mapping data that maps the actual event period to a timestamp associated with a corresponding frame in one or more frames of the media item. In some embodiments, in response to determining that a candidate event period in one or more candidate event periods satisfies one or more conditions, performing generates mapping data that maps the actual event period to a timestamp associated with a corresponding frame in one or more frames of the media item. Figure 3B Some aspects of identifying a candidate event period as an actual event period are further described.

[0066] In some embodiments, processing logic may perform annotation content analysis of a media item while the media item is being streamed to one or more users simultaneously. In some embodiments, the annotation content analysis is performed to identify annotation content that is relevant to the media item at a particular time stamp (s) associated with the media item. In some embodiments, processing logic provides the annotation content to a client device for presentation with an instance of the media item. Figure 3B Some aspects of performing annotated content analysis of media items are further described.

[0067] In some embodiments, the media item is a first media item and the client device is a first client device. In some embodiments, processing logic generates additional mapping data that maps the actual event clock to a time stamp associated with a corresponding frame in one or more frames of the second media item. The mapping data generated for the second media item can be performed in a similar manner as described herein with respect to the mapping data for generating the media item (e.g., the first media item). In some embodiments, processing logic annotates the second media item with annotation content based on the actual event clock and additional mapping data. In some embodiments, processing logic provides the second media item with annotation content for presentation at the second client device.

[0068] In some embodiments, the annotation content analysis of the second media item is not performed. For example, the processing logic may determine that the annotation content analysis of the second media item is not performed, and thus omits performing such annotation content analysis. In some embodiments, the timestamp associated with the first media item and corresponding to the specific time of the actual event clock of the event is different from the timestamp associated with the second media item and corresponding to the same specific time of the actual event clock of the event.

[0069] Figure 2B A flowchart of a method 212 for determining whether a candidate event clock satisfies one or more conditions according to an embodiment of the present disclosure is illustrated. The method 212 is performed by processing logic that may include hardware (circuitry, dedicated logic, etc.), software (e.g., instructions running on a processing device), or a combination thereof. In some embodiments, some or all of the operations of the method 212 may be performed by Figure 1 In some embodiments, one or more operations of method 212 may be performed by mapping module 110 of content delivery platform 114, such as with respect to Figure 1 It can be noted that the Figure 1 The components described are used to illustrate Figure 2B It may be noted that one or more of the conditions described with respect to method 212 may be implemented independently or in conjunction with one or more other conditions to determine whether a candidate event clock satisfies the determined condition(s). It may also be noted that the operations of method 212 may be performed on one or more frames of a media item. For purposes of illustration and not limitation, method 212 will be described as being performed on a single frame.

[0070] At box 214, the processing logic implementing method 212 generates a score for each candidate event clock in one or more candidate event clocks for frames of the media item. In some embodiments, the score may indicate the likelihood that a corresponding candidate event clock in the one or more candidate event clocks is an actual event clock for timing an event. In some embodiments, each frame may have two scores: a frame score and a total score. The frame score may refer to the score of a candidate event clock determined for a particular frame. The total score may refer to the total score or cumulative score of the previous frame scores of the candidate event clock and the current frame score of the candidate event clock. Figure 4 Frame scores and total scores are further described. Unless otherwise described or implied, the scores used in this article refer to the total score.

[0071] At block 216, processing logic determines whether the first condition is satisfied by determining whether any score of one or more candidate event clocks (of a frame) satisfies a threshold score. For example, satisfying the threshold score may include determining that the candidate event clock score (e.g., the total score) is greater than or equal to the threshold score. In some embodiments, if the candidate event clock score does satisfy the threshold score, processing logic may proceed to block 218 to determine whether the candidate event clock satisfies a second condition, such as determining which of the candidate event clocks that satisfy the threshold score has the highest score. In some embodiments, if the candidate event clock score does not satisfy the threshold score, processing logic may repeat the operation performed at block 214 for the next sequential frame of the one or more frames or for frames of an additional portion of the media item.

[0072] At block 218, processing logic determines whether the second condition is satisfied by selecting the candidate event clock with the highest score. It may be noted that multiple candidate event clocks in a frame may have scores that satisfy a threshold score in some cases. In some embodiments, selecting the candidate event clock with the highest score is in response to determining that more than one of the scores of the one or more candidate event clocks satisfies the threshold score.

[0073] At block 220, processing logic determines whether the third condition is satisfied by determining whether the selected candidate event clock is present within the frame. In some embodiments, if the candidate event clock is present within the frame, processing logic may determine that the candidate event clock is an actual event clock. It may be noted that in some embodiments, a frame without a candidate event clock may not have a frame score for the frame, but may still have an overall score for the candidate event clock for the frame. In some embodiments, if the candidate event clock is not present within the frame, processing logic returns to block 214 to process the next frame. Figure 4 Some aspects of determining whether a candidate event clock satisfies one or more conditions are further described.

[0074] It may also be noted that determining whether the candidate event period satisfies one or more conditions may be performed in a similar manner as described with respect to method 212 .

[0075] Figure 3A 300 is an example pipeline of operations for mapping an actual event clock to a timestamp associated with a frame of a media item and providing annotation content to a live streaming media item for presentation at a client device in accordance with an embodiment of the present disclosure. One or more operations of pipeline 300 may be performed by mapping module 110. Although pipeline 300 is illustrated in a particular order, it is understood that some of the operations may be performed serially or in parallel. It may also be noted that the same, fewer, more, or different operations may be performed in some embodiments.

[0076] In some embodiments, at query operation 302, the content delivery platform 114 (eg, mapping module 110) may query Figure 1 The content delivery platform 114 may also query the scheduling server 106 to identify any live streaming media items that will be transmitted or broadcast in the future (or are currently being broadcast or transmitted). The content delivery platform 114 may also determine the event type of the live streaming media item by querying the scheduling server 106. The content delivery platform 114 may determine whether the event type of the live streaming media item is an event type of interest by comparing the event type with a record of the event type of interest. Event types may include, for example, different types of sporting events or event types with event clocks. In some embodiments, if the live streaming media item scheduled for transmission is an event type of interest, once the transmission of the live streaming media item has begun, the content delivery platform 114 may schedule a job to generate mapping data for the live streaming media item.

[0077] In some embodiments, at broadcast operation 304, as Figure 1 As described in detail in , the content distribution server 107 can transmit (e.g., stream) the live streaming media item to the content delivery platform 114. As described above, some portions of the live streaming media item can be streamed to the content delivery platform 114. The content delivery platform 114 can buffer these portions of the live streaming media item and perform the operations described herein (e.g., mapping and annotation of the live streaming media item, etc.).

[0078] It can be noted that an event can be streamed as multiple media items such as multiple live streaming media items. Multiple live streaming media items can be multiple instances different from the same live streaming media item. For example, multiple live media items of an event can be recorded with different cameras, transmitted by different content distributors, or created for different regional markets. Multiple live streaming media items of the same event can have different timestamps corresponding to the same actual event clock or actual event period of the event. For example, Spain is hosting a sporting event. The live stream broadcast of the sporting event in the United States can be different from the live stream broadcast of the same sporting event in Spain (for example, different regional versions of the event are provided by the same content distributor). In another example, content distributor A can broadcast a live stream of sporting event X, and content distributor B can broadcast another different live stream of sporting event X. Each of the different live streaming media items of the same event can be sent to content distribution platform 114.

[0079] In some embodiments, the content delivery platform 114 may perform a sampling operation 306. At the sampling operation 306, the content delivery platform 114 may sample portions of the live streaming media item 120. In some embodiments, the content delivery platform 114 may sample a subset of frames of portions of the received live streaming media item. In other embodiments, the content delivery platform 114 may sample all frames of portions of the received live streaming media item.

[0080] In some embodiments, the frames (of some parts) of the live streaming media item may be sampled at a specific frequency. For example, some parts of the live streaming media item 120 may be sampled at a frequency of one frame per second. In some embodiments, the frequency may change according to the event type. In some embodiments, the sampling frequency may correspond to the minimum increment of the event clock of a specific event type. For example, for an event type having an event clock that increments in units of 1 second, the frames of some parts of the live streaming media item may be sampled at one frame per second. For an event type having an event clock in units of subseconds, the frames of some parts of the live streaming media item may be sampled at a subsecond frequency, or all frames of the live streaming media item may be used. For example, an American football event type may have an event clock that increments in units of seconds, and the frames of the live streaming of the American football event type may be sampled at 1 frame per second. In another example, a 100-yard dash event type may have an event clock that increments in units of milliseconds. The frames of the live streaming of the 100-yard dash type may be sampled at a subsecond frequency.

[0081] In some embodiments, the content delivery platform 114 may perform a content analysis operation 308 (e.g., text analysis) on the sampled video frames of the live streaming media item. In some embodiments, at the content analysis operation 308, the content delivery platform 114 may analyze one or more sampled frames to retrieve frame content information describing the content of the frames. In some embodiments, the frame content information may include text within the analyzed video frame or text position information indicating the position of the text relative to the analyzed video frame.

[0082] In some embodiments, optical character recognition technology can be used to retrieve text and location information of text. It can be noted that optical character recognition technology is used for the purpose of illustration rather than limitation. In other implementations, other technologies can be used to retrieve frame content information. In some embodiments, optical character recognition operations can convert images of text into machine-encoded text. For example, an event clock image in a frame of a live streaming media item can be converted into text. In some embodiments, all text images in a frame can be converted into text.

[0083] In some embodiments, identifying the location information of the text may include identifying coordinate identifiers (e.g., four coordinate identifiers) that locate the text in a coordinate structure relative to a given frame. For example, a coordinate structure such as an X-axis and a Y-axis may be applied to each frame. The location of the text may be determined within the coordinate structure. The location of the text may be described using the coordinate identifiers. In some embodiments, for each frame, one or more of the text, the location information, and a frame identifier identifying the frame corresponding to the text and the location information may be sent to the candidate event clock operation 312. In some embodiments, for frames without text, an indication that the text was not found in the frame and the frame identifier may be sent to the candidate event clock operation 312.

[0084] In some embodiments, the content delivery platform 114 may perform a candidate event clock operation 312. At the candidate event clock operation 312, the content delivery platform 114 uses the text and position information provided by the content analysis operation 308 to identify one or more candidate event clocks within each frame or some frames of the one or more sample frames. In some embodiments, pattern recognition techniques may be used to identify candidate event clocks from the text and the position information of the text. Pattern recognition may refer to checking for the presence of a certain predetermined pattern in a given sequence of values ​​such as text and / or position information of the text.

[0085] For example, event clocks often correspond to a specific format. For example, four numbers separated by a colon ("12:45") are a common format for event clocks. In some embodiments, text and text position information retrieved from optical character recognition technology can be used as input and processed by a pattern recognition sub-operation to determine whether the text will be considered a candidate event clock (or candidate event period). The pattern recognition sub-operation can determine whether the text and text position information correspond (e.g., match) to a specific pattern or format (e.g., an event clock pattern) and filter out text and text position information that does not correspond to the pattern (e.g., noise). If the text and / or text position information matches or corresponds to the pattern, the text can be determined to be a candidate event clock. If the text does not match or correspond to the pattern, the text is not identified as a candidate event clock.

[0086] In some embodiments, a specific pattern recognition operation can be selected from multiple pattern recognition operations based on the event type. Different event types can have different event clock modes and period modes. The selected pattern recognition operation can detect a specific pattern of a specific event type.

[0087] In some embodiments, the content delivery platform 114 may perform a scoring operation 314 on the candidate event clocks identified at the candidate event clock operation 312. The determined candidate event clocks and corresponding text position information may be sent from the candidate event clock operation 312 to the scoring operation 314. The content delivery platform 114 performing the scoring operation 314 may generate a score for each of the one or more candidate event clocks in each of the one or more sample frames.

[0088] In some embodiments, a particular scoring operation may be selected from a plurality of scoring operations based on the event type. Different event types may have different characteristics. For example, the event clock for a soccer event is expected to increase relative to the sequential frames, while the event clock for an American football event is expected to decrease. The selected scoring operation may score the candidate event clocks based on the specific characteristics of the specific event type.

[0089] In some embodiments, the scoring may be based on one or more criteria (or rules). As described above, the criteria may be based on event type. In an example, the content delivery platform 114 may determine whether the time of one or more of the candidate event clocks has changed from one frame of the sampled video frame to the next sequential frame. In response to determining that the time of the candidate event clock has changed from one frame to the next sequential frame, the content delivery platform 114 may determine the magnitude and direction of the change. If the magnitude and direction of the change are consistent with the event clock of a particular event type, the content delivery platform 114 may positively score the candidate event clock of the corresponding frame (e.g., a frame score). If the magnitude of the change or the direction of the change is inconsistent with the event clock of a particular event type, the content delivery platform 114 may negatively score the candidate event clock of the corresponding frame (e.g., a frame score).

[0090] For example, for frame 1, candidate event clock X is associated with text that reads "12:20". For the next sequentially sampled frame (i.e., frame 2), candidate event clock X is associated with text that reads "12:19". Frames are sampled once per second, and certain event types have event clocks that count down in 1 second increments. The content delivery platform 114 may add 5 points to the candidate event clock frame score at frame 2 and store this data as the frame score associated with candidate event clock X at frame 2.

[0091] In some embodiments, the scores of candidate event clocks may be summed with respect to sequential frames. For example, the total score (e.g., cumulative score) of candidate event clock X at frame 2 may be the sum of the score of candidate event clock X at frame 1 (if any) and the score of candidate event clock X at frame 2. The score of candidate event clock X at frame 3 may be the sum of the scores of candidate event clock X at frames 1, 2, and 3.

[0092] The scores (frame and / or total score) for each candidate event clock for each sample frame may be sent to the conditional analysis operation 316. Figure 4 Some aspects of the scoring of candidate event clocks are further described.

[0093] In some embodiments, at conditional analysis operation 316, the content delivery platform 114 may determine whether the candidate event clock of each frame in the one or more sampled frames satisfies one or more conditions. In some embodiments, at conditional analysis operation 316, the content delivery platform 114 determines whether any score of the one or more candidate event clocks satisfies a threshold score. For example, satisfying the threshold score may include determining that the candidate event clock score (e.g., the total score) of the particular frame is greater than or equal to the threshold score. In an embodiment, the score of the candidate event clock of the particular frame corresponds to the total score of the candidate event clock of the particular frame. In some embodiments, the content delivery platform 114 determines whether any score of the one or more candidate event clocks of the particular frame satisfies the threshold score.

[0094] For example, at frame 3, there are three candidate event clocks: candidate event clock X, candidate event clock Y, and candidate event clock Z have total scores of 15, 10, and -5, respectively. The threshold score is 5. The scores of candidate event clock X (15) and candidate event clock Y (10) both meet the threshold score, while candidate event clock Z does not meet the threshold score.

[0095] In some embodiments, if any of the candidate event clocks has a total score for a particular frame that satisfies a threshold score, another condition (e.g., a second condition) may be used to evaluate those (multiple) candidate event clocks. In some embodiments, for one or more candidate event clocks having a total score that satisfies a threshold score, the content delivery platform 114 may select the candidate event clock with the highest score.

[0096] For example, both candidate event clock X (15) and candidate event clock Y (10) of frame 3 meet the threshold score. Candidate event clock X has the highest total score of 15 compared to the total score of candidate event clock Y (10).

[0097] In some embodiments, the candidate event clock with the highest score can be evaluated to determine whether the candidate event clock satisfies another condition (eg, a third condition). In some embodiments, the third condition determines whether the candidate event clock is present within a corresponding frame of the one or more frames.

[0098] For example, candidate event clock X is the candidate event clock selected for frame 3 because it meets the threshold score and has the highest total score for frame 3. Since the total scores of specific candidate event clocks can be summed, some frames that do not have candidate event clocks in the current frame can still have the total scores of candidate event clocks that exist in previous frames. An example of a frame without a candidate event clock may include a commercial inserted into a live streaming media item of an event. The total scores of candidate event clocks can be retained from previous frames included in the candidate event clocks and populated to subsequent frames that do not include candidate event clocks.

[0099] In an embodiment, if the candidate event clock satisfies one or more conditions, the candidate event clock is identified as the actual event clock. If the candidate event clock does not satisfy one or more conditions, the candidate event clock is not identified as the actual event clock.

[0100] In the above example, if the candidate event clock X of frame 3 is not within frame 3, the candidate event clock X is not identified as the actual event clock of frame 3. If the candidate event clock X of frame 3 is within frame 3 (and satisfies other conditions), the candidate event clock X is identified as the actual event clock.

[0101] In some embodiments, one or more of the above conditions may be used to determine the actual event clock. In other embodiments, different conditions may be used to determine the actual event clock.

[0102] In some embodiments, the content delivery platform 114 may perform a mapping operation 318. At the mapping operation 318, the content delivery platform 114 may generate mapping data that maps the actual event clock to a timestamp associated with a corresponding frame in one or more frames of the live streaming media item. As noted, the actual event clock may be used as a reference or reference point for multiple live streaming media items of the same event. In some embodiments, the mapping data may be stored in a record. In an example, the mapping data may identify one or more actual event clocks (e.g., the time of the actual event clock), a timestamp of a frame of the live streaming media item in which the actual event clock has been identified, and an identifier of the live streaming media item.

[0103] In some embodiments, the content delivery platform 114 may perform an annotation content generation operation 310. Figure 1 As noted, the annotation content generation operation may be performed by a service such as a third-party service (e.g., an annotation service). Figure 1 The annotation content generation server 108 may perform an annotation content generation operation 310 .

[0104] In some embodiments, the annotation content generation operation 310 may receive some portions (e.g., unsampled) of the live streaming media item received by the content delivery platform 114. In some embodiments, the annotation content generation operation 310 may receive sampled frames of the live streaming media item. In some embodiments, the annotation content generation operation 310 may provide annotation information to the content delivery platform 114, wherein the content delivery platform 114 uses the annotation information to generate annotation content. In some embodiments, at the annotation content generation operation 310, annotation content is generated or identified within the frame of the received live streaming media item. In some embodiments, the annotation content may be sent together with a timestamp range in which the annotation content may be found within the live streaming media item. In some embodiments, the annotation content may be sent together with a timestamp range indicating the time of the live streaming media item associated with the annotation content. The timestamp may be used together with the mapping data to identify the actual event clock of the event associated with the annotation content.

[0105] In an example, at the annotation content generation operation 310, the content delivery platform 114 may analyze the live streaming media item to identify relevant segments (e.g., key plays) of the live streaming media item of an American football sporting event, such as scoring segments (e.g., plays that result in a score or touchdown). At the annotation content generation operation 310, a sample frame or sequence of sample frames (e.g., timestamp(s)) containing the relevant segments may be identified. The generated annotation content and / or annotation information may be sent to the live streaming media item annotation operation 322. In some embodiments, the content delivery platform 114 may perform an annotation content discovery operation 320. In some embodiments, at the annotation content discovery operation 320, the content delivery platform 114 may use the actual event clock (or the timestamp received from the annotation content generation operation 310) to identify relevant content such as event information that may be used to generate annotation content or modify the annotation content received from the annotation content generation operation 310. In some embodiments, the content delivery platform 114 may query the data store 109 to identify event information related to the time of the actual event clock. In some embodiments, mapping data may be used to map the time of an event clock associated with the event information to timestamp(s) of the live streaming media item so that annotation content generated from the event information may be presented with the live streaming media item at an appropriate time.

[0106] For example, data store 109 may store live statistics for an event corresponding to a certain time of an event clock for the event. For example, during a basketball sporting event, player A may score 20 points at event time 11:15. Data store 109 may be queried to retrieve event information and use the event information to generate annotation content. For example, content delivery platform 114 may annotate (e.g., generate an overlay) a live streaming media item, thereby displaying the event information along with the presentation of the live streaming media item.

[0107] In some embodiments, the content delivery platform 114 may perform live streaming media item annotation operations 322. At live streaming media item annotation operations 322, the content delivery platform 114 may apply annotation content to one or more live streaming media items.

[0108] In some embodiments, the annotation content from the annotation content generation operation 310 may be sent to the live streaming item annotation operation 322 along with the corresponding timestamp(s) of the live streaming item to which the annotation content relates. The annotation content may be mapped to the actual game clock (e.g., the time(s) of the actual game clock) using mapping data.

[0109] In some embodiments, multiple live streaming media items of the same event can be processed so that mapping data for the multiple live streaming media items is generated. In some embodiments, at the annotation content generation operation 310, a single live streaming media item of the multiple live streaming media items of the same event is processed. The timestamp identified at the annotation content generation operation 310 can be used to identify the actual event clock (e.g., reference point) of the multiple live streaming media items of the same event. For example, the identified actual event clock can be used together with the mapping data to identify the corresponding timestamps of other live streaming media items in the multiple live streaming media items related to the annotation content. In some embodiments, the annotation content can be applied to other live streaming media items at the corresponding timestamps without performing the annotation content generation operation 310 on the other live streaming media items.

[0110] In some embodiments, the content delivery platform 114 may apply the event information received from the annotation content discovery operation 320 (or the annotation content generated from the event information) to one or more live streaming media items of the event. In some embodiments, the event information of the annotation content discovery operation 320 or the annotation content generated from the event information may be used as a supplement or modification to the annotation content received from the annotation content generation operation 310. In some embodiments, the event information of the annotation content discovery operation 320 or the annotation content generated from the event information may be used as annotation content independent of the annotation content generated at the annotation content generation operation 310. In some embodiments, the event information received from the annotation content discovery operation 320 (or the annotation content generated from the event information) includes a corresponding event clock. The mapping data and the corresponding event clock information may be used to identify corresponding timestamps of multiple live streaming media items, which timestamps may be applied to the event information or the annotation content generated from the event information.

[0111] In some embodiments, the content delivery platform 114 can send (multiple) annotated live streaming media items to one or more client devices 103A-103Z for presentation of the live streaming media items together with the annotated content. As noted above, the live streaming media items received by the content delivery platform 114 from the content distribution server 107 can be buffered for a short period of time. Operations as described with respect to pipeline 300 can be performed during the buffering period, and the annotated live streaming media items can be provided to the client devices 103A-103Z with minimal delay.

[0112] Figure 3B is an example pipeline of operations for mapping actual event periods to timestamps associated with frames of live streaming media items in accordance with an embodiment of the present disclosure. One or more operations of pipeline 350 may be performed by mapping module 110. Although pipeline 350 is illustrated in a particular order, it is understood that some of the operations may be performed in series or in parallel. It may also be noted that in some embodiments the same, fewer, more, or different operations may be performed. It may be noted that the operations of pipeline 350 may be similar to Figure 3A In the embodiment, unless otherwise described, Figure 3A The operations of pipeline 300 are applied to Figure 3B and for the sake of clarity, they are not repeated.

[0113] In some embodiments, the content delivery platform 114 may perform a candidate event period operation 326. In some embodiments, at the candidate event period operation 326, the content delivery platform 114 identifies one or more candidate event periods within one or more sample frames. In some embodiments, pattern recognition may be used to identify candidate event periods from text identified within one or more sample frames. For example, an event period may correspond to a specific pattern. For example, the word "Quarter" (or its abbreviation) followed by a colon and then a number (Quarter: 1) may be a pattern of an event period. In some embodiments, the text and the position information of the text retrieved from the content analysis operation 308 may be used as input to the pattern recognition sub-operation of the candidate event period operation 326. A specific pattern recognition sub-operation may be selected from a plurality of pattern recognition sub-operations based on the event type of the live streaming media item. The pattern recognition sub-operation may determine whether the text (and / or the position information of the text) corresponds to a specific pattern (e.g., an event period pattern) and filter out text and text position information (e.g., noise) that do not correspond to a specific pattern. In some embodiments, in response to determining that the text (and / or the text position information) corresponds to a specific pattern, the text may be identified as a candidate event period.

[0114] In some embodiments, the content delivery platform 114 may perform a scoring operation 328. The determined candidate event periods and corresponding location information may be sent from the candidate event period operation 326 to the scoring operation 328. The content delivery platform 114 performing the scoring operation 328 may generate a score for each of the one or more candidate event periods for each of the one or more sample frames.

[0115] In some embodiments, a specific scoring operation can be selected from multiple scoring operations based on the event type. Different event types can have different characteristics. For example, the event period of a football game can be divided into the first half and the second half, while the event period of an American football game can be divided into 4 quarters. The selected scoring operation can score the candidate event period based on the specific characteristics of the specific event type.

[0116] In some embodiments, the scoring may be based on one or more criteria (e.g., rules). As described above, the criteria may be based on event types. In an example, the content delivery platform 114 may determine whether one or more of the candidate event periods have changed from one frame of the sampled video frame to the next sequential frame. In response to determining that the candidate event period has changed from one frame to the next sequential frame, the content delivery platform 114 may determine the magnitude and direction of the change. If the magnitude of the change and the direction of the change are consistent with the event period of a particular event type, the content delivery platform 114 may score the candidate event period positively (e.g., frame scoring). If the magnitude of the change or the direction of the change is inconsistent with the event period of a particular event type, the content delivery platform 114 may score the candidate event period negatively (e.g., frame scoring). For example, for frame 1, candidate event period X is associated with text that reads "Quarter: 2". For the next sequential frame (i.e., frame 2), candidate event period X is associated with text that reads "Quarter: 3". For an event type with 4 sections, the change from "Quarter: 2" to "Quarter: 3" may be determined to correspond to an event period pattern, and frame 2 may be scored positively (eg, frame score).

[0117] In some embodiments, the candidate event period scores may be summed with respect to sequential frames. For example, the total score of candidate event period X at frame 2 may be the sum of the scores of candidate event period A at frames 1 and 2.

[0118] In some embodiments, the content delivery platform 114 may perform a mapping operation 330. At the mapping operation 330, the content delivery platform 114 may generate mapping data that maps the actual event period to a timestamp associated with a corresponding frame in one or more frames of the live streaming media item. As noted, the actual event period may be used as a reference or reference point for multiple live streaming media items of the same event. In some embodiments, the mapping data may be stored in a record. In an example, the mapping data may identify one or more actual event periods, a timestamp of the live streaming media item for which the actual event period has been identified within the corresponding frame, and an identifier of the live streaming media item.

[0119] In some embodiments, the mapping data generated by the mapping operation 330 may be associated with Figure 3A The mapping data may be combined with the mapping data generated in the mapping operation 318 in the above. For example, the actual event clock may be read as "10:45" and the event period may be read as "Quarter: 3". The mapping data may associate the actual event clock, the actual event period, the timestamp of the live streaming media item of the actual event clock and the actual event period that have been identified in the corresponding frame, and the identifier of the live streaming media item.

[0120] Figure 4is an example block diagram 400 illustrating scoring operations for generating scores (eg, frame scores and total scores) for candidate event clocks in accordance with an embodiment of the present disclosure. In some embodiments, the scoring operations of diagram 400 may be similar to those of Figure 3A The scoring operation 314 is described. For clarity, relative to Figure 4 Describe the scoring operation for the candidate event clock. It can be understood that the scoring operation of diagram 400 can be applied to the scoring of the candidate event period.

[0121] Diagram 400 illustrates sample frames 402A-402Z (generally referred to as frames 402). It should be noted that sample frames 402 may include the same, more, or fewer video frames. In some embodiments, sample video frames 402 may be ordered based on timestamps associated with each sample frame 402. Some of the sample frames 402 are shown with candidate event clocks. Each candidate event clock is associated with a corresponding frame score and a corresponding total score.

[0122] For example, candidate event clock A in sample frame 402A has a frame score of 0 and a total score of 0. In sample frame 402B, candidate event clock A has a frame score of 5 and a total score of 5. It may be noted that the following numbers, dimensions, equations are used for the purpose of illustration and not limitation.

[0123] As shown in diagram 400, some of the sample frames 402 include two candidate event clocks: candidate event clock A and candidate event clock B. Both candidate event clocks are illustrated with corresponding frame scores and corresponding total scores. The scoring operation generates a frame score for each candidate event clock based on one or more criteria, such as Figure 3A The one or more criteria selected may be based on a particular event type. In Figure 400, the event type is a basketball event type, and the corresponding one or more criteria for scoring may be for the basketball event type.

[0124] In one example, the criteria include analyzing the change of the candidate event clock relative to the sequentially sampled frames. Frame 402A is the initial frame in which the candidate event clock A and the candidate event clock B are first identified. The frame score of each of the candidate event clocks A and B is initialized to 0, and the cumulative score is also initialized to 0. From frame 402A to frame 402B, the candidate event clock A changes from "12:00" to "11:59", and the candidate event clock B changes from "2:32" to "7:53". As described above, the event type in Figure 400 is a basketball event type, and if the frame is sampled at 1 frame per second, it is expected that the event clock will be reduced by 1 second for the sequentially sampled frames. At frame 401B, the scoring operation assigns a positive frame score to the candidate event clock A because the candidate event clock A decreases by 1 second from frame 402A to frame 402B (e.g., according to the criteria). At frame 402B, the total score of the candidate event clock A is 5. At frame 402B, the scoring operation assigns a negative frame score to candidate event clock B because candidate event clock B increases from frame 402A to frame 402B and thus does not meet the criteria. At frame 402B, candidate event clock B has an overall score of -5.

[0125] In some embodiments, the scoring criteria may include the presence or absence of a candidate event clock score relative to a sequentially sampled frame.An example of a frame where no candidate event clock is present is during a commercial break.

[0126] At frame 402C, the presentation of the commercial pauses, and the presentation of the basketball game is temporarily stopped. As illustrated, candidate event clock A and candidate event clock B do not exist in frame 402C. Since both candidate event clock A and candidate event clock C do not exist in frame 402C, no frame score is assigned to candidate event clock A and candidate event clock B at frame 402C. The total score of candidate event clock A and candidate event clock B is passed along and is unchanged for the duration of the commercial pause. It can be noted that frames 402D to 402Y represent frames of commercials and are omitted for clarity.

[0127] From frame 402B to frame 402Z, candidate event clock A changes from "11:59" to "11:58", while candidate event clock B changes from "7:53" to "1:02". Candidate event clock A and candidate event clock B of frame 402Z are compared with the corresponding candidate event clocks of frame 402B (rather than the immediately preceding sampling frame) because frame 402B is the most recent frame in which candidate event clock A and candidate event clock B exist. At frame 420Z, the scoring operation assigns a positive frame score to candidate event clock A because candidate event clock A decreases by 1 second from frame 402B to frame 402Z (e.g., according to the criteria of the event type). At frame 420Z, the frame score of candidate event clock A is 5, and the total score of candidate event clock A is 10. At frame 420Z, the scoring operation assigns a negative frame score to candidate event clock B because candidate event clock B increases from frame 402B to frame 402Z, thereby not meeting the criteria of the event type. At frame 420Z, candidate event clock B has a frame score of -5, and the total score of candidate event clock B is -10.

[0128] Figure 5 is an example illustrating presentation of a live streaming media item with annotated content at a client device in accordance with an embodiment of the present disclosure.

[0129] The graphical user interface (GUI) 500 of the client device shows the presentation of a live streaming item 504. The live streaming item 504 is of the basketball game type. Below the live streaming item 504 are annotation content 502A, 502B, and 502C (generally referred to as "annotation content 502").

[0130] Annotation content 502 is an example of a relevant segment of a live streaming media item. For example, annotation content 502A corresponds to a relevant segment of basketball player Kevin D receiving an alley-oop dunk in the 4th quarter (e.g., event period) at 2:25 of the event clock. As illustrated, annotation content 502B corresponds to a relevant segment of basketball player Kevin L making a two-point shot. Annotation content 502C corresponds to a relevant segment of basketball player Lebron J blocking a shot.

[0131] The relevant fragment can be composed of Figure 1 The annotation content generation server 108 or Figure 3A Examples of annotation content identified by the annotation content generation operation 310. For example, the annotation content 502A-502C can be selected to display a video of the related segment at the GUI 500.

[0132] Some of the content in the annotation content 502A may be from Figure 1 Data storage 109 or Figure 3AThe annotation information identified by the annotation content discovery operation 320 is an example of annotation information that has been identified at the data store 109. For example, the name of the athlete, the type of game, and the photo of the athlete may be examples of annotation information that has been identified at the data store 109. The annotation information from the data store 109 has been added to the annotation content from the annotation content generation server 108 to generate the annotation content 502.

[0133] Annotation content 502 is provided as an illustration and not limitation. In other embodiments, the annotation content may be different or presented differently. For example, the annotation content may be directly added to the live streaming media item so as to be displayed in a particular frame. In some embodiments, the annotation content may be presented in a media viewer together with the live streaming media item. In some embodiments, the annotation may be overlaid on the live streaming media item in the media view.

[0134] Figure 6 1 is a block diagram illustrating an exemplary computer system 600 according to an embodiment of the present disclosure. The computer system 600 executes one or more instruction sets that cause the machine to perform any one or more methods discussed herein. Instruction sets, instructions, etc. may refer to instructions that cause the computer system 600 to perform one or more operations of the mapping module 110 when the computer system 600 is executed. The machine may operate in the capacity of a server or client device in a client-server network environment, or as a peer machine in a peer (or distributed) network environment. The machine may be a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a mobile phone, a network device, a server, a network router, a switch or a bridge, or any machine capable of executing an instruction set (sequentially or otherwise) that specifies the actions to be taken by the machine. In addition, although only a single machine is illustrated, the term "machine" should also be considered to include any collection of machines that independently or jointly execute instruction sets to perform any one or more methods discussed herein.

[0135] The computer system 600 includes a processing device 602, a main memory 604 (e.g., read-only memory (ROM), flash memory, dynamic random access memory (DRAM) such as synchronous DRAM (SDRAM) or Rambus DRAM (RDRAM)), etc.), a static memory 606 (e.g., flash memory, static random access memory (SRAM), etc.), and a data storage device 616 that communicate with each other via a bus 608.

[0136] The processing device 602 represents one or more general-purpose processing devices such as a microprocessor, a central processing unit, etc. More specifically, the processing device 602 may be a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, or a processing device that implements other instruction sets or a processing device that implements a combination of instruction sets. The processing device 602 may also be one or more special-purpose processing devices such as an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), a network processor, etc. The processing device 602 is configured to execute instructions of the system architecture 100 and the mapping module 110 to perform the operations discussed herein.

[0137] The computer system 600 may also include a network interface device 622 that provides communications with other machines via a network 618, such as a local area network (LAN), an intranet, an extranet, or the Internet. The computer system 600 may also include a display device 610 (e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)), an alphanumeric input device 612 (e.g., a keyboard), a cursor control device 614 (e.g., a mouse), and a signal generating device 620 (e.g., a speaker).

[0138] The data storage device 616 may include a non-transitory computer-readable storage medium 624 having stored thereon a set of instructions for the system architecture 100 or mapping module 110 that implements any one or more of the methodologies or functions described herein. The set of instructions for the system architecture 100 or mapping module 110 may also reside completely or at least partially within the main memory 604 and / or within the processing device 602 during execution thereof by the computer system 600, the main memory 604, and the processing device 602, which also constitute computer-readable storage media. The set of instructions may also be sent or received over the network 618 via the network interface device 622.

[0139] Although the example of computer-readable storage medium 624 is shown as a single medium, the term "computer-readable storage medium" may include a single medium or multiple media (e.g., a centralized or distributed database and / or associated caches and servers) that store instruction sets. The term "computer-readable storage medium" may include any medium that can store, encode, or carry instruction sets for execution by a machine and cause the machine to perform any one or more of the methods of the present disclosure. The term "computer-readable storage medium" may include, but is not limited to, solid-state memories, optical media, and magnetic media.

[0140] In the foregoing description, numerous details are set forth. However, it will be apparent to those of ordinary skill in the art who benefit from the present disclosure that the present disclosure may be practiced without these specific details. In some cases, in order to avoid obscuring the present disclosure, well-known structures and devices are shown in block diagram form rather than in detail.

[0141] Some parts of the specific implementation are presented in terms of algorithms and symbolic representations that operate on data bits in a computer memory. These algorithmic descriptions and representations are the means used by technicians in the field of data processing to most effectively convey the content of their work to other technicians in the field. The algorithms generally contemplated here are a series of operations that are organized to cause the desired results. These operations are operations that require physical manipulation of physical quantities. Typically, these quantities take the form of electrical or magnetic signals that can be stored, transferred, combined, compared, and otherwise manipulated, but not necessarily. Mainly for general reasons, it is sometimes convenient to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, etc.

[0142] It should be remembered, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise, it is clear that throughout this specification, discussions utilizing terms such as "generate," "analyze," "determine," "confirm," "provide," "annotate," "determine," "select," and the like refer to the actions and processes of a computer system or similar electronic computing device that manipulates data represented as physical (electronic) quantities within a computer system's memory or registers and transforms that data into other data similarly represented as physical quantities within a computer system's memory or registers or other such information storage, transmission, or display device.

[0143] The present disclosure also relates to an apparatus for performing the operations herein. The apparatus may be specially constructed for the desired purpose, or it may include a general-purpose computer selectively enabled or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a computer-readable storage medium, such as but not limited to any type of disk, including a floppy disk, an optical disk, a compact disk read-only memory (CD-ROM), a magneto-optical disk, a read-only memory (ROM), a random access memory (RAM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic card or an optical card, or any type of medium suitable for storing electronic instructions.

[0144] The words "example" or "exemplary" are used herein to mean serving as an example, instance or illustration. Any aspect or design described herein as "example" or "exemplary" is not necessarily to be interpreted as being preferred or advantageous over other aspects or designs. Specifically, the use of the words "example" or "exemplary" is intended to present the concept in a specific way. As used in this application, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or". That is, unless otherwise specified or clear from the context, "X includes A or B" is intended to mean any natural inclusive arrangement. That is, if X includes A; X includes B; or X includes both A and B, then "X includes A or B" is satisfied in any of the foregoing situations. In addition, the articles "one" and "an" used in this application and the appended claims can generally be interpreted as meaning "one or more", unless otherwise specified or clearly related to the singular form from the context. In addition, the terms "embodiment" or "one embodiment" or "implementation" or "one implementation" used throughout the text are not intended to mean the same embodiment or implementation, unless described as such. The terms “first,” “second,” “third,” “fourth,” etc. used herein are labels intended to distinguish different elements and may not necessarily have ordinal meanings according to their numerical labels.

[0145] For simplicity of explanation, the method herein is depicted and described as a series of actions or operations. However, the actions according to the present disclosure may occur in various orders and / or simultaneously, and occur together with other actions not presented and described herein. In addition, not all exemplified actions will be required to implement the method according to the disclosed subject matter. In addition, those skilled in the art will understand and appreciate that, alternatively, these methods can be represented as a series of related states via state diagrams or events. In addition, it should be understood that the method disclosed in this specification can be stored on an article of manufacture, so that this method is transmitted and transferred to a computing device. The term article of manufacture used herein is intended to cover a computer program accessible from any computer-readable device or storage medium.

[0146] In additional embodiments, one or more processing devices for performing the operations of the above-described embodiments are disclosed. In addition, in embodiments of the present disclosure, non-transitory computer-readable storage media stores instructions for performing the operations of the described embodiments. Also in other embodiments, systems for performing the operations of the described embodiments are disclosed.

[0147] It is to be understood that the above description is intended to be illustrative rather than restrictive. After reading and understanding the above description, many other embodiments will be clear to those skilled in the art. Therefore, the scope of the present disclosure can be determined with reference to the attached claims together with the full range of equivalents of the rights of such claims.

Claims

1. A method comprising: analyzing, by a processing device, one or more frames sampled from a first media item of an event to identify one or more candidate event periods within the one or more frames; For each of the one or more frames, determining whether a candidate event period of the one or more candidate event periods satisfies one or more conditions; as well as In response to determining that the candidate event period of the one or more candidate event periods satisfies the one or more conditions, identifying the candidate event period as an actual event period for dividing the time of the event, and Mapping data is generated that maps the actual event period to a timestamp associated with a corresponding frame of the one or more frames of the first media item.

2. The method according to claim 1, further comprising: identifying annotation content associated with the first media item; annotating the first media item with the annotation content based on the mapping data; as well as The first media item with the annotated content is provided for presentation at a client device.

3. The method according to claim 1, wherein: Determining whether the candidate event period in the one or more candidate event periods satisfies the one or more conditions includes: For each of the one or more frames, generating a score for each of the one or more candidate event periods, the score indicating a likelihood that a corresponding candidate event period of the one or more candidate event periods is the actual event period for dividing the time of the event; and A determination is made as to whether any of the scores for the one or more candidate event periods satisfies a threshold score.

4. The method according to claim 3, wherein: Determining whether the candidate event period in the one or more candidate event periods satisfies the one or more conditions further includes: In response to determining that more than one of the scores for the one or more candidate event periods satisfies the threshold score, a candidate event period having a highest score is selected.

5. The method according to claim 4, wherein: Determining whether the candidate event period in the one or more candidate event periods satisfies the one or more conditions further includes: It is determined whether the candidate event period having the highest score is present within a corresponding frame of the one or more frames.

6. The method according to claim 1, further comprising: causing annotation content analysis to be performed on the first media item while the first media item is being streamed simultaneously to one or more client devices, the annotation content analysis being operable to identify annotation content relevant to the first media item at a particular timestamp associated with the first media item; identifying the actual event period of the event using the timestamp associated with the first media item and the mapping data; as well as The annotation content is provided to a first client device for presentation with the instance of the first media item.

7. The method according to claim 1, further comprising: Additional mapping data is generated that maps the actual event period of the event to a timestamp associated with a corresponding frame of the one or more frames of the second media item.

8. The method according to claim 7, further comprising: annotating a second media item with annotation content based on the actual event period and the additional mapping data; as well as The second media item with the annotated content is provided for presentation at a second client device.

9. The method according to claim 1, comprising: analyzing the one or more frames to identify one or more candidate event clocks within the one or more frames; For each of the one or more frames, determining whether a candidate event clock among the one or more candidate event clocks satisfies one or more conditions; as well as In response to determining that the candidate event clock among the one or more candidate event clocks satisfies the one or more conditions, the candidate event clock is identified as an actual event clock for timing the event, and mapping data is generated, which maps the actual event clock to a timestamp associated with a corresponding frame among the one or more frames of the first media item.

10. The method according to claim 1, wherein: Analyzing the one or more frames to identify the one or more candidate event periods within the one or more frames includes: An optical character recognition operation is performed on the one or more frames to identify i) text indicating the one or more candidate event periods, and ii) position information describing the position of the one or more candidate event periods relative to the one or more frames.

11. A system comprising: Memory; as well as a processing device, coupled to the memory, configured to: analyzing one or more frames sampled from a first media item of an event to identify one or more candidate event periods within the one or more frames; For each of the one or more frames, determining whether a candidate event period of the one or more candidate event periods satisfies one or more conditions; as well as In response to determining that the candidate event period of the one or more candidate event periods satisfies the one or more conditions, identifying the candidate event period as an actual event period for dividing the time of the event, and Mapping data is generated that maps the actual event period to a timestamp associated with a corresponding frame of the one or more frames of the first media item.

12. The system according to claim 11, wherein the processing device is further configured to: identifying annotation content associated with the first media item; annotating the first media item with the annotation content based on the mapping data; and The first media item with the annotated content is provided for presentation at a first client device.

13. The system according to claim 11, wherein: In order to determine whether the candidate event period in the one or more candidate event periods satisfies the one or more conditions, the processing device is configured to: For each of the one or more frames, generating, for each of the one or more candidate event periods, a score indicating a likelihood that a corresponding candidate event period of the one or more candidate event periods is the actual event period for timing the event; as well as A determination is made as to whether any of the scores for the one or more candidate event periods satisfies a threshold score.

14. The system according to claim 13, wherein: In order to determine whether the candidate event period in the one or more candidate event periods satisfies the one or more conditions, the processing device is further configured to: In response to determining that more than one of the scores for the one or more candidate event periods satisfies the threshold score, a candidate event period having a highest score is selected.

15. The system of claim 14, wherein: In order to determine whether the candidate event period in the one or more candidate event periods satisfies the one or more conditions, the processing device is further configured to: It is determined whether the candidate event period having the highest score is present within a corresponding frame of the one or more frames.

16. The system according to claim 11, wherein the processing device is further configured to: analyzing the one or more frames to identify one or more candidate event clocks within the one or more frames; For each of the one or more frames, determining whether a candidate event clock of the one or more candidate event clocks satisfies one or more conditions; and In response to determining that the candidate event clock among the one or more candidate event clocks satisfies the one or more conditions, the candidate event clock is identified as an actual event clock for timing the event, and mapping data is generated, which maps the actual event clock to a timestamp associated with a corresponding frame among the one or more frames of the first media item.

17. A non-transitory computer-readable medium comprising instructions that, in response to execution by a processing device, cause the processing device to perform operations comprising: analyzing, by the processing device, one or more frames sampled from a first media item of an event to identify one or more candidate event periods within the one or more frames; For each of the one or more frames, determining whether a candidate event period of the one or more candidate event periods satisfies one or more conditions; as well as In response to determining that the candidate event period of the one or more candidate event periods satisfies the one or more conditions, identifying the candidate event period as an actual event period for timing the event, and Mapping data is generated that maps the actual event period to a timestamp associated with a corresponding frame of the one or more frames of the first media item.

18. The non-transitory computer readable medium of claim 17, the operations further comprising: identifying annotation content associated with the first media item; annotating the first media item with the annotation content based on the mapping data; as well as The first media item with the annotated content is provided for presentation at a first client device.

19. The non-transitory computer readable medium of claim 17, the operations further comprising: causing annotation content analysis to be performed on the first media item while the first media item is being streamed simultaneously to one or more client devices, the annotation content analysis being operable to identify annotation content relevant to the first media item at a particular timestamp associated with the first media item; identifying the actual event period of the event using the timestamp associated with the first media item and the mapping data; as well as The annotation content is provided to a first client device for presentation with the instance of the first media item.

20. The non-transitory computer readable medium of claim 19, the operations further comprising: generating additional mapping data that maps the actual event period of the event to a timestamp associated with a corresponding frame of one or more frames of a second media item; annotating a second media item with the annotation content based on the actual event period and the additional mapping data; as well as The second media item with the annotated content is provided for presentation at a second client device.

Images