Techniques for automatically generating playback clips of media content for critical events

Abstract

Techniques for dynamically generating playback clips for key events occurring are disclosed herein. According to some embodiments, a technique can be implemented at a computing device and includes the steps of: (1) providing media content to at least one machine learning model to output a plurality of segments of the media content, where each segment is marked with a respective at least one classification describing a property of the segment; (2) receiving a plurality of key events; and (3) for each of the plurality of key events: analyzing at least one of the plurality of segments according to the key event to determine a start point and an end point of a playback clip for the key event, and generating the playback clip based on (i) the media content and (ii) the start point and the end point.

Description

Technical Field

[0001] The described implementation scheme illustrates a technique for automatically generating replay clips of media content for key events. These key events and their corresponding replay clips can then be presented to users for viewing. Background Technology

[0002] The number of real-world events that an individual might be interested in only continues to grow over time. This is due to, for example, the increasing number of events occurring around the world, and the decreasing production and distribution costs of capturing event reports and disseminating them to viewers worldwide. In this respect, it would be overwhelming for an individual to watch all the events that might interest them. Furthermore, selecting replay clips that an individual might be interested in watching again could also be difficult. Summary of the Invention

[0003] This application describes a technique for automatically generating replay clips of media content for key events. These key events and their corresponding replay clips can then be presented to users for viewing.

[0004] One implementation describes a method for dynamically generating replay clips for key events that occur. According to some implementations, the method can be implemented at a computing device and includes the following steps: (1) providing media content to at least one machine learning model to output multiple segments of the media content, wherein each segment is labeled with at least one corresponding classification describing the properties of the segment; (2) receiving multiple key events; and (3) for each of the multiple key events: analyzing at least one segment of the multiple segments based on the key event to determine a start point and an end point of the replay clip for the key event, and generating the replay clip based on (i) the media content and (ii) the start point and the end point.

[0005] The method may further include selecting the at least one machine learning model based on one or more of the type of the media content, the type of the event corresponding to the media content, or the type of device that generates the media content.

[0006] The method may further include: for each key event, analyzing the optical flow of the plurality of segments based on the key event to determine the start point and the end point of the replay clip for the key event. The optical flow may include one or more of the following: camera panning direction, change of camera panning direction, change of camera panning speed, change of camera zoom level, change of camera zoom speed, or change of camera source video.

[0007] The method may further include: for each key event, analyzing the audio data of the plurality of segments based on the key event to determine the start point and the end point of the replay clip for the key event.

[0008] The media content may include media content from multiple different video sources. The replay clip may be generated using the media content from the multiple different video sources. For example, the replay clip may be generated using the media content by splicing different video sources from the multiple different video sources to create an optimal replay clip. One or more segments of the multiple segments of the media content between the start point and the end point may be omitted from the replay clip based on their corresponding classification.

[0009] Other embodiments include a non-transitory computer-readable storage medium configured to store instructions that, when executed by a processor included in a computing device, cause the computing device to perform various steps of any of the methods described above. Further embodiments include a computing device configured to perform various steps of any of the methods described above.

[0010] Other aspects and advantages of the embodiments described herein will become apparent from the following detailed description, taken in conjunction with the accompanying drawings illustrating the principles of the described embodiments by way of example. Attached Figure Description

[0011] This disclosure will be more readily understood from the following detailed description taken in conjunction with the accompanying drawings, wherein similar reference numerals denote similar structural elements.

[0012] Figure 1 Block diagrams illustrating different components of a system for implementing the various technologies described herein, according to some implementation schemes.

[0013] Figure 2 Examples are given based on some implementation schemes. Figure 1 The sequence diagram shows the interaction between the event activity provider, media content analyzer, media content library, and client computing device to enable the implementation of the technologies discussed in this paper.

[0014] Figure 3 An example is given of a method for dynamically generating replay clips based on key events that occur, according to some implementation schemes.

[0015] Figures 4A to 4F A conceptual diagram illustrating an example process according to some implementation schemes, through which a client computing device categorizes media content.

[0016] Figure 5Detailed views of representative computing devices that can be used to implement the various technologies described herein, according to some implementation schemes, are illustrated. Detailed Implementation

[0017] This section describes representative applications of the methods and apparatus according to this application. These examples are provided only to add context and aid in understanding the described embodiments. Therefore, it will be apparent to those skilled in the art that the described embodiments can be practiced without some or all of these specific details. In other instances, well-known processing steps have not been described in detail to avoid unnecessarily obscuring the described embodiments. Other applications are possible, such that the following examples should not be considered limiting.

[0018] In the following detailed description, reference is made to the accompanying drawings, which form part of this specification, and specific embodiments according to the described embodiments are illustrated by way of example in the drawings. While these embodiments are described in sufficient detail to enable those skilled in the art to practice the described embodiments, it should be understood that these examples are not limiting; other embodiments are permitted for use, and modifications may be made without departing from the spirit and scope of the described embodiments.

[0019] Figure 1 Block diagrams illustrating different components of a system 100 for implementing the various technologies described herein, according to some implementation schemes, are shown. For example... Figure 1 As shown, system 100 includes one or more event activity providers 102, one or more media content analyzers 112, one or more media content libraries 118, one or more replay clip distributors 120, and one or more client computing devices 124.

[0020] According to some implementations, a given event activity provider 102 can be configured to acquire information related to a real-world event that has occurred, and this information is... Figure 1The event activity information is exemplified as 103. The event activity provider 102 can then be configured to generate a key event 110 based on the event activity information 103. In one example, the event activity provider 102 may implement an application programming interface (API) that allows information associated with a real-world event to be provided to the event activity provider 102. Without departing from the scope of this disclosure, any conceivable method may be used to obtain / provide real-world information. For example, one or more computing devices may be located at the stadium and configured to collect information using automated, semi-automated, or manual methods. As an example of an automated method, one or more sensors, cameras, etc., may be used to automatically parse and report the event as it occurs (e.g., using machine learning). As an example of a semi-automated method, the aforementioned equipment may be used to present information to an individual (e.g., an operator), which is then analyzed / customized by the individual. As an example of a manual method, one or more individuals may be assigned to collect information and use the aforementioned equipment to input that information. Additionally, it should be noted that the API / event activity provider 102 can be configured to identify events by analyzing raw / unprocessed information received through the API. For example, the API can receive audio and / or video streams of any event (e.g., sports events, concert events, online gaming events, etc.) and utilize various techniques (e.g., machine learning) to efficiently parse the occurring events. It should be noted that the foregoing examples are not intended to be limiting, and the event activity provider 102 can utilize any method to efficiently obtain and organize useful information about real-world events.

[0021] According to some implementations, the event activity provider 102 may be configured to implement event analysis logic to enable the identification of critical events 110. More specifically, the event analysis logic may be configured to implement a set of critical event rules that defines various criteria by which critical events 110 can be identified. In one example, the critical event rule set may specify that, for a baseball game, critical events include scoring and home runs. Other more common events (such as strikeouts, walks, inning changes, etc.) may not qualify as critical events 110. In another example, the critical event rule set may specify that, for an ice hockey game, critical events include goals and brawls. In yet another example, the critical event rule set may specify that, for a football (English football) game, critical events include goals and scoring opportunities. It should be noted that the foregoing examples are not intended to be limiting, and the event activity provider may be configured to identify critical events using any criteria and at any level of granularity without departing from the scope of this disclosure. Once a key event 110 is identified, the event activity provider 102 can store information about situations related to the key event 110 (e.g., one or more times associated with the key event 110, contextual information associated with the key event 110, effects associated with the key event 110 (e.g., speaker / audience response, change in outcome, etc.), audio / video content associated with the key event, etc.).

[0022] According to some implementations, when a key event 110 is identified, the event activity provider 102 may be configured to provide information about the key event 110 to the media content analyzer 112. According to some implementations, the media content analyzer 112 may be configured to receive media content 115 related to various real-world events (e.g., concert events, gaming events, sporting events, award events, etc.) from various sources. The media content 115 may represent, for example, live (with or without delay) or recorded audio and / or video streams. The media content 115 may include a main audio and / or video stream synthesized from multiple different audio and / or video sources over a continuous timeline. That is, multiple different audio and / or video sources may be used to provide content for the main audio and / or video streams, but the content utilized from the different audio and / or video sources does not overlap in time. The media content 115 may also include one or more secondary audio and / or video streams. Each secondary audio and / or video stream may include an audio and / or video stream from a corresponding one of the multiple different audio and / or video sources.

[0023] like Figure 1As shown, media content analyzer 112 can be configured to implement media content analysis logic 114 for analyzing media content 115 to generate replay clips 116. According to some embodiments, media content analysis logic 114 is configured to identify and classify multiple segments within media content 115. In one embodiment, machine learning is used to segment and classify media content 115. Each identified segment is labeled with at least one corresponding classification describing the nature of that segment. The identified segments can be used to determine information to be extracted from media content 115 based on key events 110. Examples of different classifications include, but are not limited to, “close-up,” “wide-angle,” “crowd,” “graphic,” “replay,” “side angle,” “split screen,” “substitutes’ bench,” “transition,” “sports-specific (subclass),” “referee,” “time,” “scoreboard,” “unknown,” etc.

[0024] "Close-up" can be used to categorize clips as close-ups of people, players, or other participants in a real-world event—a magnified view. "Wide-angle" can be used to categorize clips as wide-angle views of the venue where a real-world event takes place. For example, if the real-world event is a sporting event, the venue is the playing field, such as a stadium, football field, ice rink, etc. "Crowd" can be used to categorize clips as showing multiple spectators watching a real-world event. "Graphics" can be used to categorize clips as displaying graphic information in addition to (or as a supplement to) the real-world event. Graphics can be used to display various types of information, such as a statistical summary of the real-world event up to that point, details about the people, players, or other participants in the real-world event, other related real-world events, advertisements, etc. "Replay" can be used to categorize clips as replays of previous activities in a real-world event. For example, a clip can be analyzed to determine if it is presented in slow motion, as slow motion presents a high probability that the clip is a replay. A clip can also be analyzed to determine if it matches previously presented content, which also indicates that the clip is a replay.

[0025] "Side Angle" relates to the gap between the "Close-up" and "Wide Angle" categories. In this respect, "Side Angle" can be used to categorize clips focusing on players performing actions of interest, goal celebrations, referee interventions, etc. "Split Screen" can be used to categorize clips showing two or more signal sources simultaneously (e.g., cameras, animations, etc.), such as Video Assistant Referee (VAR) replays. "Substitutes' Bench" is similar to "Close-up" and can be used to categorize clips focusing on coaches, players, managers, etc., in the substitutes' bench area (e.g., the players' bench, sidelines, penalty area, etc.). "Transition" can be used to categorize clips that transition smoothly between key moments and one or more replays (e.g., when graphics associated with a team, league, broadcaster, etc. appear). "Sports-Specific (Subcategories)" can be used to categorize clips as specific to a particular sport. For example, in football, there are often camera views from behind the goal, which could be categorized as "Football (Camera Behind the Goal)". In another example, in soccer, there are often camera views of penalty kicks from in front of the goal, which can be categorized as "Soccer (Penalty Camera)". In yet another example, in soccer, there are often camera views of the corners of the field, which can be categorized as "Soccer (Corner Camera [1-4])".

[0026] "Referee" can be used to categorize segments into those involving one or more referees in an event. "Time" can be used to categorize segments into those including one or more representations of any time, timer, clock, etc., related to the event (such as a running match timer, sub-event timer (e.g., attack time limit), timeout timer, current real-world time, etc.). "Scoreboard" can be used to categorize segments into those including one or more representations of scores, statistics, etc., related to the event. The "Time" and / or "Scoreboard" categorization can be used, for example, to detect correlations between the times when events occurred during an event. Given that scoreboard-related graphics are not displayed during replays in most cases, such categorization can also be used to detect when a replay occurred. "Unknown" can be used to categorize segments that display activities (such as commercials) unrelated to the event of interest.

[0027] Additionally, the audio content accompanying the segments can be analyzed to effectively categorize the segments (e.g., a speaker uttering words instructing playback, such as "Let's watch it again"), graphic / text content (e.g., captions instructing playback), and so on. It should be noted that the foregoing examples are not intended to be limiting, and any information associated with the media content can be analyzed at any level of granularity to effectively segment and categorize the media content.

[0028] In addition, such as Figure 1As shown, the media content analyzer 112 can be configured to manage optical flow rule sets 113. According to some embodiments, each optical flow rule set 113 may correspond to one or more types of real-world events (e.g., baseball, hockey, football, English football, boxing, etc.), one or more types of media content being analyzed (e.g., audio, video, text, etc.), the method of obtaining the media content (e.g., professional camera systems, audience cameras (e.g., mobile device cameras), the desired segments / categories, etc.). It should be noted that the foregoing examples do not represent an exhaustive list, and any number of optical flow rule sets 113 can be configured and selected for use based on any information at any granularity without departing from the scope of this disclosure.

[0029] According to some implementation schemes, a given set of optical flow rules 113 can define optical flow rules that are effective for generating and classifying segments of media content associated with real-world events. Optical flow can be defined as the apparent motion patterns of objects, surfaces, and edges in a visual scene caused by the relative motion between the observer and the scene. Optical flow can also be defined as the distribution of apparent movement speeds of brightness patterns in an image. Therefore, the nature of the real-world event will determine the optical flow of the video stream for that event. Examples of optical flow characteristics of the video stream may include camera panning direction (e.g., left or right), changes in camera panning direction (e.g., from left to right or from right to left), changes in camera panning speed without changing direction (e.g., acceleration or deceleration of panning), changes in camera zoom level (e.g., from zooming in to zooming out and from zooming out to zooming in), changes in camera zoom speed (e.g., acceleration or deceleration of zooming), slowing down the video stream (indicating playback), changes in the video source, etc.

[0030] For real-world events that are variable and lack a clear definition of significance, the ability to infer significant events is particularly useful. For example, in a football (English football) match, a considerable amount of time can pass between the time of a reset (such as a throw-in or goal kick) and the time of a goal. Therefore, one or more optical flow rule sets 113 can be used to infer significant events from the optical flow of one or more content streams (e.g., audio, video, text, etc.). For example, in a football (English football) match, a change of possession before a goal (which could constitute a key event 110) could be a suitable / relevant starting point for replay clip 116. It is noteworthy that a change of possession typically results in a change in camera panning direction. Therefore, optical flow rule set 113 could include rules for detecting changes in camera panning direction shortly before a goal to identify the starting point of replay clip 116. Once a goal is scored, the camera signal source can change to capture celebrations from players, managers, coaches, spectators, etc. Therefore, optical flow rule set 113 could include rules for detecting changes in camera signal source shortly after a goal to identify the ending point of replay clip 116. In one implementation, it may be desirable to include the goal celebration in replay clip 116, in which case different / supplementary rules for the finish line may be used.

[0031] Similarly, in hockey, for example, a considerable amount of time can pass between a reset (such as a face-off) and a goal. For instance, in hockey, a one-on-one situation prior to a goal that could constitute key event 110 could be the appropriate / relevant starting point for replay clip 116. Typically, a one-on-one situation causes a change in camera panning speed. Therefore, the optical flow rule set 113 could include rules for detecting changes in camera panning speed shortly before a goal for replay clip 116. Once a goal is scored, the camera signal source can be changed to capture celebrations from players, managers, coaches, spectators, etc. Therefore, the optical flow rule set 113 could include rules for detecting changes in camera signal source shortly after a goal to identify the end point of replay clip 116.

[0032] Therefore, it should be understood that the optical flow rule set 113 may include different rules for different real-world events based on the optical flow associated with the visual capture event. Furthermore, multiple optical flow characteristics can be analyzed in various combinations to improve the identification of important events within an event. Additionally, as described herein, the identification of important events may also depend on the classification of segments. Furthermore, the audio stream of the media content 115 can also be used to supplement the optical flow rule set. For example, increases or decreases in crowd noise volume, increases or decreases in narrator volume, etc., can be used to identify important events.

[0033] Therefore, in one example, media content analyzer 112 receives a key event 110 indicating a goal was scored at 5:55:22 PM in a specific football (English football) match. Media content analyzer 112 can then determine whether media content 115 includes information related to that specific football (English football) match, such as a live stream / ongoing recording of that specific football (English football) match. Then, assuming that media content 115 does indeed include this information, media content analysis logic 114 can (e.g., using the selection techniques discussed herein) identify an appropriate set of optical flow rules 113 to extract information from media content 115 based on key event 110. For example, media content analysis logic 114 can extract a segment based on the time of 5:55:22 PM indicated in key event 110. A change of possession (e.g., at 5:48:44 PM) can be inferred from the optical flow analysis of the segment (and / or other information included in the media content). Therefore, the start point of replay clip 116 can be set to occur shortly before the time of possession change (e.g., based on the fill time specified in optical flow rule set 113). The end point of replay clip 116 can be set to occur at the end point of the analyzed segment.

[0034] The replay clip 116 generated by the media content analyzer 112 can be provided by the media content analyzer 112 to one or more media content libraries 118. According to some embodiments, the media content library 118 receiving the replay clip 116 can be configured to store the replay clip 116 using any conceivable method. For example, the media content library 118 can store the replay clip 116 in a database communicatively coupled to the media content library 118. In doing so, the media content library 118 can generate a unique identifier to identify the replay clip 116. In one embodiment, the media content library 118 can provide the unique identifier to the media content analyzer 112. The media content analyzer 112 can then associate the unique identifier with a key event 110 associated with the replay clip 116. It should be noted that the key event 110 described herein is not limited to a single / corresponding replay clip 116. Rather, a given key event 110 can refer to any number of replay clips 116 that the media content analyzer 112 determines are associated with the key event 110. When this occurs, the media content library 118 can generate a corresponding unique identifier for the media content clip 116.

[0035] Replay clip 116 and each associated unique identifier may also be provided by media content library 118 to one or more replay clip distributors 120. In one embodiment, replay clip distributor 120 may maintain necessary information to manage the subscription preferences of multiple users of client computing device 124. For example, fans of a particular sports team may subscribe to receive replay clips 116 of that sports team in real-time (or near real-time). Replay clip distributor 120 may also be configured to respond to requests for replay clip 116. For example, fans of a particular sports team may search for replay clips of that sports team. Replay clip distributor 120 may push key event 110 and associated replay clip 116 to a user's client computing device 124 based on subscription preferences. In one example, key event 110 and replay clip 116 may be sent separately. That is, key event 110 may be pushed to client computing device 124 and displayed on that device (e.g., on a display device communicatively coupled to client computing device 124) to interact with the user. Then, in response to a request from the user during the interaction, the client computer device 124 may retrieve (e.g., stream or download) the replay clip 116. In another example, the replay clip distributor 120 may push the replay clip 116 along with the key event 110 to the user / client computing device 124. In such an example, the user of the client computing device 124 will not need to obtain the replay clip 116 and the key event 110 separately. In either example, the user of the client computing device 124 may be an individual user or an administrator of a broadcasting platform (such as one or more social media platforms).

[0036] therefore, Figure 1 This paper describes a system that enables the automatic generation and provision of replay clips 116 associated with key real-world events to user / client computing devices 124. The following section combines... Figure 2 Provided Figure 1 A more detailed breakdown of the interactions between the various entities illustrated (and described above).

[0037] Figure 2 A sequence diagram 200 illustrates the interaction between a media content analyzer 112, a media content library 118, a replay clip distributor 120, and a client computing device 124 according to some embodiments, enabling the implementation of the techniques discussed herein. Step 202 may involve the media content analyzer 112 identifying segments in the media content 115 and classifying them (e.g., using the techniques described herein).

[0038] Step 204 involves the media content analyzer 112 receiving a key event 110 from the event activity provider 102. The request may include, for example, the key event 110 (or a subset of information stored by the key event 110) and the time of the key event 110.

[0039] Next, step 206 involves the media content analyzer 112 (e.g., using the techniques described herein) generating replay clip 116. Then, step 208 involves the media content analyzer 112 (e.g., using the techniques described herein) providing the replay clip 116 to the media content library 118. At step 210, the media content library 118 stores the replay clip 116 and generates a unique identifier.

[0040] Subsequently, at step 212, the media content library 118 (e.g., using the techniques described herein) provides a unique identifier to the replay clip assigner 120. Next, at step 214, the replay clip assigner 120 (e.g., using the techniques described herein) identifies one or more users who have subscribed to the information.

[0041] Therefore, at step 216, the replay clip assigner 120 (e.g., using the techniques described herein) provides one or more key events 110 to the client computing device 124. Again, the replay clip assigner 120 may provide one or more key events 110 in response to a request issued by the client computing device 124 (e.g., a search query, page load, etc.). In another example, the replay clip assigner 120 may determine, for example, by analyzing the preferences and / or subscriptions of a user associated with the client computing device 124, that a user may be interested in one or more key events 110.

[0042] At step 218, the client computing device 124 receives a selection of one of the key events 110 provided by the playback clip assigner 120 in step 216. This selection may occur, for example, when a user is interacting with a specific key event 110 among the one or more key events 110 and selecting available options associated with that specific key event 110 (e.g., viewing options for media content clips 116 associated with that specific key event 110).

[0043] Next, at step 220, the client computing device 124 obtains a unique identifier for the selected key event 110 and provides the unique identifier to the media content library 118 (e.g., as described above in conjunction with...). Figure 1 (As described above). Subsequently, at step 222, the media content library 118 obtains the media content clip 116 based on a unique identifier and provides the media content clip 116 to the client computing device 124 (e.g., as described above in conjunction with...). Figure 1 (As described). After that (and not in) Figure 2 (As illustrated in the example), the client computing device 124 enables its users to interact with the media content clip 116 (e.g., play back the media content clip 116, share the media content clip 116 with others, etc.).

[0044] In another implementation, replay clip allocator 120 may retrieve replay clip 116 from media content library 118 instead of providing the key event and ID at step 216. Replay clip allocator 120 may then send replay clip 116 to client computing device 124 without client computing device 124 explicitly requesting replay clip 116.

[0045] Figure 3 An example of a method 300 for dynamically generating replay clips 116 based on key events 110 that occur is illustrated according to some embodiments. According to some embodiments, this method may be implemented by one or more computing devices associated with a media content analyzer 112. For example... Figure 3 As shown, method 300 begins at step 302, in which media content analyzer 112 receives media content 115. As previously described, media content 115 may be a live (with or without delay) or recorded audio stream, video stream, text stream, etc.

[0046] At step 304, the media content analyzer 112 utilizes at least one machine learning model configured to analyze the media content 115 and output multiple segments of the media content 115. In one embodiment, machine learning is used to analyze the main audio stream and / or video stream (i.e., media stream) of the media content 115 frame-by-frame (or otherwise) to detect transitions in the media stream. According to some embodiments, detecting transitions may involve identifying the end of the current segment and the beginning of the next segment. Specifically, once a transition is detected, the frame immediately preceding the frame including the transition is identified as the last frame in the current segment, and the frame including the transition is identified as the first frame in the next segment. Examples of transitions include changes in camera signal sources (e.g., changes in auxiliary audio streams and / or video streams used for the main audio stream and / or video stream). Another example of a transition includes introducing graphics into the main video stream. Yet another example of a transition includes changes in graphics within the main video stream. Still another example of a transition includes a fade from one video stream (i.e., shot) to another video stream (this typically occurs between replayed shots).

[0047] At step 306, the media content analyzer 112 utilizes at least one machine learning model to classify the segments as each segment is identified among the multiple segments (e.g., using one or more optical flow rule sets in the optical flow rule set 113 described herein). According to some embodiments, each segment can be labeled with at least one corresponding category describing the nature of the segment (e.g., stadium shot, close-up, slow-motion replay, etc.). The corresponding confidence level in the classification can also be determined on a scale of 0.00 to 1.00 (or other scales), which can be used as the corresponding weight when generating the replay clip 116 from the segments.

[0048] At step 308, the media content analyzer 112 receives a plurality of key events 110. At step 310, for each key event 110, the media content analyzer 112 analyzes at least one segment of the plurality of segments based on the key event 110 to determine the start and end points of the replay clip 116 of the key event 110. For example, as previously described, the received key event 110 may include an identifier of the key event 110 and the time at which the key event 110 occurs. In one embodiment, at step 310a, the media content analyzer 112 identifies and retrieves the main segment that corresponds temporally to the key event 110. In this way, the retrieved segment may include the content of the key event 110 itself (e.g., a goal scored). The media content analyzer 112 may also retrieve any number of additional segments that appear before, during (e.g., another camera signal source, audio signal source, etc.), and / or after the main segment. In some implementations, segments that are purely classified as graphics (e.g., score screens, player profile screens, team profile screens, etc.) may be ignored because these segments may not have any content added that is relevant to the replay clip 116.

[0049] At step 310b, the media content analyzer 112 determines the start and end points of the replay clip 116 based on the segments (e.g., using the techniques described herein). For example, the start and end points of the replay clip 116 may be determined as the start and end points of a main segment, the start and / or end points of a main segment adjusted based on other segments (e.g., their classification), etc.

[0050] At step 310c, the media content analyzer 112 generates a replay clip 116 based on the media content and the start and end points. In one embodiment, the replay clip 116 may include only audio and / or video from the main audio stream and / or video stream. Thus, if the start point is determined to be at time 1 and the end point to be at time 2, the replay clip 116 may include the main audio stream and / or video stream from time 1 to time 2. In another embodiment, the replay clip 116 may include audio and / or video from one or more secondary audio streams and / or video streams. Thus, for example, the replay clip 116 may include the main audio stream and / or video stream from time 1 to time 2, and audio streams and / or video streams from time 1 to time 2 for one or more secondary audio streams and / or video streams. It should be understood that if one of the secondary audio streams and / or video streams is substantially the same as the main audio stream and / or video stream, it may be omitted from the replay clip 116 to avoid redundancy. In yet another embodiment, the replay clip 116 may include portions of different audio and / or video streams from time 1 to time 2.

[0051] Figures 4A to 4F A conceptual diagram 400 illustrates an example process according to some implementations, through which a media content analyzer 112 identifies and classifies segments of media content 115 accessible to the media content analyzer 112. As a first step, the media content analyzer 112 may employ media content analysis logic 114 to identify one or more optical flow rule sets 113 suitable for analyzing the media content 115 (e.g., based on the type of media content 115 (e.g., audio, video, text, etc.), the type of event corresponding to the media content 115 (e.g., sports, event, etc.), the system used to generate the media content 115 (e.g., the type of camera, microphone, detection device, etc.)). It should be noted that the foregoing examples are not intended to be limiting, and the optical flow rule set 113 can be formulated and selected based on any type of information and at any level of granularity without departing from the scope of this disclosure. In any case, once the media content analyzer 112 selects an appropriate optical flow rule set 113, it can then begin analyzing the media content 115 to generate segments and classify them according to the techniques described herein.

[0052] like Figure 4A As shown, the video stream of media content 115 includes a close-up view 402 from the first camera 1. Figure 4BAs shown, the video stream has switched from the second camera, camera 2, to a wide-angle view of the court 404. Therefore, the media content analyzer 112 detects the camera change and creates the first segment 406. The first segment 406 ends just before the camera switch and is classified as a "close-up" with a confidence level of 1.00 (i.e., complete confidence). As described herein, it should be noted that two or more classifications can be assigned to any segment that meets the requirements, characteristics, attributes, etc., of the classification.

[0053] like Figure 4C As shown, the video stream is still a wide-angle view from the second camera, camera 2, but graph 408 has been inserted into the video stream. Graph 408 includes, for example, the score of the match and the current time. Therefore, the media content analyzer 112 detects the addition of graph 408 and creates a second segment 410. The second segment 410 begins immediately after the end of the first segment 406 and ends just before the addition of graph 408. The second segment 410 is classified as "Sports Field" with a confidence level of 1.00.

[0054] like Figure 4D As shown, the video stream is still a wide-angle view from the second camera, camera 2, but image 408 has changed to image 412. Specifically, image 412 no longer includes the time frame included in image 408. Therefore, the media content analyzer 112 detects the change from image 408 to image 412 and creates a third segment 414. The third segment 414 begins immediately after the end of the second segment 410 and ends just before the modification of image 408 / the introduction of image 412. Although the third segment 414 includes image 408, image 408 does not substantially interfere with the view of the court. Therefore, the third segment 414 is classified as "court" rather than "image" with a confidence level of 0.90.

[0055] like Figure 4E As shown, the video stream is still a wide-angle view from the second camera, camera 2, but image 412 has been completely removed. Therefore, the media content analyzer 112 detects the removal of image 412 and creates a fourth segment 416. The fourth segment 416 begins immediately after the end of the third segment 414 and ends just before the removal of image 412. Similar to... Figure 4D Although the fourth segment 416 includes figure 412, figure 412 does not substantially interfere with the view of the court. Therefore, the fourth segment 416 is classified as "court" rather than "figure" with a confidence level of 0.90.

[0056] like Figure 4FAs shown, the video stream has switched to a close-up view 418 from the third camera (camera 3). Therefore, the media content analyzer 112 detects the camera change and creates a fifth segment 420. The fifth segment 420 begins immediately after the end of the fourth segment 416 and ends just before the camera switch. The fifth segment 420 is categorized as "Sports Field" with a confidence level of 1.00.

[0057] Therefore, upon receiving the corresponding combination above Figures 4A to 4F When a critical event 110 occurs in the described segment, the media content analyzer 112 determines that the critical event 110 is included in the fifth segment 420 (i.e., the main segment) from the time the critical event 110 occurs. In one embodiment, and as described herein, the media content analyzer 112 retrieves the fifth segment 420 and identifies the start and end points of the replay clip 116 as the start and end points of the fifth segment 420, respectively. In another embodiment, the media content analyzer 112 may apply football (i.e., English football) specific rules from one or more optical flow rule sets 113 to the fifth segment 420 (or any other segment). If the media content analyzer 112 detects a significant event in the segment, such as a change of possession (e.g., indicated by a camera panning shift), the start point of the replay clip 116 may (e.g., based on dynamic fill rules defined in the optical flow rule set 113) be set exactly before the significant event. In another implementation, the media content analyzer 112 may wait to receive and analyze additional (i.e., future) media content 115 to generate additional segments that can be used to generate and classify replay clips 116.

[0058] Figure 5 Detailed views of a computing device 500, according to some embodiments, that can be used to implement the various components described herein are illustrated. In particular, these detailed views illustrate components that may be included in the foregoing. Figure 1 The individual components of any computing device described.

[0059] like Figure 5As shown, computing device 500 may include a processor 502 representing a microprocessor or controller for controlling the overall operation of computing device 500. Computing device 500 may also include a user input device 508 that allows a user of computing device 500 to interact with it. For example, user input device 508 may take various forms, such as buttons, keypads, dial pads, touchscreens, audio input interfaces, visual / image capture input interfaces, sensor data input, etc. Furthermore, computing device 500 may include a display 510 (screen display) that can be controlled by processor 502 to display information to a user. Data bus 516 facilitates data transfer between at least storage device 540, processor 502, and controller 513. Controller 513 can be used to interface with and control different devices via equipment control bus 514. Computing device 500 may also include a network / bus interface 511 coupled to data link 512. In the case of wireless connectivity, network / bus interface 511 may include a wireless transceiver.

[0060] The computing device 500 also includes a storage device 540, which may include a single disk or multiple disks (e.g., an SSD), and includes a storage management module for managing one or more partitions within the storage device 540. In some embodiments, the storage device 540 may include flash memory, semiconductor (solid-state) memory, etc. The computing device 500 may also include random access memory (RAM) 520 and read-only memory (ROM) 522. ROM 522 may store programs, utilities, or processes that will be executed in a non-volatile manner. RAM 520 may provide volatile data storage and store instructions related to the operation of the computing device described herein.

[0061] Aspects, embodiments, specific implementations, or features of the described embodiments may be used individually or in any combination. Various aspects of the described embodiments may be implemented by software, hardware, or a combination of hardware and software. The described embodiments may also be embodied as computer-readable code on a computer-readable medium. A computer-readable medium is any data storage device that can store data that can be read by a computer system. Examples of such computer-readable media include read-only memory, random access memory, CD-ROM, DVD, magnetic tape, hard disk drive, solid-state drive, and optical data storage devices. Computer-readable media may also be distributed across network-coupled computer systems, such that the computer-readable code is stored and executed in a distributed manner.

[0062] For illustrative purposes, the foregoing description uses specific names to provide a thorough understanding of the described embodiments. However, it will be apparent to those skilled in the art that specific details are not required to practice the described embodiments. Therefore, the foregoing description of specific embodiments is presented for illustrative and descriptive purposes. The foregoing description is not intended to be exhaustive or to limit the described embodiments to the precise forms disclosed. It will be apparent to those skilled in the art that many modifications and variations are possible in light of the teachings above.

Claims

1. A method for dynamically generating a replay clip for an occurring key event, the method comprising, at a computing device: providing media content to at least one machine learning model to output a plurality of segments of the media content, wherein each segment is tagged with a respective at least one classification describing a property of the segment; receiving a plurality of key events; and for each key event of the plurality of key events: analyzing at least one segment of the plurality of segments against the key event to determine a start point and an end point of a replay clip for the key event, and generating the replay clip based on (i) the media content and (ii) the start point and the end point.

2. The method of claim 1, further comprising, for each key event: analyzing an optical flow of the plurality of segments against the key event to determine the start point and the end point of the replay clip for the key event.

3. The method of claim 2, wherein the optical flow comprises one or more of: a camera pan direction, a change in camera pan direction, a change in camera pan speed, a change in camera zoom level, a change in camera zoom speed, or a change in camera source video.

4. The method of claim 1, further comprising, for each key event: analyzing audio data of the plurality of segments against the key event to determine the start point and the end point of the replay clip for the key event.

5. The method of claim 1, wherein the media content comprises media content from a plurality of different video sources.

6. The method of claim 5, wherein the replay clip is generated using the media content from the plurality of different video sources.

7. The method of claim 6, wherein the replay clip is generated using the media content by stitching different video sources of the plurality of different video sources to create a best replay clip.

8. The method of claim 1, further comprising: omitting one or more segments of the plurality of segments of the media content between the start point and the end point from the replay clip based on the respective at least one classification.

9. The method of claim 1, further comprising: selecting the at least one machine learning model based on one or more of: a type of the media content, a type of an event to which the media content corresponds, or a type of a device that generated the media content.

10. A non-transitory computer-readable storage medium configured to store instructions that, when executed by at least one processor included in a computing device, cause the computing device to dynamically generate a replay clip for an occurring key event by performing steps comprising: providing media content to at least one machine learning model to output a plurality of segments of the media content, wherein each segment is tagged with a respective at least one classification describing a property of the segment; receiving a plurality of key events; and for each key event of the plurality of key events: For each key event of the plurality of key events: analyze at least one segment of the plurality of segments against the key event to determine a start point and an end point of a replay clip for the key event, and generate the replay clip based on (i) the media content and (ii) the start point and the end point.

11. The non-transitory computer-readable storage medium of claim 10, wherein the steps further comprise, for each key event: analyzing an optical flow of the plurality of segments against the key event to determine the start point and the end point of the replay clip for the key event.

12. The non-transitory computer-readable storage medium of claim 11, wherein the optical flow comprises one or more of: a camera pan direction, a change in camera pan direction, a change in camera pan speed, a change in camera zoom level, a change in camera zoom speed, or a change in camera source video.

13. The non-transitory computer-readable storage medium of claim 10, wherein the steps further comprise, for each key event: analyzing audio data of the plurality of segments against the key event to determine the start point and the end point of the replay clip for the key event.

14. The non-transitory computer-readable storage medium of claim 10, wherein the media content comprises media content from a plurality of different video sources.

15. The non-transitory computer-readable storage medium of claim 14, wherein the replay clip is generated using the media content from the plurality of different video sources.

16. The non-transitory computer-readable storage medium of claim 15, wherein the replay clip is generated using the media content by stitching different video sources of the plurality of different video sources to create a best replay clip.

17. The non-transitory computer-readable storage medium of claim 10, wherein the steps further comprise: omitting, from the replay clip, one or more segments of the plurality of segments of the media content between the start point and the end point based on the respective at least one classification.

18. The non-transitory computer-readable storage medium of claim 10, wherein the steps further comprise: selecting the at least one machine learning model based on one or more of: a type of the media content, a type of an event to which the media content corresponds, or a type of a device that generated the media content.

19. A computing device configured to dynamically generate a replay clip for a key event of an occurrence, the computing device comprising: at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the computing device to perform steps comprising: providing media content to at least one machine learning model to output a plurality of segments of the media content, wherein each segment is tagged with a respective at least one classification that describes a property of the segment; receiving a plurality of key events; and for each key event of the plurality of key events: analyzing at least one segment of the plurality of segments against the key event to determine a start point and an end point of a replay clip for the key event, and generating the replay clip based on (i) the media content and (ii) the start point and the end point.

20. The computing device of claim 19, wherein the steps further comprise, for each key event: analyzing an optical flow of the plurality of segments against the key event to determine the start point and the end point of the replay clip for the key event.

21. The computing device of claim 20, wherein the optical flow comprises one or more of: a camera pan direction, a change in camera pan direction, a change in camera pan speed, a change in camera zoom level, a change in camera zoom speed, or a change in camera source video.

22. The computing device of claim 19, wherein the steps further comprise, for each key event: analyzing audio data of the plurality of segments against the key event to determine the start point and the end point of the replay clip for the key event.

23. The computing device of claim 19, wherein the media content comprises media content from a plurality of different video sources.

24. The computing device of claim 23, wherein the replay clip is generated using the media content from the plurality of different video sources.

25. The computing device of claim 24, wherein the replay clip is generated using the media content by stitching different video sources of the plurality of different video sources to create an optimal replay clip.

26. The computing device of claim 19, wherein the steps further comprise: omitting one or more segments of the plurality of segments of the media content from the replay clip based on the respective at least one classification between the start point and the end point.

27. The computing device of claim 19, wherein the steps further comprise: selecting the at least one machine learning model based on one or more of: a type of the media content, a type of an event to which the media content corresponds, or a type of a device that generated the media content.

28. A computing device configured to dynamically generate a replay clip for a key event of an occurrence, the computing device comprising: means for providing media content to at least one machine learning model to output a plurality of segments of the media content, wherein each segment is tagged with a respective at least one classification that describes a property of the segment; means for receiving a plurality of key events; and means for, for each key event of the plurality of key events: analyzing at least one segment of the plurality of segments against the key event to determine a start point and an end point of a replay clip for the key event, and generating the replay clip based on (i) the media content and (ii) the start point and the end point.

29. The computing device of claim 28, further comprising means for, for each key event: analyzing optical flow of the plurality of segments against the key event to determine the start point and the end point of the playback clip for the key event.

30. The computing device of claim 29, wherein the optical flow comprises one or more of: a camera pan direction, a change in camera pan direction, a change in camera pan speed, a change in camera zoom level, a change in camera zoom speed, or a change in camera source video.

31. The computing device of claim 28, further comprising means for, for each key event: analyzing audio data of the plurality of segments against the key event to determine the start point and the end point of the playback clip for the key event.

32. The computing device of claim 28, wherein the media content comprises media content from a plurality of different video sources.

33. The computing device of claim 32, wherein the playback clip is generated using the media content from the plurality of different video sources.

34. The computing device of claim 33, wherein the playback clip is generated using the media content by stitching different ones of the plurality of different video sources to create an optimal playback clip.

35. The computing device of claim 28, further comprising means for: omitting, from the playback clip, one or more of the plurality of segments of the media content between the start point and the end point based on the respective at least one classification.

36. The computing device of claim 28, further comprising means for: selecting the at least one machine learning model based on one or more of a type of the media content, a type of an event to which the media content corresponds, or a type of a device that generated the media content.

Images