Augmenting sports videos with gaze-moderated embedded visualizations

Abstract

A system and method are provided for embedding visualizations in a video stream of an athletic game, according to some embodiments. The method may include identifying visualizable entities in video frames of the video stream. The method may also include segmenting the visualizable entities from background in the video frames using image segmentation models. The method may also include mapping the visualizable entities to predetermined visualizations. The method may also include embedding the visualizations in the background in the video frames, which may be based on tracking gaze of a user. The method may also include highlighting players in the video stream based on ranking the players according to an importance level for the players. The method may also include ranking the players based on gaze points of a user to adjust the players' importance levels, as well as controlling whose abilities to show.

Description

Augmenting Sports Videos with Gaze-Moderated Embedded VisualizationsGOVERNMENT FUNDING

[0001] This invention was made with government support under (i) contract no. HI- 2107328 awarded by the National Science Foundation (NSF) and (ii) contract no. RO 1HD 104969 awarded by the National Institutes of Health. The government has certain rights in this invention.TECHNICAL FIELD

[0002] This application relates generally to natural language interfaces and video processing, and more specifically to the augmentation of sports videos with visualizations.BACKGROUND

[0003] Team sports have continued to attract new fans over the past decades due to the proliferation of video broadcasting and online video platforms. However, unlike experienced fans, new or casual fans often get confused when watching sports videos. This is because they lack sufficient sports knowledge to understand the players’ complex teamwork and in-game decisions. Existing methods of providing extra information, such as scoreboards in broadcasting videos and online webpages, often fail to adequately address their confusion. These methods either cannot provide on-demand data or distract fans from the game by showing data in separate windows.

[0004] Embedded visualizations can allow audiences to access additional data without being distracted from the game video by directly displaying data in the actual scenes. However, conventional systems that augment sports videos with embedded visualizations either focus on post-game analysis rather than game-watching scenarios, or only use simple, non-interactive text labels and progress bars to show data. Some systems that incorporate embedded visualizations for game-watching scenarios use low-fidelity simulated environments (e.g., 3D simulated sports games, moving charts on white backgrounds).SUMMARY

[0005] Accordingly, there is a need for systems, methods and interfaces that address at least some of the limitations described above. Described herein are implementations of interactive embedded visualizations in real sports videos that facilitate game understanding andincrease engagement of fans when watching games. Techniques described herein enable seamless access to additional data while watching sports games that particularly benefit casual fans in their understanding and engagement of games.

[0006] Some embodiments embed visualizations into actual scenes of team sports. For team sports, such as basketball, players overlap heavily and the camera moves rapidly. To tackle this challenge, some embodiments provide a computer vision pipeline that pre-processes team sports videos for embedding visualizations. Experiments were performed to evaluate the pipeline. Some embodiments provide embedded visualizations that are informative but not overwhelming for individual audiences, who may have various levels of game literacy, data needs, and personal interests. Some embodiments provide a set of gaze-moderated embedded visualizations that leverage the user’s gaze to seamlessly present the data the user is interested in and suppress others.

[0007] In another aspect, a method is provided for embedding visualizations in a video stream of an athletic game, according to some embodiments. The method may include identifying visualizable entities in video frames of the video stream. The method may also include segmenting the visualizable entities from background in the video frames using one or more image segmentation models. The method may also include mapping the visualizable entities to predetermined visualizations. The method may also include embedding the visualizations on the background in the video frames.

[0008] In some embodiments, identifying the visualizable entities may include detecting video objects that correspond to each visualizable entity and the visualizable entities’ key body points.

[0009] In some embodiments, the video objects may include one or more bounding boxes and an identity for each visualizable entity, and wherein the key body points comprise key points for each visualizable entity.

[0010] In some embodiments, detecting the video objects may include obtaining the one or more bounding boxes and the identity in the video frames using an object detection model to locate and classify each visualizable entity into different identities.

[0011] In some embodiments, output of the object detection model may be a set of bounding boxes associated with their identities and confidence scores.

[0012] In some embodiments, detecting the video objects may include detecting bounding boxes with a confidence score greater than a predetermined threshold.

[0013] In some embodiments, identifying the visualizable entities may further include using obj ect trackers to take into account a physical entity ’ s motion information to complement detection of the video objects.

[0014] In some embodiments, each object tracker may store history of an object’s bounding boxes in previous frames and predicts the object’s bounding box in a next frame by using a motion predictor, such as a Kalman filter.

[0015] In some embodiments, using the object trackers may include: for a frame Ft, dividing detected bounding boxes into three clusters based on their score: high-quality boxes with a score greater than a first predetermined threshold, low-quality boxes with a score between the first predetermined threshold, and rejected boxes with a score lower than a second predetermined threshold. The method may also include, for each high quality box, matching it with trackers in a previous frame Ft-1 by calculating an Intersection over Union (loU) between the box and the predicted boxes of the trackers. A tracker is considered matches with the high- quality box if it maximizes the loU, wherein if matching is successful, the matched tracker is assigned to the box, and otherwise a new object tracker is initialized for the box. The method may also include, for each low-quality box, matching it with remaining trackers that have not been matched with any high-quality boxes, including if matching is successful, assigning the matched tracker to the box, and outputting all boxes with matched trackers.

[0016] In some embodiments, using the object trackers may include dividing bounding boxes into a high-quality group, a low-quality group, and a rejected group, based on confidence score for each bounding box and a predetermined threshold.

[0017] In some embodiments, using the object trackers may include matching existing object trackers with bounding boxes in the high-quality group, or initializing new object trackers for bounding boxes in the high-quality group if matching is not successful.

[0018] In some embodiments, using the object trackers includes matching existing object trackers with bounding boxes in the low-quality group.

[0019] In some embodiments, using the object trackers includes outputting bounding boxes with matched trackers.

[0020] In some embodiments, identifying the visualizable entities may further include using a pose estimation model to obtain each visualizable entities’ key points selected from the group consisting of: head, hands, hip and feet.

[0021] In some embodiments, detecting the video objects includes find-tuning an object detection model on a dataset of the visualizable entities.

[0022] In some embodiments, the method may further include concurrently performing identifying the visualizable entities in the video frames of the video stream, segmenting the visualizable entities from the background in the video frames using semantic segmentation, mapping the visualizable entities to predetermined visualizations, and / or embedding the visualizations on the background in the video frames.

[0023] In some embodiments, the method may further include obtaining camera parameters for a camera used to generate the video stream or estimating the camera parameters using one or more camera calibration techniques, and displaying visualizations that are static relative to ground using the camera parameters.

[0024] In some embodiments, the visualizable entities may include physical entities that appear in the video frames of the video stream.

[0025] In some embodiments, the visualizable entities may be players in a basketball game, and the background may be the ground.

[0026] In some embodiments, embedding the visualizations on the background in the video frames may be based on tracking gaze of a user.

[0027] In some embodiments, embedding the visualizations in the background in the video frames may be performed without tracking gaze of a user.

[0028] In some embodiments, the method may further include highlighting players in the video stream based on ranking the players according to an importance level for the players.

[0029] In some embodiments, ranking the players may include assigning a highest rank to a first one or more players with a ball, a next ball receiver, and any offensive player with open space, thereby highlighting one or more offensive players.

[0030] In some embodiments, ranking the players may include assigning an intermediate rank to one or more players who are defending a player with a ball, thereby highlighting one or more important defensive players.

[0031] In some embodiments, ranking the players may include assigning a lowest rank to all other players.

[0032] In some embodiments, ranking the players may include detecting if a player is in offense or defense using positional tracking data of the players and a ball, including determining that a player is in offense, if the player or one of teammates is a closest player to the ball within a predefined time interval, and otherwise determining that the player is in defense.

[0033] In some embodiments, the method may further include detecting if a player is defending the player with the ball by checking whether the player is within a predefined distance from the player with the ball during a predefined time interval.

[0034] In some embodiments, ranking the players may include preprocessing the video stream, using positional tracking data to identify which players had a ball or were in open spaces in each frame.

[0035] In some embodiments, ranking the players may include using a buffer time to identify a next ball receiver by looking ahead a predetermined time period.

[0036] In some embodiments, ranking the players may include using one or more machine learning models detect a next ball receiver.

[0037] In some embodiments, ranking the players may be based on gaze points of a user to adjust importance levels of the players, as well as controlling whose abilities to show.

[0038] In some embodiments, ranking the players may be based on tracking dwell time for a user gazing at a player in the video stream.

[0039] In some embodiments, the method may further include providing a visual feedback to a user gazing at a player, wherein the visual feedback gradually increases until the player is assigned a higher rank for highlighting.

[0040] In some embodiments, the method may further include, in response to detecting a user gazing away from a player, assigning a lower rank to the player for highlighting.

[0041] In some embodiments, the method may further include centering a display at a user’s gaze point with a filter radius of a predetermined number of pixels.

[0042] In another aspect, an electronic device includes one or more processors, memory, a display, and one or more programs stored in the memory. The programs are configured for execution by the one or more processors and are configured to perform any of the methods described herein.

[0043] In another aspect, a non-transitory computer readable storage medium stores one or more programs configured for execution by a computing device having one or more processors, memory, and a display. The one or more programs are configured to perform any of the methods described herein.

[0044] Thus methods, systems, and interfaces are disclosed that augment sports videos with embedded visualizations.BRIEF DESCRIPTION OF THE DRAWINGS

[0045] For a better understanding of the various described implementations, reference should be made to the Detailed Description below, in conjunction with the following drawings in which like reference numerals refer to corresponding parts throughout the figures.

[0046] Figure 1 is a schematic diagram of an example system for augmenting sports videos with gaze-moderated embedded visualizations, in accordance with one aspect of the present disclosure.

[0047] Figure 2 is a schematic diagram of an example formative study and a review phase where a participant’s gaze is visualized and overlaid on a video, in accordance with one aspect of the present disclosure.

[0048] Figure 3 shows a table for game videos used in the formative study, in accordance with one aspect of the present disclosure.

[0049] Figure 4 shows graph plots for gaze distribution for two games, in accordance with one aspect of the present disclosure.

[0050] Figure 5 shows a table with design requirements or options for assisting casual fans in game watching derived from the formative study, in accordance with one aspect of the present disclosure.

[0051] Figure 6 is a schematic diagram of an example system for embedding visualizations using a computer vision pipeline, according to some embodiments, in accordance with one aspect of the present disclosure.

[0052] Figure 7 shows a table for performance of player detection and post-processing steps, according to some embodiments, in accordance with one aspect of the present disclosure.

[0053] Figure 8 shows a table for time performance of different steps, according to some embodiments, in accordance with one aspect of the present disclosure.

[0054] Figure 9 is a schematic diagram of an example system for gaze-moderated embedded visualizations, in accordance with one aspect of the present disclosure.

[0055] Figure 10 is a schematic diagram of example visualizations of importance levels, according to some embodiments, in accordance with one aspect of the present disclosure.

[0056] Figure 11 shows an example Expected Point Value (EPV) map, according to some embodiments, in accordance with one aspect of the present disclosure.

[0057] Figure 12 shows example embedded visualizations, according to some embodiments, according to some embodiments in accordance with one aspect of the present disclosure.

[0058] Figure 13 shows design alternatives for offense ring and an EPV map, in accordance with one aspect of the present disclosure.

[0059] Figure 14 shows example gaze interactions, according to some embodiments.

[0060] Figure 15 is a flowchart for an example method for embedding visualizations in a video stream of an athletic game, according to some embodiments.DETAILED DESCRIPTION

[0061] Reference will now be made in detail to implementations, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the various described implementations. However, it will be apparent to one of ordinary skill in the art that the various described implementations may be practiced without these specific details. In other instances, well-known methods, procedures, components, circuits, and networks have not been described in detail so as not to unnecessarily obscure aspects of the implementations.

[0062] Many modifications and variations of this disclosure can be made without departing from its spirit and scope, as will be apparent to those skilled in the art. The specific implementations described herein are offered by way of example only, and the disclosure is to be limited only by the terms of the appended claims, along with the full scope of equivalents to which such claims are entitled.

[0063] Visualization has long been used in sports to present data, including box scores, tracking data, and metadata. Sports visualizations may be used for post-game analysis or in-game informing purposes. Sports games usually involve complex in-game decision-making. To better understand, analyze, and appreciate players’ in-game decisions, spectators often look for additional information when watching a sports game. To fulfill individual spectators’ information needs, some embodiments provide interactive game-watching systems. Some embodiments use mobile augmented reality to enhance the experience of live sports events. Some embodiments use a conversational interface to help users request data visualizations while watching basketball games. Some embodiments use simple visualizations (e.g., line charts) to show in-game box scores of basketball games. Some embodiments use interactive embedded visualizations to support in-game analysis of basketball games. Some embodiments allow inspectors to review basketball in-game data through simple, non-interactive embedded visualizations (e.g., text labels, progress bars). Conventional E-Sports provide a personalized game viewing experience by default. Multiplayer Online Battle Arena (MOBA) games, such as Defense of the Ancients2 (Dota2) and League of Legends (LoL), allow spectators to interact with the systems to inspect in-game data (e.g., points over time) of players or teams. However, these systems either display the data in separated panels or require viewers to explicitly interact with the system to request the data, inevitably distracting viewers from the game. In contrast, techniques described herein may use embedded visualizations and gaze interactions to present extra data in game videos, providing an intuitive, seamless, and engaging watching experience.Embedded Visualizations in Sports Videos

[0064] Embedded visualizations may be used for sports data due to their ability to show the data into its physical context (e.g., a basketball court). Conventional systems mainly embedded the data into static court diagrams. Some such systems display density maps on top of court diagrams to show sports events, such as successful shots, computer vision may be used to embed visualizations directly into sports videos instead of just court diagrams. Some embodiments provide a method to automatically extract and visualize data from and in soccer videos. Some embodiments provide augmented sports videos and provide fast prototyping tools to help users create augmented videos for racket-based sports by using direct manipulation and textual comments. These techniques mainly target experts for analytic and authoring purposes.

[0065] Figure 1 is a schematic diagram of an example system 100 for augmenting sports videos with gaze-moderated embedded visualizations, according to some embodiments. The system 100 may augment sports videos 106 (e.g., raw footage plus data, basketball videos)with gaze-moderated embedded visualizations (to obtain an augmented video 108) to facilitate game understanding and engagement of casual fans. The system may embed data visualizations into basketball raw footage 106 using a computer vision pipeline 104, and / or automatically adapt the visualizations (gaze-moderated embedded visualization 102) based on the game context and gaze 112 of users 110.

[0066] In some embodiments, embedded visualizations are used in live game-watching scenarios. Some embodiments use the notion of visualization in motion to depict visualizations that are moving relative to the viewer. Some embodiments use a design framework for embedded visualizations to facilitate in-game analysis when watching basketball games. Beyond simulated scenarios (e.g., moving charts on white backgrounds, 3D virtual sports games), in some embodiments embedded visualizations target real basketball videos, improve casual fans’ game-watching experience.Example Computer Vision Techniques for Embedded Visualizations

[0067] In some embodiments, computer vision techniques are based on deep learning. Computer vision techniques may be used for object detection, object tracking, pose estimation, and / or segmentation. Data may be extracted from videos for sports analytics. For example, the positions of the players and the ball, as well as other tracking data, of a basketball game may be extracted and shared online. Some embodiments use content-aware video analysis for sports, computer vision techniques may be used to embed visualizations into the video scenes, in situated visualization. Embedding visualizations into sports videos may use a computer vision pipeline to complete tasks, such as detecting, tracking, and segmenting the players from the video, estimating their pose, calibrating the camera, and / or reconstructing the 3D scene. Some embodiments use a simplified computer vision pipeline to process racket-based sports videos, in which the players are separated, and the camera is mostly static. Some embodiments embed visualizations into team sports videos, where players overlap heavily and the camera typically moves rapidly. While conventional systems can achieve good embedding results, they require videos collected from multiple cameras to register the visualizations. Techniques described herein use a computer vision solution that can embed visualizations into sports videos based solely on broadcasting videos. Visualizations may be embedded in complex, dynamic scenarios, such as team sports. Some embodiments use a computer vision pipeline that mayinclude open-sourced modular components to process team sports videos for embedding visualizations.Example Applications of Gaze Interactions

[0068] Gaze for interactions may be used due to its efficiency, expressiveness, and applicability in hands-free scenarios. Gaze interactions either explicitly or implicitly leverage the gaze to interact with digital content. Implicit gaze-based systems may use gaze as an implicit input source, usually in combination with other input modalities, to facilitate interactions. Reliable eye trackers may be integrated into desktop and laptop computers, and / or implicit gaze interactions may be used to support a variety of applications, such as content annotation, video editing, and remote collaborations. Some embodiments adapt viewing content based on users’ gaze. Some embodiments use gaze-contingent display that shows a higher resolution on the area the user is focusing on. Some embodiments adjust the playback speed of lecture videos based on the user’s gaze. In some embodiments, a tourist guide directs a user’s gaze to highlighted features in a panorama and adapts the audio introductions accordingly. Some embodiments use a gaze-adaptive system that dynamically adjusts video captions’ placement to optimize the viewing experience. Some embodiments use gaze to adjust video content but focus on augmented sports videos. Some embodiments use gaze-aware embedded visualizations to improve the sports-watching experience.

[0069] To understand the practices, pain points, and solutions of casual fans in watching basketball videos, a formative study was conducted. Figure 2 (a) shows a schematic diagram of an example formative study 200. (a) shows a setup where a participant 202 is gazing at a screen 206 displaying a sports video stream, (b) shows a review phase where the participant’s gaze is visualized and overlaid (204) on the video. Participants were recruited using university mailing lists and forums and participants were screened based on their fandom level, game-watching frequency, and basketball knowledge. In total, eight casual fans were recruited (P1-P8; M=3, F=5; Age: 18 - 35), who only knew “basic rules of basketball” and watched “1 - 10 games per year”. To better identify the pain points specific to casual fans, 8 die-hard fans were recruited (P9 - P16; M=8; Age: 18 - 55); these fans knew “basketball tactics and pros and cons of specific players” and watched “at least 1 game per week”. All participants had normal vision or wore contact lenses or glasses to correct to normal vision. In some embodiments, each session is started by introducing the research motivation and studyprotocol. An experimenter then conducted a semi-structured interview with each participant, focusing on their current practices, pain points, and solutions when watching live basketball games. Next, a format of contextual query was used to ask participants to watch two videos on a 24-inch monitor. Figure 3 shows a table 300 (Table 1) for game videos used in the formative study. These two games were rated as top-30 games of the season and have been watched millions of times. Think-aloud and gaze data were collected during the game-watching process. To collect the gaze data, a commercial software that leverages Apple’s TrueDepth camera to track the participant’s head and gaze was used. The participants were at approximately 60 cm from the screen and were asked to adjust the chair before watching the videos (Figure 2 (a)).

[0070] The system was then calibrated and the participants were allowed to move the head freely after the calibration. A TrueDepth camera-based tracker provided sufficient accuracy for inspecting what video objects participants were looking at while watching the game. For more fine-grained gaze data (e.g., saccades, fixation), more proficient eye-tracker may be used. After watching each game video, participants were asked to rewatch the game with their gaze data overlaid (Figure 2 (b)) and to elaborate on any confusion, data needs, insights, and excitement they had felt when watching the game for the first time. Participants could pause the video in the review phase. Each participant was compensated with a $20 gift card for their time (1 hour).

[0071] Interviews and think-alouds were audio-recorded, transcribed, and analyzed using a reflexive thematic analysis. Three authors coded independently on the transcriptions to form sets of plausible codes and iteratively refined the codes to converge on a single coding schema. Besides, three authors analyzed the gaze data by manually annotating the video objects each participant was looking at while watching the games. Figure 4 shows graph plots 400 for gaze distribution for two games (game 1 and game 2). The gaze distribution in seconds shows that, compared to die-hard fans, casual fans spent more time watching the player with the ball in both games. Due to the small sample size, focus in this description is more on other behavioral observations. The categories of objects (x axis in Figure 4) were generated based on the data and prior knowledge. Participants were classified as looking at an object only when their gaze rested on the object for at least 0.25 seconds (fixation duration). The duration when the gaze was moving to the object was also annotated as looking at the object.Results of Formative Study and Discussions

[0072] The casual fans only watched “important games, such as semifinals or finals.” (Pl) They were neither familiar with basketball nor the NBA. In comparison, the die-hard fans watched basketball games much more frequently. They had a rich knowledge of basketball (e.g., tactics), knowing almost all NBA players and even their strengths and shortcomings. TV was the main way for all the participants to watch live basketball games. Overall, for the casual fans, watching live basketball games was a leisure activity, such as hiking, but it was a more serious hobby for the die-hard fans.Casual Fans’ Confusion in Watching Basketball Games

[0073] In terms of the watching experience, all participants confirmed that they were confused from time to time when watching basketball games and that they would like to seek extra information, other than the data provided by the scoreboard and commentaries. Some confusion is common among both casual and die-hard fans, such as questions like “who got a foul?” and “which team called the timeout?” These questions can usually be resolved by “watching the replay” (Pl 5) or simply by searching Google. However, experiments identified some confusing aspects specific to casual fans that cannot be easily resolved by the current methods and thus lead to a poor watching experience:• Cl : Casual fans are unsure about which players they should focus on. When watching basketball games, the casual fans often could not identify the important players and felt that the players were just “moving objects.” (P4) The casual fans’ inability to identify key players was also reflected in their gaze patterns. In the study, it was found that casual fans spent more time on the player with the ball than the die-hard fans (Figure 4), since they “didn’t notice other players’ [of-ball] movement” (Pl) when watching the game. As a result, the casual fans often missed important of-ball movements and felt that the ball “magically fly to an open player.” (Pl) Moreover, in some casual fans’ gaze, we noticed some rapid zigzag movement between the player with the ball and the other players, revealing their attempts to scan through the players. P4, for example, explained that she was “scan[ning] other players” to predict the ball receiver at the next pass while keeping an eye on the player with the ball, leading to a heavy cognitive load. In contrast, the die-hard fans scanned through the players much more predictively and often could directly identify the next ball receiver.• C2: Casual fans are confused about the in-game decisions of players. The casual fans could hardly understand the in-game decisions of the players, since the situated factors (e.g., players’ abilities) behind these decisions were hard to interpret from the videos. Consequently, the casual fans could not appreciate the game at the same deep level as die-hard fans and had difficulties keeping pace with their experienced friends. This was also revealed in the think-aloud data of the participants. When watching the two videos, the most frequent verbal comments from the casual fans were interjections, e.g., “Oops,” “Wooooow!” Even in the follow-up review session, casual fans could hardly describe their thoughts while watching the games. P3 acknowledged that she sometimes actually “didn’t totally understand” what was going on but just felt excited. By contrast, the die-hard fans could clearly elaborate, comment on, and even suggest players’ tactics when watching the games. Generally speaking, the study suggested that the experience of watching games for the casual fans was closer to “feeling” while the experience for the die-hard fans was closer to “reading.”• C3 : Casual fans have trouble seeking customized data while watching game videos. All the casual fans never searched the internet to seek data to resolve their confusion when watching the games. This was because the games were so fast and overwhelming that they could miss key events when looking up websites. Additionally, the casual fans sometimes could not search for a player’s data because they did not know the player’s name. In contrast, the die-hard fans would search websites (e.g., ESPN) when watching the games, though they also complained about the context switching between the games and the webpages. According to the casual fans, perhaps the best way to seek information about game understanding was to “ask my [experienced] friends.” (P2) Otherwise, they would just “let it [the confusions] go.”Die-hard Fans’ Suggestions for Understanding Basketball Games

[0074] Since casual fans preferred to “ask experienced friends” to seek information, it is important to understand what information die-hard fans suggest for understanding a live basketball game. Several critical insights were suggested by die-hard fans:Distinguishing between offense and defense. Basketball, from a certain perspective, is a turn-based game. A basketball game consists of multiple possessions (i.e., turns), in which the team that has possession of the ball is on offense, and the other team is ondefense. A player can have completely different roles, tactics, and behaviors between offense and defense. Being aware of players’ offense and defense status can help casual fans better understand and follow the game.• Identifying Key Players. While basketball is a team sport, the importance of each player, especially when she / he is on offense, is different. Generally speaking, on the offensive side, the player with the ball and the ball receiver at the next pass are the most important ones. Players with open spaces are also critical to the offensive team as they have a higher chance of making the goal. On the defensive side, all the defenders guarding the player with the ball are important. By identifying these key players, the die-hard fans could watch the game more effectively and predictably. In addition to the aforementioned key players, other players were discussed with the die-hard fans, such as offensive helpers who play screens. Overall, they suggested not helping casual fans identify these players, as their contributions to the possession outcome (e.g., a goal) are not explicit and thus can confuse casual fans.• Understanding In-game Decisions. Knowing players’ offensive and defensive abilities is essential to understanding their in-game decisions. The die-hard fans suggested two metrics to help casual fans understand the players’ abilities. For offensive players, it is possible to present their location-based expected point value, which measures how many points a player is expected to make if they shoot at a specific location. For defensive players, it is possible to present their location-based percentage points difference, which measures how much the field goal percentage of a player changes when being defended by the defensive player. Both metrics can be calculated or directly obtained by using the data from the Official NBA Stats website. The die-hard fans also suggested visualizing the one-on-one relationships between offensive and defensive players, which can reveal interactions between the players and their tactics (e.g., defensive switching).

[0075] In summary, the casual fans were often confused about the key players and their in-game decisions, but rarely sought data to resolve their confusion because the searching process is slow and distracting. To help casual fans better understand the game, the diehard fans suggested a few critical insights, including distinguishing between offense and defense, identifying key players, and understanding players’ in-game decisions. Figure 5 shows a table 500 (Table 2) with three design requirements or options for assisting casual fans in game watching derived from the formative study. Described below is a computer vision pipeline toaddress these requirements or options, followed by a set of gaze-moderated embedded visualizations.Example Computer Vision Pipeline for Embedding Visualizations

[0076] To embed visualizations into a sports video, some embodiments may recognize the players (e.g., bounding box, identity, and key points) and segment them from the background. To this end, some embodiments provide a computer vision pipeline to pre-process team sports videos.

[0077] Figure 6 is a schematic diagram of an example system 600 for embedding visualizations using a computer vision pipeline, according to some embodiments. The system 600 is similar to the system 100 described above, with details of the computer vision processing pipeline 104. Game data 608 may be input to the gaze-moderated embedded visualization 102, to produce a visualization 602 based on gaze data 112, and output of the computer vision processing pipeline 104. The computer vision pipeline may take a raw video 106 as input and output a bounding box, identity and key points of each player, and / or separate the image frame into foreground (e.g., humans) and background (all others). The bounding boxes, identities and key points may be used to create visualizations, which are then composited with the foreground and background to form the augmented video.Recognizing the Players

[0078] To embed visualizations for a player, some embodiments may first recognize the player in the video. For example, to display a label with the name of a player, the system may detect the video object that corresponds to the player (bounding box and identity) and the player’s key body joints (key points) for placing the label. Given a raw video frame, some embodiments may obtain this information for each player via three steps described below.

[0079] Step 1. Player Detection (component (a) of 104, Figure 6). To obtain the players’ bounding boxes and identities in a video frame, some embodiments may use an object detection model to locate and classify each player into different categories. Different from common object detection tasks, some embodiments may use the players’ identities as their categories. Some embodiments may fine-tune a COCO pretrained YoLoX model on a sports dataset (e.g., NBA player dataset). The output of the model may be a set of bounding boxes associated with their identities and confidence scores (i.e., scored). In some embodiments,only those bounding boxes with scorec greater than a threshold Thigh may be considered successful detections.

[0080] Step 2. Post-Processing (component (b) of 104, Figure 6). One limitation of the object detector is that it only utilizes the players’ visual appearance information to determine their confidence score. Consequently, the detector can assign low confidence scores to players whose visual qualities are low (e.g., when they are occluded by others) and filter them out. Accordingly, some embodiments may use object trackers to exploit the players’ motion information to complement the detector. An object tracker may store the history of an object’s bounding boxes in the previous frames and may predict the object’s bounding box in the next frame by using a motion predictor, such as a Kalman filter. Some embodiments may use object trackers as follows: (1) For a frame Ft, some embodiments may divide all the detected bounding boxes into three clusters based on their scorec : high-quality boxes (scorec > Thigh), low- quality boxes Tiow < scorec < Thigh , and rejected boxes (scorec < Tiow). (2) For each high- quality box, some embodiments may match it with the trackers in the previous frame Ft-i by calculating the Intersection over Union (loU) between the box and the predicted boxes of the trackers. A tracker may be considered as matched with the high quality box if it maximizes the loU. If matching is successful, some embodiments may assign the matched tracker to the box; otherwise, some embodiments may initialize a new object tracker for the box. (3) For each low- quality box, some embodiments may match it with the remaining trackers (i.e., those that have not been matched with any high-quality boxes). If matching is successful, some embodiments may assign the matched tracker to the box. (4) Finally, some embodiments may output all the boxes with matched trackers. In some embodiments, the computer vision pipeline may take a raw video as the input, outputs the bounding box, identity, and key points of each player, and / or separates the image frame into the foreground (humans) and background (all others). The bounding boxes, identities, and key points may be used to create visualizations, which are then composited with the foreground and background to form the augmented video. Intuitively, this method uses the motion information of the players to select some low-quality bounding boxes to complement the output of the detector.

[0081] Step 3. Pose Estimation (component (c) of 104, Figure 6). Some embodiments may use a pose estimation model to obtain the players’ key points, such as head, hands, hip, and feet. Some embodiments may use the bounding boxes produced in Step 2 described above to extract the players from the video frame and then feed those boxes to a pose estimator (e.g., ViTPose) to estimate the key points.Separating Foreground and Background

[0082] Embedded visualizations for sports, such as empty areas, are often placed on the ground, beneath the players’ feet. To achieve this, some embodiments may separate the video frame into the foreground (e.g., the objects on the ground) and background (e.g., the ground), draw the visualizations onto the background, and / or finally overlay the foreground on the background to form an augmented video frame (component (d) of 104, Figure 6). Ideally, all the objects should be segmented from the background. Some embodiments may only segment humans from the video as the foreground and leave the remaining pixels as the background, as humans are the major objects on the ground in a basketball video. Some embodiments use a ViT-Adapter trained on COCO 164K to perform binary semantic segmentation to segment the humans.Computational Evaluation

[0083] To evaluate the performance of our pipeline, several experiments were conducted. The experiments focused on three main questions: 1) Can the object detector detect the players?; 2) Can the post-processing step improve the detections?; 3) How much time does each step take? To answer these three questions, the bounding box and identity of each player in each frame of the two game videos G1 and G2 described above were annotated. Each video was split into clips and allocated 70% for training and 30% for testing. To accelerate the training process, every tenth frame from the training clips was sampled and used for training. This is because the consecutive frames often contain redundant information. Despite this, the testing was conducted on all frames in the testing clips. The detector was trained and evaluated on G1 and G2 separately, using their default hyperparameters whenever possible.

[0084] The accuracy of the pose estimation and semantic segmentation steps were not evaluated because off-the-shelf models were used for their standard tasks without any finetuning in these two steps. Yet, their performance for basketball videos can be qualitatively evaluated by inspecting the augmented videos provided in the supplemental material.

[0085] Figure 7 shows a table 700 (Table 3) for performance of player detection and post-processing steps, according to some embodiments. Due to the lack of benchmarks, YoloX’s performances on COCO as a reference are provided. However, it does not serve as a comparative baseline. The table shows average precision for the two steps, for the fine-tunedobject detector on the testing clips of G1 and G2. To access the detector, the convention to calculate the Average Precision (AP) metrics over different loUs was followed. The higher the AP, the better it is. AP50.95 is the average AP over different loU, from 0.5 to 0.95 with step 0.05. AP50 and AP75 are the APs calculated at loU 0.5 and 0.75, respectively. The larger the loU, the stricter the metric will be. Overall, the results reveal that the fine-tuned object detector can perform well in detecting players. Furthermore, all the APs increase after applying the postprocessing step, which shows that the post-processing step is useful and can complement the detector to improve its results.

[0086] Figure 8 shows a table 800 (Table 4) for time performance of different steps, according to some embodiments. In terms of time performance, Table. 4 shows the average time in milliseconds (ms) each step takes to process a video frame. The pipeline was tested on a machine with a Nvidia Tesla VI 00 graphic card and only counted the inference time of the models by excluding the model and dataset loading time. Overall, the player detection and postprocessing steps use 34 ms for one frame, almost achieving 30 frames per second (FPS). Other steps, especially the semantic segmentation step, need longer to process one frame. These results show that the semantic segmentation model is the bottleneck for extending the pipeline to support real-time scenarios.

[0087] To improve accuracy, some embodiments may use better models or add extra components to the post-processing step to improve the detections. For example, some embodiments further interpolate and smooth the bounding boxes for the user study. To improve the efficiency, some embodiments may use faster models, more powerful graphic cards, and / or remove the semantic segmentation step if visualizations on the ground are not needed. The computer vision pipeline can be applied to other basketball videos and even other team sports videos. For example, there are about 450 players in the NBA. To generalize the pipeline to other NBA game videos, some embodiments may use a player dataset for 450 players to finetune the detector. Note that it is not necessary to develop a player dataset for each video. Our experiments showed that the detector could detect players on unseen testing clips even if it were trained only on the training clips. If the player dataset is large enough, the detector finetuned on it can be applied to any NBA game video. This is not impossible as modem deep learning-based image classifiers can achieve superhuman performance on tasks with more than 1000 classes and many priors can be used to optimize the model results, e.g., there are no more than 24 players in a game.Gaze-Moderated Embedded Visualizations

[0088] Based on the identified design requirements from the formative study, a set of gaze interactions were designed that can naturally guide and respond to the user’s attention through gaze tracking without explicit user input. In some embodiments, gaze-moderated embedded visualizations (i) guide audiences’ attention, (ii) reveal players’ offensive and defensive abilities and (iii) update the embedded visualizations based on gaze. Figure 9 is a schematic diagram of an example system 900 for gaze-moderated embedded visualizations, according to some embodiments. Figure 9 is similar to Figures 1 and 6 described above, so only components that are new in this Figure are described herein. In some embodiments, the system may take positional tracking data 902 and historical statistics 904 as input to calculate a players’ importance (component (a) of 102, Figure 9) and offensive and defensive abilities (component (b) of 102, Figure 9). Only the important players and their offensive and defensive abilities may be highlighted and visualized in the video 602. The user can use gaze points 112 to adjust the players’ importance levels, as well as controlling whose abilities to show, via gaze focus and gaze filter (component (c) of 102, Figure 9). Each of these components are described in detail below.Guiding Audiences’ Attention

[0089] To help casual fans identify the important players, some embodiments may rank the players’ importance levels (e.g., according to die-hard fans’ suggestions) and / or may then highlight the players accordingly.Ranking Players’ Importance Levels

[0090] Based on the formative study, some embodiments adopt an offensive-first method to rank the players’ importance into three levels:• Lv3 - Key offensive players: The player with the ball, the next ball receiver, and the players with open spaces may be considered as the most important offensive players. When preprocessing the game videos, some embodiments may use positional tracking data to identify which players had the ball or were with open spaces in each frame. Meanwhile, some embodiments may look ahead a time period (e.g., 1.8 seconds, selected empirically) to find the next ball receiver. To extend the system to livestreamscenarios, some embodiments may use machine learning models and / or the buffer time in video streaming to detect the next ball receiver.• Lv2 - Key defensive players: The players who are defending the player with the ball may be considered as the important defenders. Some embodiments may detect important defenders by checking which defenders were closest to the player with the ball within a time interval.• Lvl - Other players: All other players who do not belong to Lv3 and Lv2 may fall into this level.

[0091] To detect if a player is in offense or defense, some embodiments may use the positions of the players and the ball. If a player or one of her / his teammates is the closest player to the ball within a predefined time interval (e.g., 0.5 seconds), she / he may be considered to be in offense; otherwise, in defense. Some embodiments may ignore some important players, such as those who play screens or specific tactics, since casual fans usually cannot understand why these players are important.Visualizing Importance Levels

[0092] Figure 10 is a schematic diagram of example visualizations 1000 of importance levels, according to some embodiments. Example visualization of various importance levels: Lv3 (key offensive players, highlighted by a spotlight), Lv2.5 (players of interest to the user, triggered by Gaze Focus, highlighted by a glowing effect (described below)), Lv2 (key defensive players, highlighted by extra brightness) and Lvl (other players, no highlighting), are shown. Some embodiments may use multiple highlight effects to guide user attention to players at different importance levels. In some embodiments, the name of players with importance levels greater than Lv2 may be displayed. In some embodiments, the name of “star” players may be colored (e.g., in gold with an icon showing their roles, e.g., for good shooter and for good defender). Furthermore, Lv3 spotlight may encode the different offense roles with color, green for players with open space and white for other key offensive players.Revealing Players’ Abilities

[0093] To help casual fans understand the players’ abilities, some embodiments may compute and visualize two location-based metrics of the players whose attention level is higher than Lvl (component (b) in 102, Figure 9).Measuring Players’ Offensive and Defensive Abilities

[0094] Some embodiments use two metrics to indicate the players’ offensive and defensive abilities:• Offense - Expected Point Value (EPV) measures how many points a player is expected to make if he / she shoots from the current position. In basketball, it is a value between 0 and 3. Fundamentally, the goal of offensive tactics in basketball games is to maximize the EPV of the shooter. Thus, visualizing the EPV can help casual fans better understand and evaluate the in-game decisions of offensive players (e.g., pass or shoot). Some embodiments may obtain the EPV for each player based on their historical shot records. Specifically, some embodiments may create a hexbin shot chart for each player based on their historical shot records, in which the bins are grouped based on the shooting regions (e.g., defined by Official NBA Stats). Some embodiments may then calculate the EPV per region by multiplying the player’s field goal percentage and points they can make in the region. The results may be cached as an EPV map for efficient access in each frame. Figure 11 shows an example EPV map 1100, according to some embodiments. Figure 11 is an EPV map of Stephen Curry based on his shooting records in the 2015-16 season. A darker color indicates a higher EPV.• Defense - Percentage Points Difference (DIFF%) is a measure of a defender’ s ability to affect a shooter’s shot percentage. Good defenders will have a negative DIFF% since they hold their opponent to a lower percentage than normal. For example, Stephen Curry’s DIFF% is -3.6%, which means on average, a shooter’s shot percentage will decrease by 3.6% when being guarded by Curry. Some embodiments may acquire DIFF% by regions for each player directly from NBA Stats. Besides DIFF%, the distance between a defender and the offensive player with the ball (DIST) is critical to the defensive performance. Some embodiments may calculate DIST based on the positional tracking data.Visualizing Players’ Offensive and Defensive Abilities

[0095] Figure 12 shows example embedded visualizations 1200, according to some embodiments. As further described below, one or more embedded visualizations for in-game data may be shown: a) Offense Ring shows the offensive performance of an offensive player. The darker, larger, the better, b) Defense Shield shows the defensive performance of the defender. The thicker, longer, the better, c) One-on-one Line shows the one-on-one relationship between the offensive player with the ball and the defenders.

[0096] Some embodiments use one or more embedded visualizations to present the offensive (EPV) metric, defensive (DIFF% and DIST) metric, and / or the one-on-one relationship between the defenders and the offensive player with the ball:• Offense Ring ((a) in Figure 12) presents the player’ s location based EPV, where a larger ring with darker color indicates better offensive ability. The inner and outer rings represent the minimum and maximum of possible EPV (i.e., 0 and 3). We used both the size and color of the middle ring to encode the player’s EPV at the current position for easier interpretation.• Defense Shield ((b) in Figure 12) represents the defender’s location based DIFF% and DIST in an arc shape, where a thicker and longer arc indicates better defensive ability. The thickness of the “shield” encodes the inverse DIFF% (a negative value) to make the visualization intuitive. The arc length of the “shield” encodes the subtraction of DIST from maximum guarding distance, since a larger DIST indicates lower pressure from the defender to the player with the ball. We displayed an outer border of the “shield” to show the maximal guarding distance (empirically selected as 12 feet) for comparison.• One-on-one Line ((c) in Figure 12) visualize the one-on-one relationship between the key defenders and the offensive player with the ball. The player with the ball can be defended by multiple defenders. These visualizations may be updated dynamically in the game based on the players’ positions. Some embodiments may also darken the background image to provide enough contrast for reading the visualizations.Design Process and Alternatives

[0097] Figure 13 shows design alternatives 1300, according to some embodiments. Left (a)-(c) show three design alternatives for offense ring, a) Displaying the data on top of theplayer can occlude other players. Moving the visualization higher can make it hard to connect to the target player. Displaying the data aside of the players (e.g., the shot meter in NBA 2K) can also occlude other players. Right (d) shows an experimental EPV map of Steven Curry encoding his shooting frequency and EPV by using the size and divergent color scale. Different from Figure 11, the bins in this EPV map are not grouped by regions.

[0098] A ring placed on a ground may help tightly connect with a target player and avoid occluding other objects. Figure 13 (a)-(c) show some alternative designs. When designing the visual encoding of offense ring, some embodiments may use the size of the ring to encode a player’ s shooting frequency and a divergent color scale to encode the player’ s EPV, with the league average EPV as the midpoint. Figure 13 (d) shows an EPV map created based on this encoding schema. Some embodiments may use a sequential color scale instead of a conventional divergent one. Different designs may be used for other purposes or fans, e.g., for analytic purposes or die-hard fans.Gaze-based Interactions

[0099] To help casual fans seamlessly and efficiently access data of players they are interested in while watching the game, some embodiments may use gaze as an input signal and / or gaze interactions (component (c) in 102, Figure 9). The fans can still be guided by the visualizations to identify important offensive players.

[0100] Figure 14 shows example gaze interactions 1400, according to some embodiments. The example shows two gaze interactions to adjust the embedded visualizations: (a) gaze focus lifts the importance level and shows in-game data of players that are of interest to the user, and (b) gaze filter drops the importance level of video obj ects who are not the user’ s focus (e.g., open players and audiences out of focus).Gaze Focus - Fetching Data of Players of Interest to the User

[0101] Gaze focus may allow the user to express their interests in players through gaze dwelling, which lifts the importance level of the players. Gaze Focus comprises the following three considerations:Trigger: According to the formative study described above, the users’ gaze can move rapidly between, and across, players. To avoid showing data of players glanced over bythe user accidentally, some embodiments may define and use a “dwell time” for the interaction. The user needs to dwell her / his gaze on a player for the dwell time (e.g., 0.25 seconds) to trigger the interaction.• Visual feedback: To help the user realize that she / he is gazing at a player and triggering the interaction, some embodiments may use a highlight effect (e.g., Figure 10) in which the glow of the player may gradually increase when the user is gazing at the player, until the interaction is triggered. Some embodiments may provide continuous visual feedback for the user while conforming with the visual design of importance levels.• Outcome: Once the user triggers the interaction, the system may lift the targeted player to Lv2.5 if she / he is currently at a lower level ((a), Figure 14). As a result, the system may also visualize the name and offensive or defensive data of the player. Lastly, when the user moves her / his gaze away from the player, the player will stay in Lv2.5 for a predetermined time period (e.g., 1.8 seconds, selected empirically) before reverting to their original player importance level. Such a lasting duration may help cope with the users’ rapid saccade in game watching.Gaze Filter - De-emphasizing Video Objects Out of the Sight

[0102] To prevent users from being overwhelmed by too many Lv3 players, some embodiments may use a gaze filter to turn of the green spotlights of open players beyond a predefined filter radius. Gaze filter may incorporate three considerations:• Trigger: Generally speaking, the system should always avoid overwhelming the user. Thus, the gaze filter may be consistently triggered and updated when the user moves his / her gaze. Some embodiments may center (e.g., a display) at the user’s gaze point with a filter radius of a predetermined number of pixels (e.g., 650 pixels, empirically selected).• Visual feedback: To indicate the user that the interaction is being triggered, some embodiments may use a radial blurring effect that darkens the audience outside the filter radius and updates dynamically. Some embodiments may smooth the movement of the radial blurring effect to prevent it from abruptly changing location due to the user’s saccade. The blurring effect may not be applied to players and the court to ensure their readability. This visual feedback can notify the user about the existence of theinteraction while also creating a theater mode that helps the user to focus on and engage with the game.• Outcome: The green spotlights, which may be used to highlight offensive players with open spaces, outside the filter radius may be turned off ((b), Figure 14). An ease-in-out effect may be applied to the change.

[0103] Visual attention matters when designing embedded visualizations. Due to a phenomenon known as Inattentional Blindness, viewers can fail to perceive visually salient objects or activities. Accordingly, some embodiments may properly direct the user’s attention when designing embedded visualizations to effectively convey information and avoid overwhelming the user. For example, the offense ring, instead of players, may be highlighted, to help audiences efficiently identify the player with the highest EPV. Highlighting visualizations that are linked to immediate actions can increase the information salience, e.g., highlighting the offense ring when a player is about to shoot.

[0104] Some embodiments synchronize seeing and hearing when designing embedded visualizations. In some embodiments, the embedded visualizations may “overlay” commentaries. In FULL mode, participants may use gaze interactions to search for players mentioned in the commentaries. Some embodiments use the commentaries to create embedded visualizations.

[0105] Gaze interactions may shift the game from explanatory to exploratory. In FULL mode, the participants may spend more time using their gaze to highlight players, while in AUG mode, participants may follow the players highlighted by the system. This difference suggests that AUG mode is more explanatory while the addition of gaze interactions can shift it towards exploratory. This is not surprising, as the gaze interactions allows the audiences to actively explore the game more. When designing gaze interactions for game viewing systems or, broadly speaking, any situated visualization systems that involve visual guidance, some embodiments consider the ultimate goal of the systems and strike a balance between explanatory and exploratory.

[0106] Gaze interactions can enable active learning in game watching The gaze interactions can also help audiences learn basketball knowledge progressively. For example, gaze filter may only highlight open players with green spotlights when the players are near the user’s gaze. A user may use this feature to verify a hypotheses of team tactics by moving gazeto some areas and seeing if the system “showed green [highlighting]” there. In some embodiments, gaze focus may help better recognize players by showing the name of a player to confirm that the user was looking at the right person. Such a hypothesis-testing process can make participants feel more confident in interpreting the game.

[0107] Some embodiments provide customization options (e.g., for the visualizations and gaze interactions) through a “Preference” panel and allowing the visualizations to adapt to the pace of the game (e.g., e.g., showing more details in slow-paced and less in fast-paced situations). Some embodiments may generate play-by-play replays with embedded visualizations to explain the game in detail.

[0108] The techniques described herein can help improve game understanding and engagement for die-hard fans. Unlike casual fans, die-hard fans could gain a deeper understanding of the game with the embedded visualizations. For example, they could further recognize the offensive tactics of the team from the highlighted open players. For some diehard fans, the usefulness of the techniques described herein extends beyond understanding the games. The techniques can help improve in-game decisions and tactics. Besides game understanding, the techniques described herein can help enhance user engagement in game watching, especially the gaze interactions, which can provide a feeling of “participating in the game.”

[0109] In some embodiments, depending on user preference, one or the following actions may be performed: highlighting fewer open players, highlighting more, showing names to help learn about unfamiliar players, showing information on only “star” players, and / or more gaze interactions.

[0110] Embedded visualizations may not be displayed throughout the entire game. For example, visualizations maty be displayed when players are executing a coach’s strategy, such as in the first two quarters. However, when the game is decided by the “star” players’ in-game states and improvisations (during crunch time), the embedded visualizations may not be shown. Some embodiments allow users to decide when to display the embedded visualizations.

[0111] Gaze interactions may provide more adaptive data for the die-hard fans. For example, the retrieved data for the Golden State Warriors could focus on teamwork, while the data for the Cleveland Cavaliers could emphasize the performance of their “star” players. When gazing two “star” players facing off against each other, such as LeBron James vs. Steph Curry,the system may display their historical one-on-one records. Besides, the data could adapt to specific game events, such as dunking, or the intensity of the game.Example Methods for Embedding Visualizations in A Video Stream of An Athletic Game

[0112] Figure 15 is a flowchart for an example method 1500 for embedding visualizations in a video stream of an athletic game, according to some embodiments. The method may be performed by a system (e.g., one or more modules of the system 100, the system 600, and the system 900). The system may include a processor (e.g., a video processor or a separate processor, such as a CPU, a GPU, an ASIC, an FPGA), and a frame buffer (e.g., a frame buffer in the video processor or coupled to the video processor) for storing video stream comprising a temporal sequence of video frames. The system may also include an entity identification module (e.g., components (a)-(c), 104, Figure 6), executable by the processor, for identifying entities (e.g., players, objects, such as a ball, a court), a segmentation module (e.g., the component (d) of 104, Figure 6) for segmenting foreground and background, and / or a mapping and embedding module (e.g., the computer vision pipeline 104 and / or the gazemoderated embedded visualization 102), executable by the processor, for mapping the detected and segmented visualizable entities to visualizations, and embedding the visualizations in the background in the video frames.

[0113] The method may include identifying (1502) visualizable entities in video frames of the video stream. In some embodiments, the visualizable entities may include physical entities that appear in the video frames of the video stream. In some embodiments, the visualizable entities may be players in a basketball game, and the background may be the ground. Examples of identification are described above in reference to Figure 6 (e.g., the computer vision pipeline 104), according to some embodiments.

[0114] In some embodiments, identifying the visualizable entities may include detecting video objects that correspond to each visualizable entity and the visualizable entities’ key body points. In some embodiments, the video objects may include one or more bounding boxes and an identity for each visualizable entity, and the key body points may include key points for each visualizable entity. In some embodiments, detecting the video objects may include obtaining the one or more bounding boxes and the identity in the video frames using an object detection model to locate and classify each visualizable entity into different identities. In some embodiments, output of the object detection model may be a set of bounding boxesassociated with their identities and confidence scores. In some embodiments, detecting the video objects may include detecting bounding boxes with a score (e.g., 0.75) greater than a predetermined threshold (e.g., 0.4).

[0115] In some embodiments, identifying the visualizable entities may further include using obj ect trackers to take into account a physical entity ’ s motion information to complement detection of the video objects. In some embodiments, each object tracker may store history of an object’s bounding boxes in previous frames and predicts the object’s bounding box in a next frame by using a motion predictor, such as a Kalman filter.

[0116] In some embodiments, using the object trackers may include: for a frame Ft, dividing detected bounding boxes into three clusters based on their score: high-quality boxes with a score greater than a first predetermined threshold, low-quality boxes with a score between the first predetermined threshold, and rejected boxes with a score lower than a second predetermined threshold. The method may also include, for each high quality box, matching it with trackers in a previous frame Ft-1 by calculating an Intersection over Union (loU) between the box and the predicted boxes of the trackers. A tracker is considered matches with the high- quality box if it maximizes the loU, wherein if matching is successful, the matched tracker is assigned to the box, and otherwise a new object tracker is initialized for the box. The method may also include, for each low-quality box, matching it with remaining trackers that have not been matched with any high-quality boxes, including if matching is successful, assigning the matched tracker to the box, and outputting all boxes with matched trackers.

[0117] In some embodiments, using the object trackers may include dividing bounding boxes into a high-quality group, a low-quality group, and a rejected group, based on confidence score for each bounding box and a predetermined threshold (e.g., 0.4).

[0118] In some embodiments, using the object trackers includes matching existing object trackers with bounding boxes in the high-quality group, or initializing new object trackers for bounding boxes in the high-quality group if matching is not successful.

[0119] In some embodiments, using the object trackers includes matching existing object trackers with bounding boxes in the low-quality group.

[0120] In some embodiments, using the object trackers includes outputting bounding boxes with matched trackers.

[0121] In some embodiments, identifying the visualizable entities may further include using a pose estimation model to obtain each visualizable entities’ key points selected from thegroup consisting of: head, hands, hip and feet. In some embodiments, detecting the video objects may include fine-tuning an object detection model on a dataset of the visualizable entities.

[0122] The method may also include segmenting (1504) the visualizable entities from background in the video frames using one or more image segmentation models. Examples of segmentation are described above in reference to component (d) (semantic segmentation) of the computer vision processing pipeline 104, Figure 6. Some embodiments use instant segmentation.

[0123] The method may also include mapping (1506) the visualizable entities to predetermined visualizations. Examples of mapping visualizable entities to predetermined visualizations are described above in reference to gaze-moderated embedded visualizations and design process and alternatives. For example, important players may be ranked and their offensive and / or defensive abilities may be colored, highlighted and / or visualized in the video frames. Additionally, and / or alternatively, different categories of entities may be mapped to different visualizations. To achieve the visual mappings, some embodiments may use a dictionary based on a design space of augmented sports videos and an emoji searching engine. For example, players (object) are mapped to spotlight highlight effects; ball angles (data) are mapped to embedded visualizations in the court; “brilliant” (emotional cue) is mapped to a celebration emoji, and so on.

[0124] The method may also include embedding (1508) the visualizations in the background in the video frames. Examples for embedding techniques are described above in reference to computer vision techniques for embedded visualizations. To render visualizations in the videos, some embodiments may determine the camera parameters and the player identities (e.g., who is far from or near the camera). In some embodiments, the camera parameters may be obtained using camera calibration techniques on the court lines detected from the video; some embodiments may treat them and the player identities as known meta information. Additionally, and / or alternatively, to embed the visualizations into the video, some embodiments determine when a visualization should appear and disappear in the video. Some embodiments may initialize the time schedules of the visualizations (e.g., based on a commentary text).

[0125] In some embodiments, embedding the visualizations on the background in the video frames may be based on tracking gaze of a user. Examples for gaze-moderated embeddedvisualizations are described above in reference to Figures 9 and 14, according to some embodiments. In some embodiments, embedding the visualizations on the background in the video frames may be performed without tracking gaze of a user.

[0126] In some embodiments, embedding the visualizations in the background in the video frames may be performed without tracking gaze of a user.

[0127] In some embodiments, the method may further include highlighting players in the video stream based on ranking the players according to an importance level for the players. Examples for ranking are described above in the section titled “Ranking Players’ Importance Levels.” Examples for visualizing importance levels are described above in the sections titled “Visualizing Importance Levels,” and in reference to Figures 10, 12 and 13, according to some embodiments.

[0128] In some embodiments, ranking the players may include: assigning a highest rank to a first one or more players with a ball, a next ball receiver, and any offensive player with open space, thereby highlighting one or more offensive players. In some embodiments, ranking the players may include assigning an intermediate rank to one or more players who are defending a player with a ball, thereby highlighting one or more important defensive players. In some embodiments, ranking the players may include assigning a lowest rank to all other players.

[0129] In some embodiments, ranking the players may include detecting if a player is in offense or defense using positional tracking data of the players and a ball, including determining that a player is in offense, if the player or one of the player’ s teammates is a closest player to the ball within a predefined time interval, and otherwise determining that the player is in defense. In some embodiments, the method may further include detecting if a player is defending the player with the ball by checking whether the player is within a predefined distance from the player with the ball during a predefined time interval.

[0130] In some embodiments, ranking the players may include preprocessing the video stream, using positional tracking data to identify which players had the ball or were with open spaces in each frame.

[0131] In some embodiments, ranking the players may include using a buffer time to identify a next ball receiver by looking ahead a predetermined time period.

[0132] In some embodiments, ranking the players may include using one or more machine learning models to detect a next ball receiver.

[0133] In some embodiments, the method may further include calculating and displaying an expected point value for offensive players; and calculating and displaying percentage points difference and distance between a defensive player and an offensive player with a ball, for defensive players.

[0134] In some embodiments, the method may further include displaying and dynamically updating based on players’ positions: (i) an offense ring that presents an offensive player’s location-based expected point value, (ii) a defense shield that represents a defensive player’s location-based percentage points difference and distance between the defensive player and an offensive player with a ball, and (iii) an one-on-one line that visualizes one-on-one relationship between key defenders and an offensive player with a ball, wherein a player with the ball can be defended by multiple defenders.

[0135] In some embodiments, ranking the players may be based on gaze points of a user to adjust the players’ importance levels, as well as controlling whose abilities to show.

[0136] In some embodiments, ranking the players may be based on tracking dwell time for a user gazing on a player in the video stream.

[0137] In some embodiments, the method may further include providing a visual feedback to a user gazing at a player, wherein the visual feedback gradually increases until the player is assigned a higher rank for highlighting.

[0138] In some embodiments, the method may further include, in response to detecting a user gazing away from a player, assigning a lower rank to the player for highlighting.

[0139] In some embodiments, the method may further include centering a display at a user’s gaze point with a filter radius of a predetermined number of pixels.

[0140] In some embodiments, the method may further include updating centering of a display when a user moves gaze.

[0141] In some embodiments, the method may further include using a blurring effect to darken outside a filter radius of a display.

[0142] In some embodiments, the method may further include calculating and visualizing players’ offensive and defensive capabilities using embedded visualizations.

[0143] In some embodiments, calculating the players’ offensive capabilities may include calculating an expected point value for offensive players.

[0144] In some embodiments, calculating the players’ defensive capabilities may include calculating percentage points difference and distance between a defensive player and an offensive player with a ball, for defensive players.

[0145] In some embodiments, visualizing the players’ offensive capabilities may include displaying and dynamically updating an offense ring that presents an offensive player’s location-based expected point value based on players’ positions.

[0146] In some embodiments, visualizing the players’ defensive capabilities may include displaying and dynamically updating a defense shield that represents a defensive player’s location-based percentage points difference and distance between the defensive player and an offensive player with a ball, based on players’ positions.

[0147] In some embodiments, the method may further include displaying and dynamically updating based on players’ positions an one-on-one line that visualizes one-on- one relationship between key defenders and an offensive player with a ball, wherein a player with the ball can be defended by multiple defenders.

[0148] In some embodiments, the method may further include concurrently performing identifying the visualizable entities in the video frames of the video stream, segmenting the visualizable entities from the background in the video frames using semantic segmentation, mapping the visualizable entities to predetermined visualizations, and / or embedding the visualizations on the background in the video frames.

[0149] In some embodiments, the method may further include obtaining camera parameters for a camera used to generate the video stream or estimating the camera parameters using one or more camera calibration techniques, and displaying visualizations that are static relative to a ground using the camera parameters.

[0150] It will be understood that, although the terms first, second, etc., are, in some instances, used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first widget could be termed a second widget, and, similarly, a second widget could be termed a first widget, without departing from the scope of the various described implementations. The first widget and the second widget are both widgets, but they are not the same condition unless explicitly stated as such.

[0151] The terminology used in the description of the various described implementations herein is for the purpose of describing particular implementations only and isnot intended to be limiting. As used in the description of the various described implementations and the appended claims, the singular forms “a,” “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term “and / or” as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. It will be further understood that the terms “includes,” “including,” “comprises,” and / or “comprising,” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0152] The foregoing description, for purpose of explanation, has been described with reference to specific implementations. However, the illustrative discussions above are not intended to be exhaustive or to limit the scope of the claims to the precise forms disclosed. Many modifications and variations are possible in view of the above teachings. The implementations were chosen to best explain the principles underlying the claims and their practical applications, to thereby enable others skilled in the art to best use the implementations with various modifications as are suited to the particular uses contemplated.

[0153] Some embodiments or implementations are described with respect to the following clauses:Clause Al. A method of embedding visualizations in a video stream of an athletic game, the method comprising: identifying visualizable entities in video frames of the video stream; segmenting the visualizable entities from background in the video frames using one or more image segmentation models; mapping the visualizable entities to predetermined visualizations; and embedding the visualizations on the background in the video frames.Clause A2. The method of clause Al, wherein identifying the visualizable entities comprises: detecting video objects that correspond to each visualizable entity and key body points of the visualizable entities.Clause A3. The method of clause A2, wherein the video objects comprise one or more bounding boxes and an identity for each visualizable entity, and wherein the key body points comprise key points for each visualizable entity.Clause A4. The method of clause A3, wherein detecting the video objects comprises: obtaining the one or more bounding boxes and the identity in the video frames using an object detection model to locate and classify each visualizable entity into different identities.Clause A5. The method of clause A4, wherein the object detection model outputs a set of bounding boxes associated with their identities and confidence scores.Clause A6. The method of clause A5, wherein detecting the video objects comprises: detecting bounding boxes with a confidence score greater than a predetermined threshold.Clause A7. The method of clause A2, wherein identifying the visualizable entities further comprises: using obj ect trackers to take into account a physical entity ’ s motion information to complement detection of the video objects.Clause A8. The method of clause A7, wherein each object tracker stores history of an object’s bounding boxes in previous frames and predicts the object’s bounding box in a next frame by using a motion predictor.Clause A9. The method of clause A7, wherein using the object trackers comprises: dividing bounding boxes into a high-quality group, a low-quality group, and a rejected group, based on confidence score for each bounding box and a predetermined threshold.Clause A10. The method of clause A10, wherein using the object trackers comprises: matching existing object trackers with bounding boxes in the high-quality group, or initializing new object trackers for bounding boxes in the high-quality group if matching is not successful.Clause Al 1. The method of clause A9, wherein using the object trackers comprises: matching existing object trackers with bounding boxes in the low-quality group.Clause A12. The method of clause A7, wherein using the object trackers comprises: outputting bounding boxes with matched trackers.Clause Al 3. The method of clause A2, wherein identifying the visualizable entities further comprises:using a pose estimation model to obtain each visualizable entities’ key points selected from the group consisting of: head, hands, hip, and feet.Clause A14. The method of clause A2, wherein detecting the video objects comprises: fine-tuning an object detection model on a dataset of the visualizable entities.Clause A15. The method of any of clauses A1-A14, further comprising: concurrently performing identifying the visualizable entities in the video frames of the video stream, segmenting the visualizable entities from the background in the video frames using semantic segmentation, mapping the visualizable entities to predetermined visualizations, and embedding the visualizations on the background in the video frames.Clause Al 6. The method of any of clauses Al -Al 5, further comprising: obtaining camera parameters for a camera used to generate the video stream or estimating the camera parameters using one or more camera calibration techniques; and displaying visualizations that are static relative to ground using the camera parameters.Clause Al 7. The method of any of clauses Al -Al 6, wherein the visualizable entities include physical entities that appear in the video frames of the video stream.Clause A18. The method of any of clauses A1-A17, wherein the visualizable entities are players in a basketball game, and wherein the background is ground.Clause Al 9. The method of any of clauses Al -Al 8, wherein embedding the visualizations on the background in the video frames is based on tracking gaze of a user.Clause A20. The method of any of clauses Al -Al 9, wherein embedding the visualizations on the background in the video frames is performed without tracking gaze of a user.Clause A21. The method of any of clauses A1-A20, further comprising: highlighting players in the video stream based on ranking the players according to an importance level for the players.Clause A22. The method of clause A21, wherein ranking the players comprises: assigning a highest rank to a first one or more players with a ball, a next ball receiver, and any offensive player with open space, thereby highlighting one or more offensive players.Clause A23. The method of clause A21, wherein ranking the players comprises:assigning an intermediate rank to one or more players who are defending a player with a ball, thereby highlighting one or more important defensive players.Clause A24. The method of clause A21, wherein ranking the players comprises: assigning a lowest rank to all other players.Clause A25. The method of clause A21, wherein ranking the players comprises: detecting if a player is in offense or defense using positional tracking data of the players and a ball, including determining that a player is in offense, if the player or one of teammates is a closest player to the ball within a predefined time interval, and otherwise determining that the player is in defense.Clause A26. The method of clause A25, further comprising: detecting if a player is defending the player with the ball by checking whether the player is within a predefined distance from the player with the ball during a predefined time interval.Clause A27. The method of clause A21, wherein ranking the players comprises: preprocessing the video stream, using positional tracking data to identify which players had a ball or were in open spaces in each frame.Clause A28. The method of clause A21, wherein ranking the players comprises: using a buffer time to identify a next ball receiver by looking ahead a predetermined time period.Clause A29. The method of clause A21, wherein ranking the players comprises: using one or more machine learning models to detect a next ball receiver.Clause A30. The method of clause A21, wherein ranking the players is based on gaze points of a user to adjust importance levels of the players, as well as controlling whose abilities to show.Clause A31. The method of clause A30, wherein ranking the players is based on tracking dwell time for a user gazing at a player in the video stream.Clause A32. The method of clause A21, further comprising: providing a visual feedback to a user gazing at a player, wherein the visual feedback gradually increases until the player is assigned a higher rank for highlighting.Clause A33. The method of clause A21, further comprising:in response to detecting a user gazing away from a player, assigning a lower rank to the player for highlighting.Clause A34. The method of clause A21, further comprising: centering a display at a user’s gaze point with a filter radius of a predetermined number of pixels.Clause A35. The method of clause A21, further comprising: updating centering of a display when a user moves gaze.Clause A36. The method of clause A21, further comprising: using a blurring effect to darken outside a filter radius of a display.Clause A37. The method of any of clauses A1-A36, further comprising: calculating and visualizing players’ offensive and defensive capabilities using embedded visualizations.Clause A38. The method of clause A37, wherein calculating the players’ offensive capabilities comprises: calculating an expected point value for offensive players.Clause A39. The method of clause A37, wherein calculating the players’ defensive capabilities comprises: calculating percentage points difference and distance between a defensive player and an offensive player with a ball, for defensive players.Clause A40. The method of clause A37, wherein visualizing the players’ offensive capabilities comprises: displaying and dynamically updating an offense ring that presents an offensive player’s location-based expected point value based on players’ positions.Clause A41. The method of clause A37, wherein visualizing the players’ defensive capabilities comprises: displaying and dynamically updating a defense shield that represents a defensive player’s location-based percentage points difference and distance between the defensive player and an offensive player with a ball, based on players’ positions.Clause A42. The method of clause A37, further comprises:displaying and dynamically updating based on players’ positions an one-on-one line that visualizes one-on-one relationship between key defenders and an offensive player with a ball, wherein a player with the ball can be defended by multiple defenders.Clause Bl. A computer system for embedding visualizations in a video stream of an athletic game, comprising: one or more processors; and memory; wherein the memory stores one or more programs configured for execution by the one or more processors, and the one or more programs comprising instructions for any of clauses A1-A42.Clause Cl . A non-transitory computer readable storage medium storing one or more programs configured for execution by a computer system having a display, one or more processors, and memory, the one or more programs comprising instructions for any of clauses A1-A42.Clause DI . A system for embedding visualizations in a video stream of an athletic game, the system comprising: a processor; a frame buffer for storing video stream comprising a temporal sequence of video frames; an entity identification module, executable by the processor, configured to identifying visualizable entities in video frames of the video stream; a segmentation module, executable by the processor, configured to segment the visualizable entities from background in the video frames using one or more image segmentation models; and a mapping and embedding module, executable by the processor, configured to (i) mapping the visualizable entities to predetermined visualizations, and (ii) embedding the visualizations on the background in the video frames.Clause D2. The system of clause DI, wherein the entity identification module, the segmentation module, and / or the mapping and embedding module, are configured to perform operations of any of clauses A1-A42.

Claims

What is claimed is:1 . A method of embedding visualizations in a video stream of an athletic game, the method comprising: identifying visualizable entities in video frames of the video stream; segmenting the visualizable entities from background in the video frames using one or more image segmentation models; mapping the visualizable entities to predetermined visualizations; and embedding the visualizations on the background in the video frames.

2. The method of claim 1, wherein identifying the visualizable entities comprises: detecting video objects that correspond to each visualizable entity and key body points of the visualizable entities.

3. The method of claim 2, wherein the video objects comprise one or more bounding boxes and an identity for each visualizable entity, and wherein the key body points comprise key points for each visualizable entity.

4. The method of claim 3, wherein detecting the video objects comprises: obtaining the one or more bounding boxes and the identity in the video frames using an object detection model to locate and classify each visualizable entity into different identities.

5. The method of claim 4, wherein the object detection model outputs a set of bounding boxes associated with their identities and confidence scores.

6. The method of claim 5, wherein detecting the video objects comprises: detecting bounding boxes with a confidence score greater than a predetermined threshold.

7. The method of claim 2, wherein identifying the visualizable entities further comprises: using object trackers to take into account a physical entity’s motion information to complement detection of the video objects.

8. The method of claim 7, wherein each object tracker stores history of an object’s bounding boxes in previous frames and predicts the object’s bounding box in a next frame by using a motion predictor.

9. The method of claim 7, wherein using the object trackers comprises: dividing bounding boxes into a high-quality group, a low-quality group, and a rejected group, based on confidence score for each bounding box and a predetermined threshold.

10. The method of claim 9, wherein using the object trackers comprises: matching existing object trackers with bounding boxes in the high-quality group, or initializing new object trackers for bounding boxes in the high-quality group if matching is not successful.11 . The method of claim 9, wherein using the object trackers comprises: matching existing object trackers with bounding boxes in the low-quality group.

12. The method of claim 7, wherein using the object trackers comprises: outputting bounding boxes with matched trackers.

13. The method of claim 2, wherein identifying the visualizable entities further comprises: using a pose estimation model to obtain each visualizable entities’ key points selected from the group consisting of: head, hands, hip, and feet.

14. The method of claim 2, wherein detecting the video objects comprises: fine-tuning an object detection model on a dataset of the visualizable entities.

15. The method of claim 1, further comprising: concurrently performing identifying the visualizable entities in the video frames of the video stream, segmenting the visualizable entities from the background in the video frames using semantic segmentation, mapping the visualizable entities to predetermined visualizations, and embedding the visualizations on the background in the video frames.

16. The method of claim 1, further comprising: obtaining camera parameters for a camera used to generate the video stream or estimating the camera parameters using one or more camera calibration techniques; anddisplaying visualizations that are static relative to ground using the camera parameters.

17. The method of claim 1, wherein the visualizable entities include physical entities that appear in the video frames of the video stream.

18. The method of claim 1, wherein embedding the visualizations on the background in the video frames is based on tracking gaze of a user.

19. The method of claim 1, further comprising: highlighting players in the video stream based on ranking the players according to an importance level for the players.

20. A system for embedding visualizations in a video stream of an athletic game, the system comprising: a processor; a frame buffer for storing video stream comprising a temporal sequence of video frames; an entity identification module, executable by the processor, configured to identify visualizable entities in video frames of the video stream; a segmentation module, executable by the processor, configured to segment the visualizable entities from background in the video frames using one or more image segmentation models; and a mapping and embedding module, executable by the processor, configured to (i) map the visualizable entities to predetermined visualizations, and (ii) embed the visualizations on the background in the video frames.

Images