System and method for re-identifying athletes in broadcast video

Abstract

The present disclosure provides a system and method for re-identifying an athlete in a broadcast video feed. A computing system retrieves a broadcast video feed for a sporting event. The broadcast video feed includes a plurality of video frames. A computing system generates a plurality of traces based on a plurality of video frames. Each trace includes a plurality of image patches associated with at least one athlete. Each image patch of the plurality of image patches is a subset of a corresponding frame of the plurality of video frames. For each trace, the computing system generates an image patch gallery. The team service number of each athlete can be seen in each image patch of the image library. The computing system matches the cross-gallery traces via a convolutional auto-encoder. The computing system determines a similarity score for each matching trace via a neural network and correlates the two traces based on the determined similarities.

Description

[0001] This application is a divisional application of application number 202080017540.4, application date 2020.2.28, and name “System and method for athlete re-identification in broadcast video”.

[0002] Priority Document

[0003] This application claims priority to U.S. Provisional Application No. 62 / 811,889, filed on February 28, 2019, the entire contents of which are incorporated herein by reference. Technical Field

[0004] The present disclosure generally relates to systems and methods for player re-identification in broadcast video. Background Art

[0005] Athlete tracking data has been used for team and player analysis in many sports for many years. However, conventional athlete tracking systems require sports analytics companies to install fixed cameras at each venue where a team competes. This constraint limits the scalability of athlete tracking systems and data collection for current competitions. In addition, such constraints can incur significant expenses for sports analytics companies due to the costs associated with installing hardware at the necessary venues and maintaining such hardware. Summary of the invention

[0006] In some embodiments, a method for re-identifying an athlete in a broadcast video feed is disclosed herein. A computing system retrieves a broadcast video feed of a sporting event. The broadcast video feed includes a plurality of video frames. The computing system generates a plurality of traces based on the plurality of video frames. Each trace includes a plurality of image patches associated with at least one athlete. Each image patch in the plurality of image patches is a subset of a corresponding frame in the plurality of video frames. For each trace, the computing system generates a gallery of image patches. The uniform number of each athlete is visible in each image patch in the gallery. The computing system matches the traces across the gallery via a convolutional autoencoder. The computing system determines a similarity score for each matching trace via a neural network. The computing system associates two traces based on the determined similarity.

[0007] In some embodiments, a system for re-identifying athletes in a broadcast video feed is disclosed herein. The system includes a processor and a memory. The memory stores programming instructions that, when executed by the processor, perform one or more operations. The one or more operations include retrieving a broadcast video feed of a sporting event. The broadcast video feed includes a plurality of video frames. The one or more operations also include generating a plurality of traces based on the plurality of video frames. Each trace includes a plurality of image patches associated with at least one athlete. Each of the plurality of image patches is a subset of a corresponding frame in the plurality of video frames. The one or more operations also include generating an image patch gallery for each trace. The uniform number of each athlete is visible in each image patch in the gallery. The one or more operations also include matching traces across the gallery via a convolutional autoencoder. The one or more operations also include determining a similarity score for each matching trace via a neural network. The one or more operations also include associating two traces based on the determined similarity.

[0008] In some embodiments, a non-transitory computer-readable medium is disclosed herein. The non-transitory computer-readable medium includes one or more instruction sequences that, when executed by one or more processors, perform one or more operations. One or more operations include retrieving a broadcast video feed of a sporting event. The broadcast video feed includes a plurality of video frames. One or more operations also include generating a plurality of traces based on the plurality of video frames. Each trace includes a plurality of image patches associated with at least one athlete. Each of the plurality of image patches is a subset of a corresponding frame in the plurality of video frames. One or more operations also include generating an image patch gallery for each trace. The uniform number of each athlete is visible in each image patch in the gallery. One or more operations also include matching cross-gallery traces via a convolutional autoencoder. One or more operations also include determining a similarity score for each matching trace via a neural network. One or more operations also include associating two traces based on the determined similarity. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to understand the above features of the present disclosure in detail, the present disclosure briefly summarized above can be described in more detail by referring to the embodiments, some of which are shown in the accompanying drawings. However, it should be noted that the accompanying drawings only show typical embodiments of the present disclosure and should not be considered to limit the scope of the present disclosure, and the present disclosure may allow other equivalent embodiments.

[0010] Figure 1 is a block diagram illustrating a computing environment according to an example embodiment.

[0011] Figure 2 is a block diagram illustrating a computing environment according to an example embodiment.

[0012] Figure 3 is to illustrate the context according to an example embodiment in conjunction with Figure 2 and Figures 4 to 10 A block diagram of the operational aspects of the discussion.

[0013] Figure 4 is a flow chart illustrating a method of generating a player trail according to an example embodiment.

[0014] Figure 5 is a flow chart illustrating a method of generating a trackable frame according to an example embodiment.

[0015] Figure 6 It is shown that according to an example embodiment, the above combination Figure 5 A block diagram of the operational aspects of the discussion.

[0016] Figure 7 is a flow chart illustrating a method of calibrating a camera for each trackable frame according to an example embodiment.

[0017] Figure 8 It is shown that according to an example embodiment, the above combination Figure 7 A block diagram of the operational aspects of the discussion.

[0018] Fig. 9 is a flow chart illustrating a method of tracking a player according to an example embodiment.

[0019] Fig.10 is a flow chart illustrating a method of tracking a player according to an example embodiment.

[0020] Fig.11 It is shown that according to an example embodiment, the above combination Fig.10 A block diagram of the operational aspects of the discussion.

[0021] Fig.12 is a block diagram illustrating the architecture of a re-identification agent according to an example embodiment.

[0022] Fig.13A is a block diagram illustrating a computing device according to an example embodiment.

[0023] Fig. 13B is a block diagram illustrating a computing device according to an example embodiment.

[0024] To facilitate understanding, like reference numerals have been used, where appropriate, to designate like elements that are common to the figures. It is contemplated that elements disclosed in one embodiment may be suitably utilized in other embodiments without specific recitation. DETAILED DESCRIPTION

[0025] Vision-based tracking systems have been widely used in many professional sports leagues for the past decade to capture the positional information of players during competition. This data can be used to generate fine-grained trajectories, which are extremely valuable for coaches and experts to analyze and train their players. Although the players can be detected fairly well in each frame, conventional systems suffer from gaps between player trajectories that require a lot of manual annotation. Most of these gaps can be caused by player occlusion or the player wandering outside the scene.

[0026] The specific way in which one or more technologies of the present disclosure improve conventional systems is to provide a system that can generate fine-grained athlete trajectories despite athlete occlusion or athlete wandering outside the scene. For example, one or more technologies of the present disclosure may involve the operation of identifying and associating a person in different physical locations over time after having previously observed a person elsewhere. The problems to be solved by the technology described herein are more complex in the field of team sports because, unlike ordinary surveillance, there is no difference in the appearance of the athletes. The technology described herein overcomes this challenge by using athlete uniform (or uniform) information to help identify athletes. However, since athletes constantly change positions during the game, it is not easy to simply use uniform information. In order to accurately identify athlete trajectories, the system of the present disclosure can identify video frames with visible uniform numbers and can associate identities using deep neural networks.

[0027] Figure 1 1 is a block diagram illustrating a computing environment 100 according to an example embodiment. The computing environment 100 may include a camera system 102 , an organizational computing system 104 , and one or more client devices 108 in communication via a network 105 .

[0028] Network 105 can take any suitable type, including via the Internet, a separate connection such as a cellular or Wi-Fi network. In some embodiments, network 105 can use a wireless network such as radio frequency identification (RFID), near field communication (NFC), Bluetooth, etc. TM , Bluetooth Low Energy TM (BLE), Wi-Fi TM 、ZigBee TM , ambient backscatter communication (ABC) protocol, USB, WAN, or LAN to connect terminals, services, and mobile devices. Because the information transmitted may be personal or confidential, security issues may require that one or more of these types of connections be encrypted or otherwise secured. However, in some embodiments, the information transmitted may not be very personal, so the network connection may be selected for convenience rather than security.

[0029] The network 105 may include any type of computer network arrangement for exchanging data or information. For example, the network 105 may be the Internet, a dedicated data network, a virtual private network using a public network, and / or other suitable connection that enables components in the computing environment 100 to send and receive information between components of the environment 100.

[0030] The camera system 102 can be positioned in the field 106. For example, the field 106 can be configured to host a sporting event including one or more players 112. The camera system 102 can be configured to capture the movement of all players (i.e., athletes) on the playing surface and one or more other related objects (e.g., balls, referees, etc.). In some embodiments, the camera system 102 can be an optical-based system, for example, using multiple fixed cameras. For example, a system consisting of six stationary calibrated cameras can be used, which project the three-dimensional positions of the athletes and the ball onto a two-dimensional top view of the court. In another example, a mixture of stationary cameras and non-stationary cameras can be used to capture the movement of all players on the court and one or more related objects. It can be appreciated by those skilled in the art that many different court camera views (e.g., high sideline views, free throw line views, crowd views, scrum views, goal area views, etc.) can be generated using such a camera system (e.g., camera system 102). In general, the camera system 102 can be used for a broadcast feed of a given competition. Each frame in the broadcast feed can be stored in a game file 110.

[0031] The camera system 102 can be configured to communicate with the organization computing system 104 via the network 105. The organization computing system 104 can be configured to manage and analyze the broadcast feed captured by the camera system 102. The organization computing system 104 can include at least a Web client application server 114, a data store 118, an automatic editing agent 120, a data set generator 122, a camera calibrator 124, an athlete tracking agent 126, and an interface agent 128. Each of the automatic editing agent 120, the data set generator 122, the camera calibrator 124, the athlete tracking agent 126, and the interface agent 128 can be composed of one or more software modules. One or more software modules can be a collection of codes or instructions stored on a medium (e.g., a memory of the organization computing system 104), which represent a series of machine instructions (e.g., program code) that implement one or more algorithmic steps. Such machine instructions can be actual computer codes that the processor of the organization computing system 104 interprets to implement instructions, or can be high-level encodings of instructions that interpret to obtain actual computer codes. One or more software modules can also include one or more hardware components. One or more aspects of the example algorithms may be performed by hardware components (eg, circuits) themselves, rather than as a result of instructions.

[0032] The data storage 118 may be configured to store one or more game files 124. Each game file 124 may include broadcast data for a given competition. For example, the broadcast data may be a plurality of video frames captured by the camera system 102.

[0033] The automatic editing agent 120 can be configured to parse the broadcast feed of a given competition to identify a unified view of the competition. In other words, the automatic editing agent 120 can be configured to parse the broadcast feed to identify all information frames captured from the same view. In one example, in basketball, for example, the unified view can be a high sideline view. The automatic editing agent 120 can clip or split the broadcast feed (e.g., video) into its components (e.g., different scenes in a movie, advertisements in a competition, etc.). In order to generate a unified view, the automatic editing agent 120 can identify those parts that capture the same view (e.g., a high sideline view). Accordingly, the automatic editing agent 120 can remove all (or part) of the untrackable parts of the broadcast feed (e.g., close-ups of athletes, advertisements, halftime performances, etc.). The unified view can be stored in a database as a trackable frame set.

[0034] The data set generator 122 may be configured to generate a plurality of data sets based on the trackable frame. In some embodiments, the data set generator 122 may be configured to identify body posture information. For example, the data set generator 122 may use body posture information to detect athletes in the trackable frame. In some embodiments, the data set generator 122 may be configured to further track the movement of the ball or puck in the trackable frame. In some embodiments, the data set generator 122 may be configured to segment the playing field where the event is occurring to identify one or more marks of the playing field. For example, the data set generator 122 may be configured to identify court (e.g., basketball, tennis, etc.) marks, sports field (e.g., baseball, American football, soccer, rugby, etc.) marks, ice rink (e.g., hockey) marks, etc. The plurality of data sets generated by the data set generator 122 may then be used by the camera calibrator 124 to calibrate the camera of each camera system 102.

[0035] The camera calibrator 124 may be configured to calibrate the cameras of the camera system 102. For example, the camera calibrator 124 may be configured to project the players detected in the trackable frame to real-world coordinates for further analysis. The cameras in the camera system 102 are constantly moving to focus on the ball or key actions, so such cameras cannot be pre-calibrated. The camera calibrator 124 may be configured to improve or optimize the player projection parameters using the homography matrix.

[0036] Athlete tracking agent 126 can be configured to generate a track for each athlete on the field. For example, athlete tracking agent 126 can utilize athlete posture detection, camera calibration and broadcast frame to generate such a track. In certain embodiments, athlete tracking agent 126 can also be configured to generate a track for each athlete, even if, for example, the athlete is currently outside the trackable frame. For example, athlete tracking agent 126 can utilize body posture information to link the athlete who has left the view frame.

[0037] Interface agent 128 may be configured to generate one or more graphical representations corresponding to the tracks of each athlete generated by athlete tracking agent 126. For example, interface agent 128 may be configured to generate one or more graphical user interfaces (GUIs) that include a graphical representation of each predicted athlete tracked generated by athlete tracking agent 126.

[0038] Client devices 108 can communicate with organizational computing system 104 via network 105. Client devices 108 can be operated by users. For example, client devices 108 can be mobile devices, tablet computers, desktop computers, or any computing system with the capabilities described herein. Users can include, but are not limited to, individuals such as subscribers, customers, potential customers, or customers of entities associated with organizational computing system 104, such as individuals who have obtained, will obtain, or can obtain products, services, or consulting from an entity associated with organizational computing system 104.

[0039] The client device 108 may include at least an application 132. The application 132 may represent a web browser that allows access to a website or a stand-alone application. The client device 108 may access the application 132 to access one or more functions of the organization computing system 104. The client device 108 may communicate over the network 105 to request a web page, such as a web page from a web client application server 114 of the organization computing system 104. For example, the client device 108 may be configured to execute the application 132 to access content managed by the web client application server 114. The content displayed to the client device 108 may be transmitted from the web client application server 114 to the client device 108 and then processed by the application 132 for display through a graphical user interface (GUI) of the client device 108.

[0040] Figure 2 2 is a block diagram illustrating a computing environment 200 according to an example embodiment. As shown, the computing environment 200 includes an automatic editing agent 120, a data set generator 122, a camera calibrator 124, and a player tracking agent 126 that communicate via a network 205.

[0041] Network 205 can take any suitable type, including via the Internet, a separate connection such as a cellular or Wi-Fi network. In some embodiments, network 205 can use a wireless network such as radio frequency identification (RFID), near field communication (NFC), Bluetooth, etc. TM , Bluetooth Low Energy TM (BLE), Wi-Fi TM 、ZigBee TM , ambient backscatter communication (ABC) protocol, USB, WAN, or LAN to connect terminals, services, and mobile devices. Because the information transmitted may be personal or confidential, security issues may require that one or more of these types of connections be encrypted or otherwise secured. However, in some embodiments, the information transmitted may not be very personal, so the network connection may be selected for convenience rather than security.

[0042] The network 205 may include any type of computer network arrangement for exchanging data or information. For example, the network 205 may be the Internet, a private data network, a virtual private network using a public network, and / or other suitable connection, enabling components in the computing environment 200 to send and receive information between components of the environment 200.

[0043] The automatic clipping agent 120 may include a principal component analysis (PCA) agent 202, a clustering model 204, and a neural network 206. As described above, the automatic clipping agent 120 may be used to clip or segment a video into its component parts when attempting to understand and extract data from a broadcast feed. In some embodiments, the automatic clipping agent 120 may focus on separating a predefined unified view (e.g., a high-edge view) from all other parts of the broadcast stream.

[0044] The PCA agent 202 can be configured to perform each extraction of frame features from the broadcast feed using PCA analysis. For example, given a pre-recorded video, the PCA agent 202 can extract one frame every X seconds (e.g., every 10 seconds) to build a PCA model of the video. In some embodiments, the PCA agent 202 can use incremental PCA to generate the PCA model, from which the PCA agent 202 can select the previous subset of components (e.g., the first 120 components) to generate the PCA model. The PCA agent 202 can also be configured to extract one frame from the broadcast stream every X seconds (e.g., every second) and compress the frames using the PCA model. In some embodiments, the PCA agent 202 can compress the frames into a 120-dimensional form using the PCA model. For example, the PCA agent 202 can solve the principal components on a per-video basis and save the first 100 components of each frame to ensure accurate editing.

[0045] The clustering model 204 can be configured to cluster the first subset of components into clusters. For example, the clustering model 204 can be configured to center, normalize, and cluster the first 120 components into multiple clusters. In some embodiments, for clustering of compressed frames, the clustering model 204 can implement K-means clustering. In some embodiments, the clustering model 204 can set k=9 clusters. K-means clustering attempts to obtain some data x={x1, x2, ..., x n} and divide it into k subsets, S={S1,S2,...S k}, optimized as follows:

[0046]

[0047] where μ j For set S j In other words, clustering model 204 attempts to find the cluster with the smallest inter-cluster variance using the K-means clustering technique. Clustering model 204 may label each frame with its respective cluster number (eg, cluster 1, cluster 2, ..., cluster k).

[0048] The neural network 206 may be configured to classify each frame as trackable or untrackable. Trackable frames may represent frames that include a captured unified view (e.g., a high edge view). Untrackable frames may represent frames that do not capture a unified view. To train the neural network 206, an input dataset may be used that includes thousands of frames pre-labeled as trackable or untrackable that have been run through a PCA model. Each compressed frame and label pair (i.e., cluster number and trackable / untrackable) may be provided to the neural network 206 for training.

[0049] In some embodiments, the neural network 206 may include four layers. The four layers may include an input layer, two hidden layers, and an output layer. In some embodiments, the input layer may include 120 units. In some embodiments, each hidden layer may include 240 units. In some embodiments, the output layer may include 2 units. The input layer and each hidden layer may use a sigmoid activation function. The output layer may use a SoftMax activation function. To train the neural network 206, the automatic editing agent 120 may reduce (e.g., minimize) the predicted label of the sample as follows With the true label y j The binary cross entropy loss between:

[0050]

[0051] Accordingly, once trained, the neural network 206 can be configured to classify each frame as untrackable or trackable. In this way, each frame can have two labels: a cluster number and a trackable / untrackable classification. The automatic editing agent 120 can use these two labels to determine whether a given cluster is considered trackable or untrackable. For example, if the automatic editing agent 120 determines that a threshold number of frames in a cluster are considered trackable (e.g., 80%), the automatic editing agent 120 can conclude that all frames in the cluster are trackable. In addition, if the automatic editing agent 120 determines that less than a threshold number of frames in a cluster are considered untrackable (e.g., less than 30%), the automatic editing agent 120 can conclude that all frames in the cluster are untrackable. Further, if the automatic editing agent 120 determines that a specific number of frames in a cluster are considered trackable (e.g., 30% to 80%), the automatic editing agent 120 can request an administrator to further analyze the cluster. Once each frame is classified, the automatic editing agent 120 can clip or split the trackable frames. The auto-editing agent 120 may store the segments of trackable frames in a database 205 associated therewith.

[0052] Data set generator 122 can be configured to generate multiple data sets from automatic editing agent 120. As shown, data set generator 122 can include posture detector 212, ball detector 214 and field segmenter 216. Posture detector 212 can be configured to detect the athlete in the broadcast feed. Data set generator 122 can provide trackable frames stored in database 205 and broadcast video feed as input to posture detector 212. In some embodiments, posture detector 212 can implement open posture (Open Pose) to generate body posture data to detect the athlete in broadcast feed and trackable frame. In some embodiments, posture detector 212 can implement sensors positioned on the athlete to capture body posture information. Generally speaking, posture detector 212 can use any means to obtain body posture information from broadcast video feed and trackable frame. Output from posture detector 212 can be posture data stored in database 215 associated with data set generator 122.

[0053] The ball detector 214 can be configured to detect and track a ball (or puck) within a broadcast feed. The dataset generator 122 can provide the ball detector 214 with trackable frames stored in the database 205 and the broadcast video feed as input. In some embodiments, the ball detector 214 can utilize a fast regional convolutional neural network (Faster R-CNN) to detect and track balls in trackable frames and broadcast video feeds. Faster R-CNN is a region proposal network. Faster R-CNN uses a convolutional neural network to propose regions of interest and then classify objects in each region of interest. Because it is a single unified network, the regions of interest and classification steps can improve each other, allowing classification to handle objects of various sizes. The output from the ball detector 214 can be ball detection data stored in a database 215 associated with the dataset generator 122.

[0054] The field segmenter 216 may be configured to identify field markers in the broadcast feed. The dataset generator 122 may provide the field segmenter 216 with the trackable frames stored in the database 205 and the broadcast video feed as input. In some embodiments, the field segmenter 216 may be configured to utilize a neural network to identify field markers. The output of the field segmenter 216 may be the field markers stored in the database 215 associated with the dataset generator 122.

[0055] The camera calibrator 124 can be configured to solve the problem of mobile camera calibration in sports. The camera calibrator 124 can include a spatial transmission network 224 and an optical flow module 226. The camera calibrator 124 can receive as input the segmented field information, trackable clip information, and pose information generated by the field segmenter 216. Given such input, the camera calibrator 124 can be configured to project the coordinates in the image frame to real-world coordinates for tracking analysis.

[0056] Keyframe matching module 224 can receive the output from field segmenter 216 and template set as input. For each frame, keyframe matching module 224 can match the output from field segmenter 216 with the template. Those frames that can match a given template are considered keyframes. In some embodiments, keyframe matching module 224 can implement a neural network to match one or more frames. In some embodiments, keyframe matching module 224 can implement cross-correlation to match one or more frames.

[0057] The spatial transformer network (STN) 224 may be configured to receive the identified keyframes as input from the keyframe matching module 224. The STN 224 may implement a neural network to fit a playing field model to the playing field segmentation information. By fitting the playing field model to such output, the STN 224 may generate a homography matrix for each keyframe.

[0058] The optical flow module 226 can be configured to identify the motion pattern of an object from one trackable frame to another. In some embodiments, the optical flow module 226 can receive as input the trackable frame information and body pose information of the athlete in each trackable frame. The optical flow module 226 can use the body pose information to remove the athlete from the trackable frame information. Once removed, the optical flow module 226 can determine the inter-frame motion to identify the camera motion between consecutive frames. In other words, the optical flow module 226 can identify the flow field from one frame to the next.

[0059] The optical flow module 226 and the STN 224 can work together to generate a homography matrix. For example, the optical flow module 226 and the STN 224 can generate a homography matrix for each trackable frame so that the camera can be calibrated for each frame. The homography matrix can be used to project the track or position of the athlete into real-world coordinates. For example, the homography matrix can indicate a two-dimensional to two-dimensional transformation, which can be used to project the position of the athlete from the image coordinates to the real-world coordinates on the field of play.

[0060] The player tracking agent 126 may be configured to generate a track for each player in the game. The player tracking agent 126 may include a neural network 232 and a re-identification agent 232. The player tracking agent 126 may receive as input a trackable frame, pose data, calibration data, and a broadcast video frame. In a first stage, the player tracking agent 126 may match player patch pairs derived from pose information based on appearance and distance. For example, let Let be the athlete patch of the jth athlete at time t. is the image coordinates of the jth athlete at time t width and height Thus, the player tracking agent 126 can use appearance correlation and Associate any detection pair as follows:

[0061]

[0062] Where I is the bounding box location (x, y), width w, and height h; C is the cross - correlation between image patches (e.g., image cropping using the bounding box) and measures the similarity between two image patches; and L is a measure of the difference (e.g., distance) between two bounding boxes I.

[0063] Performing this operation for each pair can generate a large set of short tracklets. Then, the endpoints of these tracklets can be correlated with each other based on motion consistency and color histogram similarity.

[0064] For example, let v i be the extrapolated velocity at the end of the i - th tracklet, and let v j be the extrapolated velocity at the start of the j - th tracklet. Then c ij = v i ·v j can represent the motion consistency score. Additionally, let p(h) i represent the color likelihood, where h exists in image patch i. The athlete tracking agent 126 can measure the color histogram similarity using the Bhattacharyya Distance:

[0065] D B (p i , p j ) = -ln(BC(p i , p j ))), where

[0066] To reiterate, the tracking agent 120 finds matching tracklet pairs as follows:

[0067]

[0068] Solving for each pair of interrupted tracklets can produce a net tracklet set, while leaving some tracklets with large gaps (i.e., many frames). To connect the large gaps, the athlete tracking agent can augment the proximity metric to include motion field estimation, which can account for changes in the athlete's direction over many frames.

[0069] The motion field can be a vector field that represents the velocity magnitude and direction as vectors at each position on the playing field. Given the known velocities of many athletes on the field, cubic spline interpolation can be used to generate a complete motion field. For example, let be the field position of athlete i at each time t. Then there may be a pair of points with displacement if λ < T. Accordingly, the motion field can be as follows:

[0070]

[0071] where G(x,5) can be a Gaussian Kernel with a standard deviation of approximately five feet. In other words, the motion field can be a Gaussian Blur of all displacements.

[0072] The neural network 232 may be used to predict player trajectories given real player trajectories. i , the speed of each player in each frame can be calculated, which can provide the ground truth motion field for the neural network 232 to learn. For example, given the real player trajectory set X i , the player tracking agent 126 may be configured to generate a set in can be a predicted motion field. For example, the neural network 232 can be trained to The player trajectory agent can then generate neighbor scores for any tracking gap size λ as follows:

[0073] K ij =V(x,λ)·d ij

[0074] in is the displacement vector between all interrupted traces with a gap size of λ.

[0075] The re-identification agent 234 may be configured to link players that have left the view frame. The re-identification agent 234 may include a trace generator 236 , a conditional autoencoder 240 , and a Siamese network 242 .

[0076] The trace generator 236 can be configured to generate a trace gallery. The trace generator 236 can receive multiple traces from the database 205. For each trace X, a player identity tag y can be included, and for each player patch I, posture information p can be provided by the posture detection stage. In view of the set of player traces, the trace generator 236 can establish a gallery for each trace, wherein the player's uniform number (or some other static feature) is always visible. The body posture information generated by the data set generator 122 allows the trace generator 236 to determine the position of the player. For example, the trace generator 236 can utilize a heuristic method, wherein the standardized shoulder width can be used as follows to determine the position:

[0077]

[0078] Where l can represent the position of a body part. Shoulder width can be normalized by torso length to eliminate scaling effects. Shoulders should be separated when the athlete is facing the camera or facing away from the camera, so the trace generator 236 can use those S orientPatches larger than the threshold are used to build a gallery. After this stage, each trace X n The following galleries can be included:

[0079]

[0080] The conditional autoencoder 240 can be configured to recognize one or more features in each trace. For example, unlike conventional approaches to re-identification problems, players in team sports can have very similar appearance features, such as clothing style, clothing color, and skin color. One of the more intuitive differences may be the team number that can be displayed on the front and / or back of each uniform. In order to capture these specific features, the conditional autoencoder 240 can be trained to recognize such features.

[0081] In some embodiments, the conditional autoencoder 240 may be a three-layer convolutional autoencoder, where the kernel size of all three layers may be 3x3, with 64, 128, and 128 channels, respectively. These hyperparameters may be tuned to ensure that the uniform numbers can be identified from the reconstructed image, so that the expected features can be learned in the autoencoder. In some embodiments, f(I i ) can be used to represent the features learned from image i.

[0082] Using conditional autoencoders 240 can improve the conventional process for a variety of reasons. First, there is usually not enough training data for each athlete, as some athletes only play a short amount of time in each game. Second, different teams may have the same uniform colors and uniform numbers, so it may be difficult to classify these athletes.

[0083] Siamese network 242 can be used to determine the similarity between two image patches. For example, Siamese network 242 can be trained to determine the similarity between two image patches based on their feature representations f(I). Given two image patches, their feature representations f(I i ) and f(I j ) can be flattened, concatenated, and input into the perception network. In some embodiments, the L2 norm can be used to connect the two sub-networks f(I i ) and f(I j ). In some embodiments, the perception network may include three layers, which may include 1024, 512, and 216 hidden units, respectively. Such a network can be used to determine the similarity s(I) between each pair of image patches of two traces without temporal overlap. i ,I j ). To enhance the robustness of the prediction, the final similarity score of two traces can be the average of all pairwise scores in their respective galleries:

[0084]

[0085] This similarity score can be calculated for every two traces that do not overlap in time. If the score is above a certain threshold, the two traces can be associated.

[0086] Figure 3 is to illustrate the context according to an example embodiment in conjunction with Figure 2 and Figures 4 to 10 Block diagram 300 of the operational aspects of the discussion. Block diagram 300 may illustrate the overall workflow of organizing computing system 104 in generating athlete tracking information. Block diagram 300 may include multiple groups of operations 302 to 308. One group of operations 302 may involve generating a trackable frame (e.g., Figure 5 A set of operations 304 may involve generating one or more data sets from the trackable frames (e.g., operations performed by the data set generator 122). A set of operations 306 may involve camera calibration operations (e.g., Figure 7 A set of operations 308 may be directed to generating and predicting player traces (e.g., Fig. 9 Method 900 and Fig.10 Method 1000 in ).

[0087] Figure 4 is a flow chart illustrating a method 400 of generating a player trail according to an example embodiment. The method 400 may begin at step 402 .

[0088] At step 402, the organization computing system 104 may receive (or retrieve) a broadcast feed of the event. In some embodiments, the broadcast feed may be a live feed received in real time (or near real time) from the camera system 102. In some embodiments, the broadcast feed may be a broadcast feed of a game that has ended. In general, the broadcast feed may include multiple frames of video data. Each frame may capture a different camera perspective.

[0089] At step 404, the organizational computing system 104 may segment the broadcast feed into unified views. For example, the automatic editing agent 120 may be configured to parse multiple data frames in the broadcast feed to segment trackable frames from untrackable frames. Generally speaking, trackable frames may include those frames that point to the unified view. For example, the unified view may be considered to be a high sideline view. In other examples, the unified view may be a goal area view. In other examples, the unified view may be a top camera view.

[0090] At step 406, the organizational computing system 104 may generate a plurality of data sets from the trackable frames (i.e., the unified views). For example, the data set generator 122 may be configured to generate a plurality of data sets based on the trackable clips received from the automatic clipping agent 120. In some embodiments, the gesture detector 212 may be configured to detect athletes within a broadcast feed. The data set generator 122 may provide the trackable frames stored in the database 205 and the broadcast video feed as input to the gesture detector 212. The output from the gesture detector 212 may be gesture data stored in the database 215 associated with the data set generator 122.

[0091] Ball detector 214 may be configured to detect and track a ball (or puck) within a broadcast feed. Dataset generator 122 may provide trackable frames stored in database 205 and a broadcast video feed as input to ball detector 214. In some embodiments, ball detector 214 may utilize Faster R-CNN to detect and track balls in trackable frames and broadcast video feeds. Output from ball detector 214 may be ball detection data stored in database 215 associated with dataset generator 122.

[0092] The field segmenter 216 may be configured to identify field markers in the broadcast feed. The dataset generator 122 may provide the field segmenter 216 with the trackable frames stored in the database 205 and the broadcast video feed as input. In some embodiments, the field segmenter 216 may be configured to utilize a neural network to identify field markers. The output of the field segmenter 216 may be the field markers stored in the database 215 associated with the dataset generator 122.

[0093] Accordingly, the dataset generator 120 may generate information for player positions, ball positions, and court portions in all trackable frames for further analysis.

[0094] In step 408, the organizational computing system 104 may calibrate the camera in each trackable frame based on the data set generated in step 406. For example, the camera calibrator 124 may be configured to calibrate the camera in each trackable frame by generating a homography matrix using the trackable frame and body pose information. The homography matrix allows the camera calibrator 124 to obtain those trajectories of each athlete in a given frame and project those trajectories into real-world coordinates. By projecting the athlete positions and trajectories into real-world coordinates for each frame, the camera calibrator 124 can ensure that the camera is calibrated for each frame.

[0095] In step 410, the organization computing system 104 can be configured to generate or predict each athlete's trace. For example, the athlete tracking agent 126 can be configured to generate or predict each athlete's trace in the competition. The athlete tracking agent 126 can receive and track frames, posture data, calibration data and broadcast video frames as input. Using such input, the athlete tracking agent 126 can be configured to build the athlete's movement in the entire given competition. In addition, the athlete tracking agent 126 can be configured to predict the athlete's track in view of each athlete's previous movement.

[0096] Figure 5 is a flowchart illustrating a method 500 for generating a trackable frame according to an example embodiment. The method 500 may correspond to the above combined Figure 4 Operation 404 is discussed. Method 500 may begin at step 502 .

[0097] At step 502, the organization computing system 104 may receive (or retrieve) a broadcast feed of the event. In some embodiments, the broadcast feed may be a live feed received in real time (or near real time) from the camera system 102. In some embodiments, the broadcast feed may be a broadcast feed of a game that has ended. In general, the broadcast feed may include multiple frames of video data. Each frame may capture a different camera perspective.

[0098] In step 504, the organization computing system 104 can generate a frame set for image classification. For example, the automatic editing agent 120 can perform frame-by-frame feature extraction from the broadcast feed using PCA analysis. For example, in view of pre-recorded video, the automatic editing agent 120 can extract a frame every X seconds (e.g., every 10 seconds) to build a PCA model of the video. In some embodiments, the automatic editing agent 120 can use incremental PCA to generate a PCA model, and the automatic editing agent 120 can thus select the previous component subset (e.g., the first 120 components) to generate the PCA model. The automatic editing agent 120 can also be configured to extract a frame from the broadcast stream every X seconds (e.g., every second) and use the PCA model to compress these frames. In some embodiments, the automatic editing agent 120 can compress the frame into a 120-dimensional form using the PCA model. For example, the automatic editing agent 120 can solve the principal component in a per-video manner and save the first 100 components of each frame to ensure accurate editing. Such a compressed frame subset can be regarded as a frame set for image classification. In other words, the PCA model can be used to compress each frame into a small vector, so that the frames can be clustered more effectively. Compression can be performed by selecting the first N components from the PCA model to represent the frame. In some examples, N can be 100.

[0099] At step 506, the organization computing system 104 may assign each frame in the frame set to a given cluster. For example, the automatic editing agent 120 may be configured to center, normalize, and cluster the first 120 components into a plurality of clusters. In some embodiments, for clustering of compressed frames, the automatic editing agent 120 may implement K-means clustering. In some embodiments, the automatic editing agent 120 may set k=9 clusters. K-means clustering attempts to obtain some data x={x1, x2, ..., x n} and divide it into k subsets, S={S1,S2,...S k}, optimized as follows:

[0100]

[0101] where μ j For set S j In other words, clustering model 204 attempts to find the cluster with the smallest inter-cluster variance using the K-means clustering technique. Clustering model 204 may label each frame with its respective cluster number (eg, cluster 1, cluster 2, ..., cluster k).

[0102] At step 508, the organization computing system 104 may classify each frame as trackable or untrackable. For example, the automatic editing agent 120 may classify each frame as trackable or untrackable using a neural network. Trackable frames may represent frames that include capturing a unified view (e.g., a high edge view). Untrackable frames may represent frames that do not capture a unified view. In order to train a neural network (e.g., neural network 206), an input data set including thousands of frames pre-labeled as trackable or untrackable that are run through a PCA model may be used. Each compressed frame and label pair (i.e., cluster number and trackable / untrackable) may be provided to the neural network for training. Accordingly, once trained, the automatic editing agent 120 may classify each frame as untrackable or trackable. In this way, each frame may have two labels: a cluster number and a trackable / untrackable classification.

[0103] At step 510, the organizational computing system 104 may compare each cluster to a threshold value. For example, the automatic editing agent 120 may utilize these two labels to determine whether a given cluster is considered trackable or untrackable. In some embodiments, if the automatic editing agent 120 determines that a threshold number of frames in a cluster are considered trackable (e.g., 80%), the automatic editing agent 120 may conclude that all frames in the cluster are trackable. Additionally, if the automatic editing agent 120 determines that less than a threshold number of frames in a cluster are considered untrackable (e.g., less than 30%), the automatic editing agent 120 may conclude that all frames in the cluster are untrackable. Further, if the automatic editing agent 120 determines that a specific number of frames in a cluster are considered trackable (e.g., 30% to 80%), the automatic editing agent 120 may request an administrator to further analyze the cluster.

[0104] If the organization computing system 104 determines in step 510 that greater than a threshold number of frames in a cluster are trackable, then in step 512 the auto-editing agent 120 may classify the cluster as trackable.

[0105] However, if the organization computing system 104 determines in step 510 that less than a threshold number of frames in a cluster are trackable, then in step 514 the auto-editing agent 120 may classify the cluster as untrackable.

[0106] Figure 6 6 is a block diagram 600 illustrating operational aspects discussed above in connection with method 500 according to an example embodiment. As shown, block diagram 600 may include multiple sets of operations 602-608.

[0107] In one set of operations 602, video data (e.g., broadcast video) may be provided to an automatic editing agent 120. The automatic editing agent 120 may extract frames from the video. In some embodiments, the automatic editing agent 120 may extract frames from the video at a low frame rate. The automatic editing agent may use an incremental PCA algorithm to select the first 120 components (e.g., frames) from the set of frames extracted by the automatic editing agent 120. Such operations may generate a video-specific PCA model.

[0108] In a set of operations 604, video data (e.g., broadcast video) may be provided to the automatic editing agent 120. The automatic editing agent 120 may extract frames from the video. In some embodiments, the automatic editing agent 120 may extract frames from the video at a medium frame rate. The automatic editing agent 120 may use a video-specific PCA model to compress the frames extracted by the automatic editing agent 120.

[0109] At a set of operations 606, the compressed frames and the preselected number of desired clusters may be provided to the automatic editing agent 120. The automatic editing agent 120 may group the frames into one or more clusters as described by the preselected number of desired clusters using a K-means clustering technique. The automatic editing agent 120 may assign a cluster label to each compressed frame. The automatic editing agent 120 may also be configured to classify each frame as trackable or non-trackable. The automatic editing agent 120 may so label each corresponding frame.

[0110] In a set of operations 608, the automatic editing agent 120 can analyze each cluster to determine whether the cluster includes at least a threshold number of trackable frames. For example, as shown, if 80% of the frames in the cluster are classified as trackable, the automatic editing agent 120 can consider the entire cluster to be trackable. However, if less than 80% of the frames in the cluster are classified as trackable, the automatic editing agent can determine whether at least a second threshold number of frames in the cluster are trackable. For example, if 70% of the frames in the cluster are classified as untrackable, the automatic editing agent 120 can consider the entire cluster to be trackable. However, if less than 70% of the frames in the cluster are classified as untrackable, i.e., 30% to 70% are trackable, manual annotation can be requested.

[0111] Figure 7 700 is a flowchart illustrating a method 700 for calibrating a camera for each trackable frame according to an example embodiment. The method 700 may correspond to the method described above in conjunction with Figure 4 Operation 408 is discussed. Method 700 may begin at step 702 .

[0112] At step 702, the organizational computing system 104 may retrieve the video data and the pose data for analysis. For example, the camera calibrator 124 may retrieve the trackable frames of a given competition and the pose data of the athletes in each trackable frame from the database 205. Following step 702, the camera calibrator 124 may perform two parallel processes to generate a homography matrix for each frame. Accordingly, the following operations are not intended to be discussed as being performed sequentially, but may be performed in parallel or sequentially.

[0113] At step 704, the organizational computing system 104 may remove the athlete from each trackable frame. For example, the camera calibrator 124 may parse each trackable frame retrieved from the database 205 to identify one or more athletes contained therein. The camera calibrator 124 may remove the athlete from each trackable frame using the pose data retrieved from the database 205. For example, the camera calibrator 124 may identify those pixels that correspond to the pose data and remove the identified pixels from a given trackable frame.

[0114] At step 706, the tissue computing system 104 may identify the motion of an object (e.g., a surface, an edge, etc.) between consecutive trackable frames. For example, the camera calibrator 124 may analyze consecutive trackable frames, removing the athlete therefrom, to determine the motion of the object from one frame to the next. In other words, the optical flow module 226 may identify a flow field between consecutive trackable frames.

[0115] In step 708, the organizational computing system 104 can match the output from the field segmenter 216 with the template set. For example, the camera calibrator 124 can match one or more frames with clear field images with one or more templates. The camera calibrator 124 can parse the group of trackable clips to identify those clips that provide clear field images and tags therein. Based on the selected clips, the camera calibrator 124 can compare such images with the field templates. Each template can represent a different camera perspective of the field. Those frames that can match a given template are considered key frames. In some embodiments, the camera calibrator 124 can implement a neural network to match one or more frames. In some embodiments, the camera calibrator 124 can implement cross-correlation to match one or more frames.

[0116] At step 710, the tissue computing system 104 may fit the field model to each key frame. For example, the camera calibrator 124 may be configured to receive the identified key frame as input. The camera calibrator 124 may implement a neural network to fit the field model to the field segmentation information. By fitting the field model to such output, the camera calibrator 124 may generate a homography matrix for each key frame.

[0117] At step 712, the tissue computing system 104 may generate a homography matrix for each trackable frame. For example, the camera calibrator 124 may generate a homography matrix for each frame using the flow field identified in step 706 and the homography matrix for each key frame. The homography matrix may be used to project the track or position of the athlete into real-world coordinates. For example, given the geometric transformation represented by the homography matrix, the camera calibrator 124 may use its transformation to project the position of the athlete on the image into real-world coordinates on the field of play.

[0118] At step 714 , the tissue computing system 104 may calibrate each camera based on the homography matrix.

[0119] Figure 8 8 is a block diagram 800 illustrating operational aspects discussed above in connection with method 700 according to an example embodiment. As shown, block diagram 800 may include input 802, a first set of operations 804, and a second set of operations 806. The first set of operations 804 and the second set of operations 806 may be performed in parallel.

[0120] Input 802 may include video clips 808 and gesture detections 810. In some embodiments, video clips 808 may correspond to trackable frames generated by auto-editing agent 120. In some embodiments, gesture detections 810 may correspond to gesture data generated by gesture detector 212. As shown, only video clips 808 may be provided as input to first set of operations 804; both video clips 804 and gesture detections 810 may be provided as input to second set of operations 806.

[0121] The first group of operations 804 may include semantic segmentation 812, key frame matching 814, and STN fitting 816. At semantic segmentation 812, the field segmenter 216 may be configured to identify the field markings in the broadcast feed. In certain embodiments, the field segmenter 216 may be configured to utilize a neural network to identify the field markings. Such segmentation information may be performed in advance and provided to the camera calibration 124 from the database 215. At key frame matching 814, the key frame matching module 224 may be configured to match one or more frames of the field image with one or more templates. At STN fitting 816, STN 226 may implement a neural network to fit the field model to the field segmentation information. By fitting the field model to such an output, STN 224 may generate a homography matrix for each key frame.

[0122] The second set of operations 806 may include camera motion estimation 818. At camera flow estimation 818, the optical flow module 226 may be configured to identify motion patterns of objects from one trackable frame to another. For example, the optical flow module 226 may use body pose information to remove athletes from the trackable frame information. Once removed, the optical flow module 226 may determine inter-frame motion to identify camera motion between consecutive frames.

[0123] The first set of operations 804 and the second set of operations 806 may result in homography interpolation 816. The optical flow module 226 and the STN 224 may work together to generate a homography matrix for each trackable frame so that the camera can be calibrated for each frame. The homography matrix may be used to project the track or position of the player into real-world coordinates.

[0124] Fig. 9 900 is a flowchart illustrating a method 900 for tracking an athlete according to an example embodiment. The method 900 may correspond to the method described above in conjunction with Figure 4 Operation 410 is discussed. Method 900 may begin at step 902 .

[0125] At step 902, the organization computing system 104 may retrieve a plurality of trackable frames for matching. Each of the plurality of trackable frames may include one or more metadata sets associated therewith. Such metadata may include, for example, body posture information and camera calibration data. In some embodiments, the athlete tracking agent 126 may also retrieve broadcast video data.

[0126] At step 904, the organizational computing system 104 may generate a set of short traces. For example, the player tracking agent 126 may match player patch pairs derived from the pose information based on appearance and distance to generate a set of short traces. For example, Let be the athlete patch of the jth athlete at time t. is the image coordinates of the jth athlete at time t width and height Thus, the player tracking agent 126 can use appearance correlation and Associate any detection pair as follows:

[0127]

[0128] Doing this for each pair generates a set of short traces. The endpoints of these traces can then be correlated with each other based on motion consistency and color histogram similarity.

[0129] For example, let v i is the extrapolated velocity at the end of the ith trace, let v j is the extrapolated velocity at the beginning of the jth trace. Then c ij =v i ·v j It can represent the motion consistency score. In addition, let p(h) i represents the likelihood of color, h, existing in image patch i. The player tracking agent 126 may determine the color histogram similarity using the Bhattacharyya distance:

[0130] D B (p i ,p j )=-ln(BC(p i ,p j )),in

[0131] At step 906, the organization computing system 104 may connect the gaps between each set of short traces. For example, to reiterate, the tracking agent 120 finds matching trace pairs as follows:

[0132]

[0133] Solving each pair of interrupted traces can generate a net trace set while leaving some traces with large gaps (i.e., many frames). To bridge the large gaps, the athlete tracking agent 126 can augment the proximity metric to include a sports field estimate, which can account for changes in the athlete's direction that occur over many frames.

[0134] The sports field can be a vector field that determines the direction an athlete will be at a certain point on the playing field x after a period of time λ. For example, let be the position of athlete i on the playing field at each time t. Then there may be a pair of points with displacement if λ < T. Accordingly, the sports field can be as follows:

[0135]

[0136] where G(x, 5) can be a Gaussian kernel with a standard deviation approximately equal to five feet. In other words, the sports field can be a Gaussian blur of all displacements.

[0137] In step 908, the organizing computing system 104 can predict the movement of the player based on the sports field. For example, the athlete tracking system 126 can use a neural network (e.g., neural network 232) to predict the athlete's trajectory given the ground truth athlete trajectory. Given a set of true athlete trajectories X i , the athlete tracking agent 126 can be configured to generate a set where can be the predicted sports field. The athlete tracking agent 126 can train the neural network 232 to reduce (e.g., minimize) The athlete tracking agent 126 can then generate proximity scores for any tracking gap size λ as follows:

[0138] K ij = V(x, λ)·d ij

[0139] where is the displacement vector between all interrupted traces with a gap size of λ. Accordingly, the athlete tracking agent 126 can solve the above matching pairs. For example, given the proximity scores, the athlete tracking agent 126 can use the Hungarian Algorithm to assign each pair of interrupted traces. The Hungarian Algorithm (e.g., Kuhn - Munkres) can optimize the best matching set under the constraint that each pair has to be matched.

[0140] At step 910, organizational computing system 104 may output a graphical representation of the predictions. For example, interface agent 128 may be configured to generate one or more graphical representations corresponding to the tracks of each athlete generated by athlete tracking agent 126. For example, interface agent 128 may be configured to generate one or more graphical user interfaces (GUIs) that include a graphical representation of each predicted athlete tracked generated by athlete tracking agent 126.

[0141] In some cases, during competition, athletes or players tend to wander outside the camera's field of view. Such problems may occur during injuries, athlete fatigue, rapid turnovers, rapid offense-defense transitions, etc. Accordingly, an athlete in a first trackable frame may no longer be in a subsequent second or third trackable frame. The athlete tracking agent 126 can address this problem via the re-identification agent 234.

[0142] Fig.10 1 is a flow chart showing a method 1000 for tracking an athlete according to an example embodiment. The method 1000 may correspond to the method described above in conjunction with Figure 4 Operation 410 is discussed. Method 1000 may begin at step 1002 .

[0143] At step 1002, the organization computing system 104 may retrieve a plurality of trackable frames for matching. Each of the plurality of trackable frames may include one or more metadata sets associated therewith. Such metadata may include, for example, body posture information and camera calibration data. In some embodiments, the athlete tracking agent 126 may also retrieve broadcast video data.

[0144] In step 1004, the organization computing system 104 can identify a subset of short traces in which the athlete has left the camera's line of sight. Each trace can include multiple image patches associated with at least one athlete. An image patch can refer to a corresponding frame subset of multiple trackable frames. In some embodiments, each trace X can include an athlete identity tag y. In some embodiments, each athlete patch I in a given trace X can include gesture information generated by a data set generator 122. For example, in view of the input video, gesture detection, and trackable frames, the re-identification agent 234 can generate a trace set including several short interrupted traces of the athlete.

[0145] At step 1006, the organizational computing system 104 may generate a gallery for each trace. For example, given those small traces, the re-identification agent 234 may build a gallery for each trace. The re-identification agent 234 may build a gallery for each trace where the player's uniform number (or some other static feature) is always visible. The body posture information generated by the data set generator 122 allows the re-identification agent 234 to determine the orientation of each player. For example, the re-identification agent 234 may utilize a heuristic that may use the normalized shoulder width to determine orientation as follows:

[0146]

[0147] Where l can represent the position of a body part. The shoulder width can be normalized by the torso length to eliminate the scaling effect. The shoulders should be separated when the athlete is facing the camera or facing away from the camera, so the re-identification agent 234 can use those S orient Patches larger than the threshold are used to build a gallery. n The following galleries can be included:

[0148]

[0149] At step 1008, the organizational computing system 104 can use a convolutional autoencoder to match the traces. For example, the re-identification agent 234 can use a conditional autoencoder (e.g., the conditional autoencoder 240) to identify one or more features in each trace. For example, unlike conventional approaches to re-identification problems, players in team sports can have very similar appearance features, such as clothing style, clothing color, and skin color. One of the more intuitive differences may be the team uniform number that may be displayed on the front and / or back of each team uniform. To capture these specific features, the re-identification agent 234 can train a conditional autoencoder to recognize such features.

[0150] In some embodiments, the conditional autoencoder can be a three-layer convolutional autoencoder, where the kernel size of all three layers can be 3x3, with 64, 128, and 128 channels, respectively. These hyperparameters can be tuned to ensure that the team uniform numbers can be identified from the reconstructed image so that the expected features can be learned in the autoencoder. In some embodiments, f(I i ) can be used to represent the features learned from image i.

[0151] Using a specific example, the re-identification agent 234 may identify a first track corresponding to a first player. Using the conditional autoencoder 240, the re-identification agent 234 may learn a first set of uniform features associated with the first track, for example, based on a first set of image patches that include or are associated with the first track. The re-identification agent 234 may also identify a second track that may initially correspond to a second player. Using the conditional autoencoder 240, the re-identification agent 234 may learn a second set of uniform features associated with the second track, for example, based on a second set of image patches that include or are associated with the second track.

[0152] At step 1010, the organization computing system 104 may use a Siamese network to determine the similarity between the matching traces. For example, the re-identification agent 234 may train a Siamese network (e.g., the Siamese network 242) to determine the similarity between two image patches based on their feature representations f(I). Given two image patches, their feature representations f(I i ) and f(I i ) can be flattened, concatenated, and fed into the perception network. In some embodiments, the L2 norm can be used to connect the two sub-networks f(I i ) and f(I j ). In some embodiments, the perception network may include three layers, including 1024, 512, and 216 hidden units, respectively. Such a network can be used to determine the similarity s(I) between each pair of image patches of two traces without temporal overlap. i ,I j ). To enhance the robustness of the prediction, the final similarity score of two traces can be the average of all pairwise scores in their respective galleries:

[0153]

[0154] Continuing with the above example, the re-identification agent 234 may utilize the Siamese network 242 to calculate a similarity score between the first learned uniform feature set and the second learned uniform feature set.

[0155] At step 1012, if the similarity scores of the traces are above a predetermined threshold, the organization computing system 104 may associate the traces. For example, the re-identification agent 234 may calculate a similarity score for each two traces that do not overlap in time. If the score is above a certain threshold, the re-identification agent 234 may associate the two traces.

[0156] Continuing with the above example, the re-identification agent 234 may associate the first trace with the second trace if, for example, the similarity score generated by the Siamese network 242 is at least above a threshold. Assuming the similarity score is above the threshold, the re-identification agent 234 may determine that the first player in the first trace and the second player in the second trace are indeed the same person.

[0157] Fig.11 is a block diagram 1100 illustrating operational aspects discussed above in connection with method 1000 according to an example embodiment.

[0158] As shown, block diagram 1100 may include input video 1102, gesture detection 1104, player tracking 1106, trail collection 1108, gallery creation and pairing 1110, and trail connection 1112. Block diagram 1100 illustrates the general pipeline of method 1000 provided above.

[0159] Given the input video 1102, gesture detection information 1104 (e.g., generated by gesture detector 212), and player tracking information 1106 (e.g., generated by one or more of player tracking agent 126, automatic editing agent 120, and camera calibrator 124), the re-identification agent 234 can generate a set of tracks 1108. Each set of tracks 1108 can include multiple short interrupted tracks of a player (e.g., tracks 1114). Each track 1114 can include one or more image patches 1116 contained therein. Given the tracks 1114, the re-identification agent 234 can generate a gallery 1110 for each track. For example, the gallery 1110 can include those image patches 1118 in a given track that include images of players whose orientations meet a threshold. In other words, the re-identification agent 234 can generate a gallery 1110 for each track 1114 including an image patch 1118 for each player so that the player's number can be visible in each frame. Image patch 1118 may be a subset of image patch 1116. The re-recognition agent 234 may re-pair each frame to calculate a similarity score via a Siamese network. For example, as shown, the re-recognition agent 234 may match the first frame from trace II with the second frame from trace I and feed the frames to the Siamese network.

[0160] The re-recognition agent 234 can then connect the traces 1112 based on the similarity scores. For example, if the similarity scores of two frames exceed a certain threshold, the re-recognition agent 234 can connect or associate those traces.

[0161] Fig.121 is a block diagram illustrating an architecture 1200 of a Siamese network 242 of a re-identification agent 234 according to an example embodiment. As shown, the Siamese network 242 may include two sub-networks 1202, 1204 and one perception network 1205.

[0162] Each of the two sub-networks 1202, 1204 can be configured similarly. For example, sub-network 1202 can include a first convolutional layer 1206, a second convolutional layer 1208, and a third convolutional layer 1210. The first sub-network 1202 can receive the player patch I1 as input and output a feature set learned from the player patch I1 (denoted as f(I1)). Sub-network 1204 can include a first convolutional layer 1216, a second convolutional layer 1218, and a third convolutional layer 1220. The second sub-network 1204 can receive the player patch I2 as input and can output a feature set learned from the player patch I2 (denoted as f(I2)). The outputs from sub-network 1202 and sub-network 1204 can be encoded representations of the corresponding player patches I1, I2. In some embodiments, the outputs from sub-network 1202 and sub-network 1204 can be followed by a flattening operation, which can generate respective feature vectors f(I1) and f(I2), respectively. In some embodiments, each feature vector f(I1) and f(I2) may include 10240 units. In some embodiments, the L2 norm of f(I1) and f(I2) may be calculated and used as input to the perception network 1205.

[0163] Perception network 1205 may include three layers 1222 to 1226. In some embodiments, layer 1222 may include 1024 hidden units. In some embodiments, layer 1224 may include 512 hidden units. In some embodiments, layer 1226 may include 256 hidden units. Perception network 1205 may output a similarity score between image patches I1 and I2.

[0164] Fig.13AA system bus of a computing system architecture 1300 according to an example embodiment is shown. The system 1300 may represent at least a portion of an organization computing system 104. One or more components of the system 1300 may communicate electrically with each other using a bus 1305. The system 1300 may include a processing unit (CPU or processor) 1310 and a system bus 1305, which couples various system components including a system memory 1315 (such as a read-only memory (ROM) 1320 and a random access memory (RAM) 1325) to the processor 1310. The system 1300 may include a cache of high-speed memory that is directly connected, adjacent to the processor 1310, or integrated as part of the processor 1310. The system 1300 may copy data from the memory 1315 and / or the storage device 1330 to the cache 1312 for quick access by the processor 1310. In this way, the cache 1312 may provide a performance improvement that avoids delays incurred by the processor 1310 while waiting for data. These and other modules may control or be configured to control the processor 1310 to perform various actions. Other system memories 1315 may also be used. Memory 1315 may include a variety of different types of memory with different performance characteristics. Processor 1310 may include any general purpose processor and hardware or software modules such as Service I 1332, Service II 1334, and Service III 1336 (configured to control processor 1310) stored in storage device 1330, as well as special purpose processors, where the software instructions are incorporated into the actual processor design. Processor 1310 may essentially be a completely independent computing system, including multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0165] In order to enable a user to interact with the computing device 1300, the input device 1345 can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphic input, a keyboard, a mouse, motion input, voice, etc. The output device 1335 can also be one or more of a variety of output mechanisms known to those skilled in the art. In some cases, a multimodal system can enable a user to provide multiple types of input to communicate with the computing device 1300. The communication interface 1340 can generally control and manage user input and system output. There is no restriction on the operation on any particular hardware arrangement, so the basic features of the present disclosure can be easily replaced with improved hardware or firmware arrangements still under development.

[0166] The storage device 1330 may be a non-volatile memory or a hard disk or other type of computer-readable medium that can store data accessible to a computer, such as a magnetic tape, a flash memory card, a solid-state storage device, a digital versatile disk, a cassette tape, a random access memory (RAM) 1325, a read-only memory (ROM) 1320, and a mixture thereof.

[0167] The storage device 1330 may include services 1332, 1334, and 1336 for controlling the processor 1310. Other hardware or software modules are also contemplated. The storage device 1330 may be connected to the system bus 1305. In one aspect, a hardware module that performs a particular function may include a software component stored in a computer-readable medium that is associated with the necessary hardware components to perform the function, such as the processor 1310, the bus 1305, the display 1335, etc.

[0168] Fig. 13B A computer system 1350 having a chipset architecture that can represent at least a portion of an organizational computing system 104 is shown. Computer system 1350 can be an example of computer hardware, software, and firmware that can be used to implement the disclosed technology. System 1350 may include a processor 1355, which represents any number of physically and / or logically different resources that can execute software, firmware, and hardware configured to perform the identified calculations. Processor 1355 can communicate with chipset 1360, which can control the input and output of processor 1355. In this example, chipset 1360 outputs information to output 1365, such as a display, and can read and write information to storage device 1370, which can, for example, include magnetic media and solid-state media. Chipset 1360 can also read data from RAM 1375 and write data to RAM 1375. A bridge 1380 for docking with various user interface components 1385 can be provided to dock with chipset 1360. Such user interface components 1385 may include a keyboard, a microphone, touch detection processing circuitry, a pointing device such as a mouse, etc. In general, input to the system 1350 may come from any of a variety of sources, whether machine-generated and / or human-generated.

[0169] Chipset 1360 may also interface with one or more communication interfaces 1390 having different physical interfaces. Such communication interfaces may include interfaces for wired and wireless local area networks, broadband wireless networks, and personal area networks. Some applications of the methods for generating, displaying, and using the GUI disclosed herein may include receiving an ordered data set through a physical interface or being generated by the machine itself by analyzing data stored in memory 1370 or 1375 through processor 1355. In addition, the machine may receive input from a user through user interface component 1385 and perform appropriate functions, such as browsing functions, by using processor 1355 to interpret these inputs.

[0170] It can be appreciated that the example systems 1300 and 1350 may have more than one processor 1310 or be part of a group or cluster of computing devices networked together to provide greater processing capabilities.

[0171] Although the above is directed to the embodiments described in the present disclosure, other and more embodiments may be designed without departing from the basic scope of the present disclosure. For example, various aspects of the present disclosure may be implemented in hardware or software or a combination of hardware and software. An embodiment described in the present disclosure may be implemented as a program product for use with a computer system. The program of the program product defines the functions of the embodiment (including the method described in the present disclosure) and may be included on various computer-readable storage media. Illustrative computer-readable storage media include, but are not limited to: (i) non-writable storage media on which information is permanently stored (e.g., a read-only memory (ROM) device in a computer, such as a CD-ROM disk, flash memory, ROM chip, or any type of solid-state non-volatile memory that can be read by a CD-ROM drive); (ii) writable storage media on which information can be changed (e.g., a floppy disk in a floppy disk drive or hard disk drive or any type of solid-state random access memory). Such a computer-readable storage medium becomes an embodiment of the present disclosure when it carries computer-readable instructions that guide the functions of an embodiment of the present disclosure.

[0172] Those skilled in the art will appreciate that the above examples are exemplary only and not restrictive. After reading this specification and studying the drawings, those skilled in the art will understand that all substitutions, enhancements, equivalents, and improvements are intended to fall within the true spirit and scope of protection of the present disclosure. Therefore, the appended claims are intended to include all such modifications, substitutions, and equivalents that fall within the true spirit and scope of protection taught by the present disclosure.

Claims

1. A method comprising: Receiving historical sporting event data by a computing system; receiving, by the computing system, a plurality of trackable frames of a sporting event and corresponding trackable frame data, wherein the plurality of trackable frames correspond to a unified view of the sporting event; identifying, by the computing system, that the plurality of trackable frames includes a subset of frames of a first player; identifying, by the computing system, one or more player tracks for the first player in the subset of frames; predicting, by the computing system, a predicted trajectory for the first athlete based on the historical sporting event data and one or more athlete trajectories for the first athlete in the subset of frames; as well as A visual representation of the predicted trajectory of the first athlete is displayed by the computing system on one or more user interfaces.

2. The method according to claim 1, further comprising: providing one or more player trajectories of the first player in the subset of frames as additional training data to a machine learning model trained to output player trajectories based on the historical sporting event data; and After providing one or more player trajectories of the first player in the subset of frames as additional training data, a predicted trajectory of the first player is received from the machine learning model.

3. The method according to claim 1, further comprising: Body posture data for one or more athletes, including the first athlete, is generated by the computing system.

4. The method according to claim 3, further comprising: Based on the body pose data, second player information is removed, by the computing system, from at least one of the plurality of trackable frames.

5. The method according to claim 4, further comprising: identifying, by the computing system, a flow field between two or more consecutive trackable frames; generating, by the computing system, a homography matrix for each of the plurality of trackable frames based on the flow field; as well as Based on the homography matrix, a camera is calibrated by the computing system.

6. The method of claim 1, wherein the trained neural network is configured to analyze a plurality of frames to determine the plurality of trackable frames.

7. The method according to claim 1, further comprising: receiving, via a neural network, a set of ground truth athlete trajectories; as well as The one or more player trajectories are predicted via the neural network based on a set of ground truth player trajectories.

8. The method of claim 7, wherein predicting the one or more player trajectories based on a set of ground truth player trajectories further comprises: generating, via the neural network, a motion field vector for the first player in the plurality of trackable frames, wherein the motion field vector represents a magnitude and direction of velocity of the first player on a playing field; as well as The one or more player trajectories are predicted based on the motion field vectors via the neural network.

9. A computing system for predicting athlete trajectories, the computing system comprising: a memory having processor-readable instructions stored therein; as well as One or more processors configured to access the memory and execute the processor-readable instructions, the processor-readable instructions, when executed by the one or more processors, configuring the one or more processors to perform a plurality of functions, the functions comprising: Receive historical sports event data; receiving a plurality of trackable frames of a sporting event and corresponding trackable frame data, wherein the plurality of trackable frames correspond to a unified view of the sporting event; identifying a subset of frames of the plurality of trackable frames including a first player; identifying one or more player tracks for the first player in the subset of frames; predicting a predicted trajectory for the first player based on the historical sporting event data and one or more player trajectories for the first player in the subset of frames; and A visual representation of the predicted trajectory of the first athlete is displayed on one or more user interfaces.

10. The computing system of claim 9, further comprising: Body posture data is generated for one or more athletes including the first athlete.

11. The computing system of claim 10, further comprising: Based on the body pose data, second player information is removed from at least one of the plurality of trackable frames.

12. The computing system of claim 11, further comprising: identifying a flow field between two or more consecutive trackable frames; generating a homography matrix for each of the plurality of trackable frames based on the flow field; as well as Based on the homography matrix, the camera is calibrated.

13. The computing system of claim 9, wherein the trained neural network is configured to analyze a plurality of frames to determine the plurality of trackable frames.

14. The computing system of claim 9, further comprising: receiving, via a neural network, a set of ground truth athlete trajectories; as well as The one or more player trajectories are predicted via the neural network based on a set of ground truth player trajectories.

15. The computing system of claim 9, further comprising: providing one or more player trajectories of the first player in the subset of frames as additional training data to a machine learning model trained to output player trajectories based on the historical sporting event data; and After providing one or more player trajectories of the first player in the subset of frames as additional training data, a predicted trajectory of the first player is received from the machine learning model.

16. A non-transitory computer-readable medium comprising instructions for generating athlete tracking data, the instructions comprising: Receiving historical sporting event data by a computing system; receiving, by the computing system, a plurality of trackable frames of a sporting event and corresponding trackable frame data, wherein the plurality of trackable frames correspond to a unified view of the sporting event; identifying, by the computing system, that the plurality of trackable frames includes a subset of frames of a first player; identifying, by the computing system, one or more player tracks for the first player in the subset of frames; predicting, by the computing system, a predicted trajectory for the first athlete based on the historical sporting event data and one or more athlete trajectories for the first athlete in the subset of frames; as well as A visual representation of the predicted trajectory of the first athlete is displayed by the computing system on one or more user interfaces.

17. The non-transitory computer readable medium of claim 16, the instructions further comprising: providing one or more player trajectories of the first player in the subset of frames as additional training data to a machine learning model trained to output player trajectories based on the historical sporting event data; and After providing one or more player trajectories of the first player in the subset of frames as additional training data, a predicted trajectory of the first player is received from the machine learning model.

18. The non-transitory computer-readable medium of claim 16, wherein the trained neural network is configured to analyze a plurality of frames to determine the plurality of trackable frames.

19. The non-transitory computer readable medium of claim 16, the instructions further comprising: receiving, via a neural network, a set of ground truth athlete trajectories; as well as The one or more player trajectories are predicted via the neural network based on a set of ground truth player trajectories.

20. The non-transitory computer readable medium of claim 19, wherein predicting the one or more player trajectories based on a set of ground truth player trajectories further comprises: generating, via the neural network, a motion field vector for the first player in the plurality of trackable frames, wherein the motion field vector represents a magnitude and direction of velocity of the first player on a playing field; as well as The one or more player trajectories are predicted based on the motion field vectors via the neural network.

Images