System and method for generating athlete tracking data from broadcast video

Abstract

The present disclosure describes a system and method of generating athlete tracking predictions. A computing system retrieves a broadcast video feed for a sporting event. A computing system segments a broadcast video feed into a unified view. The computing system generates a plurality of data sets based on the plurality of trackable frames. The computing system calibrates a camera associated with each trackable frame based on the body posture information. The computing system generates a plurality of short trace tile sets based on the plurality of traceable frames and the body posture information. The computing system connects each short trace tile set by generating a playground vector for each athlete in the plurality of traceable frames. The computing system predicts a future movement of the athlete based on the athlete's playground vector using a neural network.

Description

[0001] (Division of 202080017536.8)

[0002] Priority Document

[0003] This application claims priority to U.S. Provisional Application No. 62 / 811,889, filed on February 28, 2019, the entire contents of which are incorporated herein by reference. Technical Field

[0004] The present disclosure generally relates to systems and methods for generating player tracking data from broadcast video. Background Art

[0005] Athlete tracking data has been used for team and player analysis in many sports for many years. However, conventional athlete tracking systems require sports analytics companies to install fixed cameras at each venue where a team competes. This constraint limits the scalability of athlete tracking systems and data collection for current competitions. In addition, such constraints can incur significant expenses for sports analytics companies due to the costs associated with installing hardware at the necessary venues and maintaining such hardware. Summary of the invention

[0006] In some embodiments, a method for generating a player tracking prediction is disclosed herein. A computing system retrieves a broadcast video feed of a sporting event. The broadcast video feed includes a plurality of video frames. The computing system segments the broadcast video feed into a unified view. The unified view includes a plurality of trackable frames. The plurality of trackable frames are subsets of the plurality of video frames. The computing system generates a plurality of data sets based on the plurality of trackable frames. The plurality of data sets include field segmentation information, ball tracking information, and body pose information for each player in each trackable frame. The computing system calibrates a camera associated with each trackable frame based on the field segmentation information and the body pose information. The computing system generates a plurality of short track set based on the plurality of trackable frames and the body pose information. The computing system connects each short track set by generating a motion field vector for each player in the plurality of trackable frames. The computing system uses a neural network to predict the future movement of the player based on the motion field vector of the player.

[0007] In some embodiments, a system for generating player tracking predictions is disclosed herein. The computing system includes a processor and a memory. The memory has programming instructions stored thereon that, when executed by the processor, perform one or more operations. One or more operations include retrieving a broadcast video feed of a sporting event. The broadcast video 202080017536.8 Division PX20240433|00166-0064-00139 feed includes multiple video frames. One or more operations also include segmenting the broadcast video feed into a unified view. The unified view includes multiple trackable frames, which are subsets of the multiple video frames. One or more operations also include generating multiple data sets based on the multiple trackable frames. The multiple data sets include field segmentation information, ball tracking information, and body posture information for each player in each trackable frame. One or more operations also include calibrating a camera associated with each trackable frame based on the field segmentation information and the body posture information. One or more operations also include generating multiple short track set based on the multiple trackable frames and the body posture information. The one or more operations also include connecting each set of short tracklets by generating a motion field vector for each player in the plurality of trackable frames. The one or more operations also include using a neural network to predict future motion of the player based on the player's motion field vector.

[0008] In some embodiments, a non-transitory computer-readable medium is disclosed herein. The non-transitory computer-readable medium includes one or more instruction sequences that, when executed by one or more processors, perform one or more operations. One or more operations include retrieving a broadcast video feed of a sporting event. The broadcast video feed includes a plurality of video frames. One or more operations also include segmenting the broadcast video feed into a unified view. The unified view includes a plurality of trackable frames, which are subsets of a plurality of video frames. One or more operations also include generating a plurality of data sets based on a plurality of trackable frames. The plurality of data sets include field segmentation information, ball tracking information, and body posture information for each athlete in each trackable frame. One or more operations also include calibrating a camera associated with each trackable frame based on the field segmentation information and the body posture information. One or more operations also include generating a plurality of short track sets based on a plurality of trackable frames and the body posture information. One or more operations also include connecting each short track set by generating a motion field vector for each athlete in a plurality of trackable frames. One or more operations also include predicting the future movement of an athlete based on the athlete's motion field vector using a neural network. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to understand the above features of the present disclosure in detail, the present disclosure briefly summarized above can be described in more detail by referring to the embodiments, some of which are shown in the accompanying drawings. However, it should be noted that the accompanying drawings only show typical embodiments of the present disclosure and should not be considered to limit the scope of the present disclosure, and the present disclosure may allow other equivalent embodiments.

[0010] Figure 1 is a block diagram illustrating a computing environment according to an example embodiment.

[0011] Figure 2 is a block diagram illustrating a computing environment according to an example embodiment.

[0013] Figure 3 is to illustrate the context according to an example embodiment in conjunction with Figure 2 and Figures 4 to 10 A block diagram of the operational aspects of the discussion.

[0014] Figure 4 is a flow chart illustrating a method of generating a player trail according to an example embodiment.

[0015] Figure 5 is a flow chart illustrating a method of generating a trackable frame according to an example embodiment.

[0016] Figure 6 It is shown that according to an example embodiment, the above combination Figure 5 A block diagram of the operational aspects of the discussion.

[0017] Figure 7 is a flow chart illustrating a method of calibrating a camera for each trackable frame according to an example embodiment.

[0018] Figure 8 It is shown that according to an example embodiment, the above combination Figure 7 A block diagram of the operational aspects of the discussion.

[0019] Fig. 9 is a flow chart illustrating a method of tracking a player according to an example embodiment.

[0020] Fig.10 is a flow chart illustrating a method of tracking a player according to an example embodiment.

[0021] Fig.11 It is shown that according to an example embodiment, the above combination Fig.10 A block diagram of the operational aspects of the discussion.

[0022] Fig.12 is a block diagram illustrating the architecture of a re-identification agent according to an example embodiment.

[0023] Fig.13A is a block diagram illustrating a computing device according to an example embodiment.

[0024] Fig. 13B is a block diagram illustrating a computing device according to an example embodiment.

[0025] To facilitate understanding, like reference numerals have been used, where appropriate, to designate like elements that are common to the figures. It is contemplated that elements disclosed in one embodiment may be suitably utilized in other embodiments without specific recitation. DETAILED DESCRIPTION

[0026] Athlete tracking data has always been a valuable resource for leagues and teams to evaluate not only the team itself, but also the athletes on the team. Conventional methods for collecting or generating athlete tracking data are limited and rely on installing fixed cameras at the venue where the sporting event will be held. In other words, conventional methods for teams to collect or generate athlete tracking data require the team to equip each venue with a fixed camera system. Those skilled in the art recognize that this constraint severely limits the scalability of the athlete tracking system. In addition, this constraint also limits the athlete tracking data to games played after the fixed camera system was installed, as historical athlete tracking data is simply not available.

[0027] One or more techniques described in this disclosure provide a significant improvement over conventional systems because a fixed camera system is no longer required. Specifically, one or more techniques described in this disclosure are directed to utilizing broadcast video feeds of sporting events to generate athlete tracking data. By utilizing broadcast video feeds of sporting events, not only is it no longer necessary to use a dedicated fixed camera system in each venue, but historical athlete tracking data can now be generated from historical sporting events.

[0028] However, utilizing broadcast video feeds of sporting events is not an easy task. For example, included in the broadcast video feed may be a variety of different camera angles, close-ups of athletes, close-ups of the crowd, close-ups of coaches, commentator videos, advertisements, halftime shows, etc. In addition, due to the pace of sporting events, it is inevitable that some athletes will be obscured from the broadcast video feed camera or completely out of the broadcast video feed camera's view.

[0029] In light of this, one or more techniques described in this disclosure provide various mechanisms aimed at overcoming the above-mentioned challenges so that broadcast video feeds of sporting events can be used to generate player tracking data.

[0030] Figure 1 1 is a block diagram illustrating a computing environment 100 according to an example embodiment. The computing environment 100 may include a camera system 102 , an organizational computing system 104 , and one or more client devices 108 in communication via a network 105 .

[0031] Network 105 can take any suitable type, including via the Internet, a separate connection such as a cellular or Wi-Fi network. In some embodiments, network 105 can use a wireless network such as radio frequency identification (RFID), near field communication (NFC), Bluetooth, etc. TM , Bluetooth Low Energy TM (BLE), Wi-Fi TM 、ZigBee TM , ambient backscatter communication (ABC) protocol, USB, WAN, or LAN to connect terminals, services, and mobile devices. Because the information transmitted may be personal or confidential, security issues may require that one or more of these types of connections be encrypted or otherwise secured. However, in some embodiments, the information transmitted may not be very personal, so the network connection may be selected for convenience rather than security.

[0032] The network 105 may include any type of computer network arrangement for exchanging data or information. For example, the network 105 may be the Internet, a dedicated data network, a virtual private network using a public network, and / or other suitable connection that enables components in the computing environment 100 to send and receive information between components of the environment 100.

[0033] The camera system 102 can be positioned in the field 106. For example, the field 106 can be configured to host a sporting event including one or more players 112. The camera system 102 can be configured to capture the movement of all players (i.e., athletes) on the playing surface and one or more other related objects (e.g., a ball, a referee, etc.). In some embodiments, the camera system 102 can be an optical-based system, for example, using multiple fixed cameras. For example, a system consisting of six stationary calibrated cameras can be used, which project the three-dimensional positions of the athletes and the ball onto a two-dimensional top view of the court. In another example, a mixture of stationary cameras and non-stationary cameras can be used to capture the movement of all players on the court and one or more related objects. Those skilled in the art will recognize that the use of such a camera system (e.g., camera system 102) can generate many different court camera views (e.g., high sideline views, free throw line views, crowd views, scrum views, goal area views, etc.). Generally speaking, camera system 102 may be used for a broadcast feed of a given competition. Each frame in the broadcast feed may be stored in a competition file 110.

[0034] The camera system 102 can be configured to communicate with the organization computing system 104 via the network 105. The organization computing system 104 can be configured to manage and analyze the broadcast feed captured by the camera system 102. The organization computing system 104 can include at least a Web client application server 114, a data store 118, an automatic editing agent 120, a data set generator 122, a camera calibrator 124, an athlete tracking agent 126, and an interface agent 128. Each of the automatic editing agent 120, the data set generator 122, the camera calibrator 124, the athlete tracking agent 126, and the interface agent 128 can be composed of one or more software modules. One or more software modules can be a collection of codes or instructions stored on a medium (e.g., a memory of the organization computing system 104), which represent a series of machine instructions (e.g., program code) that implement one or more algorithmic steps. Such machine instructions can be actual computer codes that the processor of the organization computing system 104 interprets to implement instructions, or can be high-level encodings of instructions that interpret to obtain actual computer codes. One or more software modules can also include one or more hardware components. One or more aspects of the example algorithms may be performed by hardware components (eg, circuits) themselves, rather than as a result of instructions.

[0035] The data storage 118 may be configured to store one or more game files 124. Each game file 124 may include broadcast data for a given competition. For example, the broadcast data may be a plurality of video frames captured by the camera system 102.

[0036] The automatic editing agent 120 can be configured to parse the broadcast feed of a given competition to identify a unified view of the competition. In other words, the automatic editing agent 120 can be configured to parse the broadcast feed to identify all information frames captured from the same view. In one example, in basketball, for example, the unified view can be a high sideline view. The automatic editing agent 120 can clip or split the broadcast feed (e.g., video) into its components (e.g., different scenes in a movie, advertisements in a competition, etc.). In order to generate a unified view, the automatic editing agent 120 can identify those parts that capture the same view (e.g., a high sideline view). Accordingly, the automatic editing agent 120 can remove all (or part) of the untrackable parts of the broadcast feed (e.g., close-ups of athletes, advertisements, halftime performances, etc.). The unified view can be stored in a database as a trackable frame set.

[0037] The data set generator 122 may be configured to generate multiple data sets based on the trackable frame. In some embodiments, the data set generator 122 may be configured to identify body posture information. For example, the data set generator 122 may use body posture information to detect athletes in the trackable frame. In some embodiments, the data set generator 122 may be configured to further track the movement of the ball or puck in the trackable frame. In some embodiments, the data set generator 122 may be configured to segment the field where the event is occurring to identify one or more markers of the field. For example, the data set generator 122 may be configured to identify court (e.g., basketball, tennis, etc.) markers, sports field (e.g., baseball, American football, soccer, rugby, etc.) markers, ice rink (e.g., hockey) markers, etc. The multiple data sets generated by the data set generator 122 may then be used by the camera calibrator 124 to calibrate the camera of each camera system 102.

[0038] The camera calibrator 124 may be configured to calibrate the cameras of the camera system 102. For example, the camera calibrator 124 may be configured to project the players detected in the trackable frame to real-world coordinates for further analysis. The cameras in the camera system 102 are constantly moving to focus on the ball or key actions, so such cameras cannot be pre-calibrated. The camera calibrator 124 may be configured to improve or optimize the player projection parameters using the homography matrix.

[0039] Athlete tracking agent 126 can be configured to generate a track for each athlete on the field. For example, athlete tracking agent 126 can utilize athlete posture detection, camera calibration and broadcast frame to generate such a track. In certain embodiments, athlete tracking agent 126 can also be configured to generate a track for each athlete, even if, for example, the athlete is currently outside the trackable frame. For example, athlete tracking agent 126 can utilize body posture information to link the athlete who has left the view frame.

[0040] Interface agent 128 may be configured to generate one or more graphical representations corresponding to the tracks of each athlete generated by athlete tracking agent 126. For example, interface agent 128 may be configured to generate one or more graphical user interfaces (GUIs) that include a graphical representation of each predicted athlete tracked generated by athlete tracking agent 126.

[0041] Client devices 108 can communicate with organizational computing system 104 via network 105. Client devices 108 can be operated by users. For example, client devices 108 can be mobile devices, tablet computers, desktop computers, or any computing system with the capabilities described herein. Users can include, but are not limited to, individuals such as subscribers, customers, potential customers, or customers of entities associated with organizational computing system 104, such as individuals who have obtained, will obtain, or can obtain products, services, or consulting from an entity associated with organizational computing system 104.

[0042] The client device 108 may include at least an application 132. The application 132 may represent a web browser that allows access to a website or a stand-alone application. The client device 108 may access the application 132 to access one or more functions of the organization computing system 104. The client device 108 may communicate over the network 105 to request a web page, such as a web page from a web client application server 114 of the organization computing system 104. For example, the client device 108 may be configured to execute the application 132 to access content managed by the web client application server 114. The content displayed to the client device 108 may be transmitted from the web client application server 114 to the client device 108 and then processed by the application 132 for display through a graphical user interface (GUI) of the client device 108.

[0043] Figure 2 2 is a block diagram illustrating a computing environment 200 according to an example embodiment. As shown, the computing environment 200 includes an automatic editing agent 120, a data set generator 122, a camera calibrator 124, and a player tracking agent 126 that communicate via a network 205.

[0044] Network 205 can take any suitable type, including via the Internet, a separate connection such as a cellular or Wi-Fi network. In some embodiments, network 205 can use a wireless network such as radio frequency identification (RFID), near field communication (NFC), Bluetooth, etc. TM , Bluetooth Low Energy TM (BLE), Wi-Fi TM 、ZigBee TM , ambient backscatter communication (ABC) protocol, USB, WAN, or LAN to connect terminals, services, and mobile devices. Because the information transmitted may be personal or confidential, security issues may require that one or more of these types of connections be encrypted or otherwise secured. However, in some embodiments, the information transmitted may not be very personal, so the network connection may be selected for convenience rather than security.

[0045] The network 205 may include any type of computer network arrangement for exchanging data or information. For example, the network 205 may be the Internet, a private data network, a virtual private network using a public network, and / or other suitable connection, enabling components in the computing environment 200 to send and receive information between components of the environment 200.

[0046] The automatic clipping agent 120 may include a principal component analysis (PCA) agent 202, a clustering model 204, and a neural network 206. As described above, the automatic clipping agent 120 may be used to clip or segment a video into its component parts when attempting to understand and extract data from a broadcast feed. In some embodiments, the automatic clipping agent 120 may focus on separating a predefined unified view (e.g., a high-edge view) from all other parts of the broadcast stream.

[0047] The PCA agent 202 can be configured to perform each extraction of frame features from the broadcast feed using PCA analysis. For example, given a pre-recorded video, the PCA agent 202 can extract one frame every X seconds (e.g., every 10 seconds) to build a PCA model of the video. In some embodiments, the PCA agent 202 can use incremental PCA to generate the PCA model, from which the PCA agent 202 can select the previous subset of components (e.g., the first 120 components) to generate the PCA model. The PCA agent 202 can also be configured to extract one frame from the broadcast stream every X seconds (e.g., every second) and compress the frames using the PCA model. In some embodiments, the PCA agent 202 can compress the frames into a 120-dimensional form using the PCA model. For example, the PCA agent 202 can solve the principal components on a per-video basis and save the first 100 components of each frame to ensure accurate editing.

[0048] The clustering model 204 can be configured to cluster the first subset of components into clusters. For example, the clustering model 204 can be configured to center, normalize, and cluster the first 120 components into multiple clusters. In some embodiments, for clustering of compressed frames, the clustering model 204 can implement K-means clustering. In some embodiments, the clustering model 204 can set k=9 clusters. K-means clustering attempts to obtain some data x={x1, x2, ..., x n} and divide it into k subsets, S={S1,S2,...S k}, optimized as follows:

[0049]

[0050] where μ j For set S jIn other words, clustering model 204 attempts to find the cluster with the smallest inter-cluster variance using the K-means clustering technique. Clustering model 204 may label each frame with its respective cluster number (eg, cluster 1, cluster 2, ..., cluster k).

[0051] The neural network 206 may be configured to classify each frame as trackable or untrackable. Trackable frames may represent frames that include a captured unified view (e.g., a high edge view). Untrackable frames may represent frames that do not capture a unified view. To train the neural network 206, an input dataset may be used that includes thousands of frames pre-labeled as trackable or untrackable that have been run through a PCA model. Each compressed frame and label pair (i.e., cluster number and trackable / untrackable) may be provided to the neural network 206 for training.

[0052] In some embodiments, the neural network 206 may include four layers. The four layers may include an input layer, two hidden layers, and an output layer. In some embodiments, the input layer may include 120 units. In some embodiments, each hidden layer may include 240 units. In some embodiments, the output layer may include 2 units. The input layer and each hidden layer may use a sigmoid activation function. The output layer may use a SoftMax activation function. To train the neural network 206, the automatic editing agent 120 may reduce (e.g., minimize) the predicted label of the sample as follows With the true label y j The binary cross entropy loss between:

[0053]

[0054] Accordingly, once trained, the neural network 206 can be configured to classify each frame as untrackable or trackable. In this way, each frame can have two labels: a cluster number and a trackable / untrackable classification. The automatic editing agent 120 can use these two labels to determine whether a given cluster is considered trackable or untrackable. For example, if the automatic editing agent 120 determines that a threshold number of frames in a cluster are considered trackable (e.g., 80%), the automatic editing agent 120 can conclude that all frames in the cluster are trackable. In addition, if the automatic editing agent 120 determines that less than a threshold number of frames in a cluster are considered untrackable (e.g., less than 30%), the automatic editing agent 120 can conclude that all frames in the cluster are untrackable. Further, if the automatic editing agent 120 determines that a specific number of frames in a cluster are considered trackable (e.g., 30% to 80%), the automatic editing agent 120 can request an administrator to further analyze the cluster. Once each frame is classified, the automatic editing agent 120 can clip or split the trackable frames. The auto-editing agent 120 may store the segments of trackable frames in a database 205 associated therewith.

[0055] Data set generator 122 can be configured to generate multiple data sets from automatic editing agent 120. As shown, data set generator 122 can include posture detector 212, ball detector 214 and field segmenter 216. Posture detector 212 can be configured to detect the athlete in the broadcast feed. Data set generator 122 can provide trackable frames stored in database 205 and broadcast video feed as input to posture detector 212. In some embodiments, posture detector 212 can implement open posture (Open Pose) to generate body posture data to detect the athlete in broadcast feed and trackable frame. In some embodiments, posture detector 212 can implement sensors positioned on the athlete to capture body posture information. Generally speaking, posture detector 212 can use any means to obtain body posture information from broadcast video feed and trackable frame. Output from posture detector 212 can be posture data stored in database 215 associated with data set generator 122.

[0056] Ball detector 214 may be configured to detect and track a ball (or puck) within a broadcast feed. Dataset generator 122 may provide trackable frames stored in database 205 and a broadcast video feed as input to ball detector 214. In some embodiments, ball detector 214 may utilize a Faster Regional Convolutional Neural Network (Faster R-CNN) to detect and track a ball in a trackable frame and a broadcast video feed.

[0057] Faster R-CNN is a region proposal network. Faster R-CNN uses a convolutional neural network to propose regions of interest and then classifies the objects in each region of interest. Because it is a single unified network, the region of interest and classification steps can improve each other, allowing classification to handle objects of various sizes. The output from the ball detector 214 can be ball detection data stored in the database 215 associated with the dataset generator 122.

[0058] The field segmenter 216 may be configured to identify field markers in the broadcast feed. The dataset generator 122 may provide the field segmenter 216 with the trackable frames stored in the database 205 and the broadcast video feed as input. In some embodiments, the field segmenter 216 may be configured to utilize a neural network to identify field markers. The output of the field segmenter 216 may be the field markers stored in the database 215 associated with the dataset generator 122.

[0059] The camera calibrator 124 can be configured to solve the problem of mobile camera calibration in sports. The camera calibrator 124 can include a spatial transmission network 224 and an optical flow module 226. The camera calibrator 124 can receive as input the segmented field information, trackable clip information, and pose information generated by the field segmenter 216. Given such input, the camera calibrator 124 can be configured to project the coordinates in the image frame to real-world coordinates for tracking analysis.

[0060] Keyframe matching module 224 can receive the output from field segmenter 216 and template set as input. For each frame, keyframe matching module 224 can match the output from field segmenter 216 with the template. Those frames that can match a given template are considered keyframes. In some embodiments, keyframe matching module 224 can implement a neural network to match one or more frames. In some embodiments, keyframe matching module 224 can implement cross-correlation to match one or more frames.

[0061] The spatial transformer network (STN) 224 may be configured to receive the identified keyframes as input from the keyframe matching module 224. The STN 224 may implement a neural network to fit a playing field model to the playing field segmentation information. By fitting the playing field model to such output, the STN 224 may generate a homography matrix for each keyframe.

[0062] The optical flow module 226 can be configured to identify the motion pattern of an object from one trackable frame to another. In some embodiments, the optical flow module 226 can receive as input the trackable frame information and body pose information of the athlete in each trackable frame. The optical flow module 226 can use the body pose information to remove the athlete from the trackable frame information. Once removed, the optical flow module 226 can determine the inter-frame motion to identify the camera motion between consecutive frames. In other words, the optical flow module 226 can identify the flow field from one frame to the next.

[0063] The optical flow module 226 and the STN 224 can work together to generate a homography matrix. For example, the optical flow module 226 and the STN 224 can generate a homography matrix for each trackable frame so that the camera can be calibrated for each frame. The homography matrix can be used to project the track or position of the athlete into real-world coordinates. For example, the homography matrix can indicate a two-dimensional to two-dimensional transformation, which can be used to project the position of the athlete from the image coordinates to the real-world coordinates on the field of play.

[0064] The player tracking agent 126 may be configured to generate a track for each player in the game. The player tracking agent 126 may include a neural network 232 and a re-identification agent 232. The player tracking agent 126 may receive as input a trackable frame, pose data, calibration data, and a broadcast video frame. In a first stage, the player tracking agent 126 may match player patch pairs derived from pose information based on appearance and distance. For example, let Let be the athlete patch of the jth athlete at time t. is the image coordinates of the jth athlete at time t width and height Thus, the player tracking agent 126 can use appearance correlation and Associate any detection pair as follows:

[0065]

[0066] where I is the bounding box position (x, y), width w and height h; C is the cross-correlation between image patches (e.g., image cropping using bounding boxes) and measures the similarity between two image patches; and L is a measure of the difference (e.g., distance) between two bounding boxes I.

[0067] Doing this for each pair can generate a large set of short tracklets. The endpoints of these tracklets can then be correlated based on motion consistency and color histogram similarity. In some embodiments, a tracklet can refer to a subset of a tracklet.

[0068] For example, let v i is the extrapolated velocity at the end of the ith trace, let v j is the extrapolated velocity at the beginning of the jth trace. Then c ij =v i ·v j It can represent the motion consistency score. In addition, let p(h) i represents the color likelihood, h, present in image patch i. The player tracking agent 126 may determine the color histogram similarity using the Bhattacharyya distance:

[0069] D B (p i ,p j )=-ln(BC(p i ,p j )),in

[0070] To reiterate, the tracking agent 120 finds matching tracklet pairs as follows:

[0071]

[0072] Solving each pair of interrupted tracklets can produce a net tracklet set, while leaving some tracklets with large gaps (i.e., many frames). To bridge the large gaps, the athlete tracking agent can augment the nearest neighbor metric to include运动场估计, which can account for changes in athlete direction occurring over many frames.

[0073] The运动场 can be a vector field that represents the velocity magnitude and direction as vectors at each position on the playing field. Given the known velocities of the numerous athletes on the playing field, cubic spline interpolation can be used to generate the complete运动场. For example, let be the court position of athlete i at each time t. Then there may be a pair of points with displacement if λ < T. Accordingly, the运动场 can be as follows:

[0074]

[0075] where G(x, 5) can be a Gaussian Kernel with a standard deviation approximately equal to five feet. In other words, the运动场 can be a Gaussian Blur of all displacements.

[0076] The neural network 232 can be used to predict the athlete运动场 given the true athlete trajectories. Given a set of true athlete trajectories, X i , the velocity of each athlete at each frame can be calculated, which can provide the ground truth运动场 for the neural network 232 to learn. For example, given a set of true athlete trajectories X i , the athlete tracking agent 126 can be configured to generate the set where can be the predicted运动场. For example, the neural network 232 can be trained to minimize . The athlete trajectory agent can then generate the nearest neighbor score for any tracking gap size λ as follows:

[0077] K ij = V(x, λ)·d ij

[0078] where is the displacement vector between all interrupted traces with a gap size of λ.

[0079] The re-identification agent 234 can be configured to link athletes who have left the field of view. The re-identification agent 234 can include a tracklet generator 236, a conditional autoencoder 240, and a Siamese network 242. It should be noted that the term "运动场估计" in the original text seems to be an incorrect or incomplete expression. It might need to be further clarified in the context to ensure a more accurate translation.

[0080] The trace generator 236 can be configured to generate a trace gallery. The trace generator 236 can receive multiple traces from the database 205. For each trace X, a player identity tag y can be included, and for each player patch I, posture information p can be provided by the posture detection stage. In view of the set of player traces, the trace generator 236 can establish a gallery for each trace, wherein the player's uniform number (or some other static feature) is always visible. The body posture information generated by the data set generator 122 allows the trace generator 236 to determine the position of the player. For example, the trace generator 236 can utilize a heuristic method, wherein the standardized shoulder width can be used as follows to determine the position:

[0081]

[0082] Where l can represent the position of a body part. Shoulder width can be normalized by torso length to eliminate scaling effects. Shoulders should be separated when the athlete is facing the camera or facing away from the camera, so the trace generator 236 can use those S orient Patches larger than the threshold are used to build a gallery. After this stage, each trace X n The following galleries can be included:

[0083]

[0084] The conditional autoencoder 240 can be configured to recognize one or more features in each trace. For example, unlike conventional approaches to re-identification problems, players in team sports can have very similar appearance features, such as clothing style, clothing color, and skin color. One of the more intuitive differences may be the team number that can be displayed on the front and / or back of each uniform. In order to capture these specific features, the conditional autoencoder 240 can be trained to recognize such features.

[0085] In some embodiments, the conditional autoencoder 240 may be a three-layer convolutional autoencoder, where the kernel size of all three layers may be 3x3, with 64, 128, and 128 channels, respectively. These hyperparameters may be tuned to ensure that the uniform numbers can be identified from the reconstructed image, so that the expected features can be learned in the autoencoder. In some embodiments, f(I i ) can be used to represent the features learned from image i.

[0086] Using conditional autoencoders 240 can improve the conventional process for a variety of reasons. First, there is usually not enough training data for each athlete, as some athletes only play a short amount of time in each game. Second, different teams may have the same uniform colors and uniform numbers, so it may be difficult to classify these athletes.

[0087] Siamese network 242 can be used to determine the similarity between two image patches. For example, Siamese network 242 can be trained to determine the similarity between two image patches based on their feature representations f(I). Given two image patches, their feature representations f(I i ) and f(I j ) can be flattened, concatenated, and input into the perception network. In some embodiments, the L2 norm can be used to connect the two sub-networks f(I i ) and f(I j ). In some embodiments, the perception network may include three layers, which may include 1024, 512, and 216 hidden units, respectively. Such a network can be used to determine the similarity s(I) between each pair of image patches of two traces without temporal overlap. i ,I j ). To enhance the robustness of the prediction, the final similarity score of two traces can be the average of all pairwise scores in their respective galleries:

[0088]

[0089] This similarity score can be calculated for every two traces without temporal overlap. If the score is above a certain threshold, the two traces can be associated.

[0090] Figure 3 is to illustrate the context according to an example embodiment in conjunction with Figure 2 and Figures 4 to 10 Block diagram 300 of the operational aspects of the discussion. Block diagram 300 may illustrate the overall workflow of organizing computing system 104 in generating athlete tracking information. Block diagram 300 may include multiple groups of operations 302 to 308. One group of operations 302 may involve generating a trackable frame (e.g., Figure 5 A set of operations 304 may involve generating one or more data sets from the trackable frames (e.g., operations performed by the data set generator 122). A set of operations 306 may involve camera calibration operations (e.g., Figure 7 A set of operations 308 may be directed to generating and predicting player traces (e.g., Fig. 9 Method 900 and Fig.10 Method 1000 in ).

[0091] Figure 4 is a flow chart illustrating a method 400 of generating a player trail according to an example embodiment. The method 400 may begin at step 402 .

[0092] At step 402, the organization computing system 104 may receive (or retrieve) a broadcast feed of the event. In some embodiments, the broadcast feed may be a live feed received in real time (or near real time) from the camera system 102. In some embodiments, the broadcast feed may be a broadcast feed of a game that has ended. In general, the broadcast feed may include multiple frames of video data. Each frame may capture a different camera perspective.

[0093] At step 404, the organizational computing system 104 may segment the broadcast feed into unified views. For example, the automatic editing agent 120 may be configured to parse multiple data frames in the broadcast feed to segment trackable frames from untrackable frames. Generally speaking, trackable frames may include those frames that point to the unified view. For example, the unified view may be considered to be a high sideline view. In other examples, the unified view may be a goal area view. In other examples, the unified view may be a top camera view.

[0094] At step 406, the organizational computing system 104 may generate multiple data sets from the trackable frames (i.e., unified views). For example, the data set generator 122 may be configured to generate multiple data sets based on the trackable clips received from the automatic clipping agent 120. In some embodiments, the posture detector 212 may be configured to detect athletes within a broadcast feed. The data set generator 122 may provide the trackable frames stored in the database 205 and the broadcast video feed as input to the posture detector 212. The output from the posture detector 212 may be posture data stored in the database 215 associated with the data set generator 122.

[0095] Ball detector 214 may be configured to detect and track a ball (or puck) within a broadcast feed. Dataset generator 122 may provide trackable frames stored in database 205 and a broadcast video feed as input to ball detector 214. In some embodiments, ball detector 214 may utilize Faster R-CNN to detect and track balls in trackable frames and broadcast video feeds. Output from ball detector 214 may be ball detection data stored in database 215 associated with dataset generator 122.

[0096] The field segmenter 216 may be configured to identify field markers in the broadcast feed. The dataset generator 122 may provide the field segmenter 216 with the trackable frames stored in the database 205 and the broadcast video feed as input. In some embodiments, the field segmenter 216 may be configured to utilize a neural network to identify field markers. The output of the field segmenter 216 may be the field markers stored in the database 215 associated with the dataset generator 122.

[0097] Accordingly, the dataset generator 120 may generate information for player positions, ball positions, and court portions in all trackable frames for further analysis.

[0098] In step 408, the organizational computing system 104 can calibrate the camera in each trackable frame based on the data set generated in step 406. For example, the camera calibrator 124 can be configured to calibrate the camera in each trackable frame by using the trackable frame and information to generate a homography matrix. The homography matrix allows the camera calibrator 124 to obtain those trajectories of each athlete in a given frame and project those trajectories into real-world coordinates. By projecting the athlete positions and trajectories into real-world coordinates for each frame, the camera calibrator 124 can ensure that the camera is calibrated for each frame.

[0099] In step 410, the organization computing system 104 can be configured to generate or predict each athlete's trace. For example, the athlete tracking agent 126 can be configured to generate or predict each athlete's trace in the competition. The athlete tracking agent 126 can receive and track frames, posture data, calibration data and broadcast video frames as input. Using such input, the athlete tracking agent 126 can be configured to build the athlete's movement in the entire given competition. In addition, the athlete tracking agent 126 can be configured to predict the athlete's track in view of each athlete's previous movement.

[0100] Figure 5 is a flowchart illustrating a method 500 for generating a trackable frame according to an example embodiment. The method 500 may correspond to the above combined Figure 4 Operation 404 is discussed. Method 500 may begin at step 502 .

[0101] At step 502, the organization computing system 104 may receive (or retrieve) a broadcast feed of the event. In some embodiments, the broadcast feed may be a live feed received in real time (or near real time) from the camera system 102. In some embodiments, the broadcast feed may be a broadcast feed of a game that has ended. In general, the broadcast feed may include multiple frames of video data. Each frame may capture a different camera perspective.

[0102] In step 504, the organization computing system 104 can generate a frame set for image classification. For example, the automatic editing agent 120 can perform frame-by-frame feature extraction from the broadcast feed using PCA analysis. For example, in view of pre-recorded video, the automatic editing agent 120 can extract a frame every X seconds (e.g., every 10 seconds) to build a PCA model of the video. In some embodiments, the automatic editing agent 120 can use incremental PCA to generate a PCA model, and the automatic editing agent 120 can thus select the previous component subset (e.g., the first 120 components) to generate the PCA model. The automatic editing agent 120 can also be configured to extract a frame from the broadcast stream every X seconds (e.g., every second) and use the PCA model to compress these frames. In some embodiments, the automatic editing agent 120 can compress the frame into a 120-dimensional form using the PCA model. For example, the automatic editing agent 120 can solve the principal component in a per-video manner and save the first 100 components of each frame to ensure accurate editing. Such a compressed frame subset can be regarded as a frame set for image classification. In other words, the PCA model can be used to compress each frame into a small vector, so that the frames can be clustered more effectively. Compression can be performed by selecting the first N components from the PCA model to represent the frame. In some examples, N can be 100.

[0103] At step 506, the organization computing system 104 may assign each frame in the frame set to a given cluster. For example, the automatic editing agent 120 may be configured to center, normalize, and cluster the first 120 components into a plurality of clusters. In some embodiments, for clustering of compressed frames, the automatic editing agent 120 may implement K-means clustering. In some embodiments, the automatic editing agent 120 may set k=9 clusters. K-means clustering attempts to obtain some data x={x1, x2, ..., x n} and divide it into k subsets, S={S1,S2,...S k}, optimized as follows:

[0104]

[0105] where μ j For set S j In other words, clustering model 204 attempts to find the cluster with the smallest inter-cluster variance using the K-means clustering technique. Clustering model 204 may label each frame with its respective cluster number (eg, cluster 1, cluster 2, ..., cluster k).

[0106] At step 508 , the tissue computing system 104 may classify each frame as trackable or non-trackable.

[0107] For example, the automatic editing agent 120 can use a neural network to classify each frame as trackable or untrackable. Trackable frames can represent frames that include capturing a unified view (e.g., a high-edge view). Untrackable frames can represent frames that do not capture a unified view. In order to train a neural network (e.g., neural network 206), an input data set including thousands of frames pre-labeled as trackable or untrackable that are run through a PCA model can be used. Each compressed frame and label pair (i.e., cluster number and trackable / untrackable) can be provided to the neural network for training. Accordingly, once trained, the automatic editing agent 120 can classify each frame as untrackable or trackable. In this way, each frame can have two labels: a cluster number and a trackable / untrackable classification.

[0108] At step 510, the organizational computing system 104 may compare each cluster to a threshold value. For example, the automatic editing agent 120 may utilize these two labels to determine whether a given cluster is considered trackable or untrackable. In some embodiments, if the automatic editing agent 120 determines that a threshold number of frames in a cluster are considered trackable (e.g., 80%), the automatic editing agent 120 may conclude that all frames in the cluster are trackable. Additionally, if the automatic editing agent 120 determines that less than a threshold number of frames in a cluster are considered untrackable (e.g., less than 30%), the automatic editing agent 120 may conclude that all frames in the cluster are untrackable. Further, if the automatic editing agent 120 determines that a specific number of frames in a cluster are considered trackable (e.g., 30% to 80%), the automatic editing agent 120 may request an administrator to further analyze the cluster.

[0109] If the organization computing system 104 determines in step 510 that greater than a threshold number of frames in a cluster are trackable, then in step 512 the auto-editing agent 120 may classify the cluster as trackable.

[0110] However, if the organization computing system 104 determines in step 510 that less than a threshold number of frames in a cluster are trackable, then in step 514 the auto-editing agent 120 may classify the cluster as untrackable.

[0111] Figure 6 6 is a block diagram 600 illustrating operational aspects discussed above in connection with method 500 according to an example embodiment. As shown, block diagram 600 may include multiple sets of operations 602-608.

[0112] In one set of operations 602, video data (e.g., broadcast video) may be provided to an automatic editing agent 120. The automatic editing agent 120 may extract frames from the video. In some embodiments, the automatic editing agent 120 may extract frames from the video at a low frame rate. The automatic editing agent may use an incremental PCA algorithm to select the first 120 components (e.g., frames) from the set of frames extracted by the automatic editing agent 120. Such operations may generate a video-specific PCA model.

[0113] In a set of operations 604, video data (e.g., broadcast video) may be provided to the automatic editing agent 120. The automatic editing agent 120 may extract frames from the video. In some embodiments, the automatic editing agent 120 may extract frames from the video at a medium frame rate. The automatic editing agent 120 may use a video-specific PCA model to compress the frames extracted by the automatic editing agent 120.

[0114] At a set of operations 606, the compressed frames and the preselected number of desired clusters may be provided to the automatic editing agent 120. The automatic editing agent 120 may group the frames into one or more clusters as described by the preselected number of desired clusters using a K-means clustering technique. The automatic editing agent 120 may assign a cluster label to each compressed frame. The automatic editing agent 120 may also be configured to classify each frame as trackable or non-trackable. The automatic editing agent 120 may so label each corresponding frame.

[0115] In a set of operations 608, the automatic editing agent 120 can analyze each cluster to determine whether the cluster includes at least a threshold number of trackable frames. For example, as shown, if 80% of the frames in the cluster are classified as trackable, the automatic editing agent 120 can consider the entire cluster to be trackable. However, if less than 80% of the frames in the cluster are classified as trackable, the automatic editing agent can determine whether at least a second threshold number of frames in the cluster are trackable. For example, if 70% of the frames in the cluster are classified as untrackable, the automatic editing agent 120 can consider the entire cluster to be trackable. However, if less than 70% of the frames in the cluster are classified as untrackable, i.e., 30% to 70% are trackable, manual annotation can be requested.

[0116] Figure 7 700 is a flowchart illustrating a method 700 for calibrating a camera for each trackable frame according to an example embodiment. The method 700 may correspond to the method described above in conjunction with Figure 4 Operation 408 is discussed. Method 700 may begin at step 702 .

[0117] At step 702, the organizational computing system 104 may retrieve the video data and the pose data for analysis. For example, the camera calibrator 124 may retrieve the trackable frames of a given competition and the pose data of the athletes in each trackable frame from the database 205. Following step 702, the camera calibrator 124 may perform two parallel processes to generate a homography matrix for each frame. Accordingly, the following operations are not intended to be discussed as being performed sequentially, but may be performed in parallel or sequentially.

[0118] At step 704, the organizational computing system 104 may remove the athlete from each trackable frame. For example, the camera calibrator 124 may parse each trackable frame retrieved from the database 205 to identify one or more athletes contained therein. The camera calibrator 124 may remove the athlete from each trackable frame using the pose data retrieved from the database 205. For example, the camera calibrator 124 may identify those pixels that correspond to the pose data and remove the identified pixels from a given trackable frame.

[0120] At step 706, the tissue computing system 104 may identify the motion of an object (e.g., a surface, an edge, etc.) between consecutive trackable frames. For example, the camera calibrator 124 may analyze consecutive trackable frames, removing the athlete therefrom, to determine the motion of the object from one frame to the next. In other words, the optical flow module 226 may identify a flow field between consecutive trackable frames.

[0121] In step 708, the organizational computing system 104 can match the output from the field segmenter 216 with the template set. For example, the camera calibrator 124 can match one or more frames with clear field images with one or more templates. The camera calibrator 124 can parse the group of trackable clips to identify those clips that provide clear field images and tags therein. Based on the selected clips, the camera calibrator 124 can compare such images with the field templates. Each template can represent a different camera perspective of the field. Those frames that can match a given template are considered key frames. In some embodiments, the camera calibrator 124 can implement a neural network to match one or more frames. In some embodiments, the camera calibrator 124 can implement cross-correlation to match one or more frames.

[0122] At step 710, the tissue computing system 104 may fit the field model to each key frame. For example, the camera calibrator 124 may be configured to receive the identified key frame as input. The camera calibrator 124 may implement a neural network to fit the field model to the field segmentation information. By fitting the field model to such output, the camera calibrator 124 may generate a homography matrix for each key frame.

[0123] At step 712, the tissue computing system 104 may generate a homography matrix for each trackable frame. For example, the camera calibrator 124 may generate a homography matrix for each frame using the flow field identified in step 706 and the homography matrix for each key frame. The homography matrix may be used to project the track or position of the athlete into real-world coordinates. For example, given the geometric transformation represented by the homography matrix, the camera calibrator 124 may use the transformation to project the position of the athlete on the image into real-world coordinates on the field of play.

[0124] At step 714 , the tissue computing system 104 may calibrate each camera based on the homography matrix.

[0125] Figure 8 8 is a block diagram 800 illustrating operational aspects discussed above in connection with method 700 according to an example embodiment. As shown, block diagram 800 may include input 802, a first set of operations 804, and a second set of operations 806. The first set of operations 804 and the second set of operations 806 may be performed in parallel.

[0126] Input 802 may include video clips 808 and gesture detections 810. In some embodiments, video clips 808 may correspond to trackable frames generated by auto-editing agent 120. In some embodiments, gesture detections 810 may correspond to gesture data generated by gesture detector 212. As shown, only video clips 808 may be provided as input to first set of operations 804; both video clips 804 and gesture detections 810 may be provided as input to second set of operations 806.

[0127] The first group of operations 804 may include semantic segmentation 812, key frame matching 814, and STN fitting 816. At semantic segmentation 812, the field segmenter 216 may be configured to identify the field markings in the broadcast feed. In certain embodiments, the field segmenter 216 may be configured to utilize a neural network to identify the field markings. Such segmentation information may be performed in advance and provided to the camera calibration 124 from the database 215. At key frame matching 814, the key frame matching module 224 may be configured to match one or more frames of the field image with one or more templates. At STN fitting 816, STN 226 may implement a neural network to fit the field model to the field segmentation information. By fitting the field model to such an output, STN 224 may generate a homography matrix for each key frame.

[0128] The second set of operations 806 may include camera motion estimation 818. At camera flow estimation 818, the optical flow module 226 may be configured to identify motion patterns of objects from one trackable frame to another. For example, the optical flow module 226 may use body pose information to remove athletes from the trackable frame information. Once removed, the optical flow module 226 may determine inter-frame motion to identify camera motion between consecutive frames.

[0129] The first set of operations 804 and the second set of operations 806 may result in homography interpolation 816. The optical flow module 226 and the STN 224 may work together to generate a homography matrix for each trackable frame so that the camera can be calibrated for each frame. The homography matrix may be used to project the track or position of the player into real-world coordinates.

[0130] Fig. 9 900 is a flowchart illustrating a method 900 for tracking an athlete according to an example embodiment. The method 900 may correspond to the method described above in conjunction with Figure 4 Operation 410 is discussed. Method 900 may begin at step 902 .

[0131] At step 902, the organization computing system 104 may retrieve a plurality of trackable frames for matching. Each of the plurality of trackable frames may include one or more metadata sets associated therewith. Such metadata may include, for example, body posture information and camera calibration data. In some embodiments, the athlete tracking agent 126 may also retrieve broadcast video data.

[0132] At step 904, the organizational computing system 104 may generate a set of short traces. For example, the player tracking agent 126 may match player patch pairs derived from the pose information based on appearance and distance to generate a set of short traces. For example, Let be the athlete patch of the jth athlete at time t. is the image coordinates of the jth athlete at time t width and height Thus, the player tracking agent 126 can use appearance correlation and Associate any detection pair as follows:

[0133]

[0134] Doing this for each pair generates a set of short traces. The endpoints of these traces can then be correlated with each other based on motion consistency and color histogram similarity.

[0135] For example, let v i is the extrapolated velocity at the end of the ith trace, let v jis the extrapolated velocity at the start of the j-th trace segment. Then c ij = v i · v j can represent the motion consistency score. Additionally, let p(h) i represent the color likelihood, where h exists in image patch i. The athlete tracking agent 126 can use the Bhattacharyya distance to measure the color histogram similarity:

[0136] D B (p i , p j ) = -ln(BC(p i , p j ))), where

[0137] In step 906, the tissue computing system 104 can connect the gaps between each short trace segment set. For example, reiterating, the tracking agent 120 finds matching trace segment pairs as follows:

[0138]

[0139] Solving each pair of interrupted trace segments can produce a net trace segment set, while leaving some trace segments with large gaps (i.e., many frames). To connect large gaps, the athlete tracking agent 126 can augment the nearest neighbor metric to include motion field estimation, which can account for changes in the athlete's direction occurring over many frames.

[0140] The motion field can be a vector field that determines the direction an athlete will be in at a certain point on the playing field x after a period of time λ. For example, let be the playing field position of athlete i at each time t. Then there may be a pair of points with displacement if λ < T. Accordingly, the motion field can be as follows:

[0141]

[0142] where G(x, 5) can be a Gaussian kernel with a standard deviation approximately equal to five feet. In other words, the motion field can be a Gaussian blur of all displacements.

[0143] In step 908, the tissue computing system 104 can predict the movement of the players based on the motion field. For example, the athlete tracking system 126 can use a neural network (e.g., neural network 232) to predict the athlete's trajectory given the ground truth athlete trajectory. Given a set of true athlete trajectories X i , the athlete tracking agent 126 can be configured to generate a set where can be the predicted motion field. The athlete tracking agent 126 can train the neural network 232 to reduce (e.g., minimize) The player tracking agent 126 may then generate a neighbor score for any tracking gap size λ as follows:

[0144] K ij =V(x,λ)·d ij

[0145] in is the displacement vector between all interrupted traces with a gap size of λ. Based on this, the player tracking agent 126 can solve the above matching pairs. For example, the player tracking agent 126 can use the Hungarian Algorithm to assign each pair of interrupted traces given the neighbor scores. The Hungarian algorithm (e.g., Kuhn-Munkres) can optimize the best matching set under the constraint that each pair must be matched.

[0146] At step 910, organizational computing system 104 may output a graphical representation of the predictions. For example, interface agent 128 may be configured to generate one or more graphical representations corresponding to the tracks of each athlete generated by athlete tracking agent 126. For example, interface agent 128 may be configured to generate one or more graphical user interfaces (GUIs) that include a graphical representation of each predicted athlete tracked generated by athlete tracking agent 126.

[0147] In some cases, during competition, athletes or players tend to wander outside the camera's field of view. Such problems may occur during injuries, athlete fatigue, rapid turnovers, rapid offense-defense transitions, etc. Accordingly, an athlete in a first trackable frame may no longer be in a subsequent second or third trackable frame. The athlete tracking agent 126 can address this problem via the re-identification agent 234.

[0148] Fig.10 1 is a flow chart showing a method 1000 for tracking an athlete according to an example embodiment. The method 1000 may correspond to the method described above in conjunction with Figure 4 Operation 410 is discussed. Method 1000 may begin at step 1002 .

[0149] At step 1002, the organization computing system 104 may retrieve a plurality of trackable frames for matching. Each of the plurality of trackable frames may include one or more metadata sets associated therewith. Such metadata may include, for example, body posture information and camera calibration data. In some embodiments, the athlete tracking agent 126 may also retrieve broadcast video data.

[0150] In step 1004, the organization computing system 104 can identify a subset of short traces in which the athlete has left the camera's line of sight. Each trace can include multiple image patches associated with at least one athlete. An image patch can refer to a corresponding frame subset of multiple trackable frames. In some embodiments, each trace X can include an athlete identity tag y. In some embodiments, each athlete patch I in a given trace X can include gesture information generated by a data set generator 122. For example, in view of the input video, gesture detection, and trackable frames, the re-identification agent 234 can generate a trace set including several short interrupted traces of the athlete.

[0151] At step 1006, the organizational computing system 104 may generate a gallery for each trace. For example, given those small traces, the re-identification agent 234 may build a gallery for each trace. The re-identification agent 234 may build a gallery for each trace where the player's uniform number (or some other static feature) is always visible. The body posture information generated by the data set generator 122 allows the re-identification agent 234 to determine the orientation of each player. For example, the re-identification agent 234 may utilize a heuristic that may use the normalized shoulder width to determine orientation as follows:

[0152]

[0153] Where l can represent the position of a body part. The shoulder width can be normalized by the torso length to eliminate the scaling effect. The shoulders should be separated when the athlete is facing the camera or facing away from the camera, so the re-identification agent 234 can use those S orient Patches larger than the threshold are used to build a gallery. n The following galleries can be included:

[0154]

[0155] At step 1008, the organizational computing system 104 can use a convolutional autoencoder to match the traces. For example, the re-identification agent 234 can use a conditional autoencoder (e.g., the conditional autoencoder 240) to identify one or more features in each trace. For example, unlike conventional approaches to re-identification problems, players in team sports can have very similar appearance features, such as clothing style, clothing color, and skin color. One of the more intuitive differences may be the team uniform number that may be displayed on the front and / or back of each team uniform. To capture these specific features, the re-identification agent 234 can train a conditional autoencoder to recognize such features.

[0156] In some embodiments, the conditional autoencoder can be a three-layer convolutional autoencoder, where the kernel size of all three layers can be 3x3, with 64, 128, and 128 channels, respectively. These hyperparameters can be tuned to ensure that the team uniform numbers can be identified from the reconstructed image so that the expected features can be learned in the autoencoder. In some embodiments, f(I i ) can be used to represent the features learned from image i.

[0157] Using a specific example, the re-identification agent 234 may identify a first track corresponding to a first player. Using the conditional autoencoder 240, the re-identification agent 234 may learn a first set of uniform features associated with the first track, for example, based on a first set of image patches that include or are associated with the first track. The re-identification agent 234 may also identify a second track that may initially correspond to a second player. Using the conditional autoencoder 240, the re-identification agent 234 may learn a second set of uniform features associated with the second track, for example, based on a second set of image patches that include or are associated with the second track.

[0158] At step 1010, the organization computing system 104 may use a Siamese network to determine the similarity between the matching traces. For example, the re-identification agent 234 may train a Siamese network (e.g., the Siamese network 242) to determine the similarity between two image patches based on their feature representations f(I). Given two image patches, their feature representations f(I i ) and f(I j ) can be flattened, concatenated, and fed into the perception network. In some embodiments, the L2 norm can be used to connect the two sub-networks f(I i ) and f(I j ). In some embodiments, the perception network may include three layers, including 1024, 512, and 216 hidden units, respectively. Such a network can be used to determine the similarity s(I) between each pair of image patches of two traces without temporal overlap. i ,I j ). To enhance the robustness of the prediction, the final similarity score of two traces can be the average of all pairwise scores in their respective galleries:

[0159]

[0160] Continuing with the above example, the re-identification agent 234 may utilize the Siamese network 242 to calculate a similarity score between the first learned uniform feature set and the second learned uniform feature set.

[0161] At step 1012, if the similarity scores of the traces are above a predetermined threshold, the organization computing system 104 may associate the traces. For example, the re-identification agent 234 may calculate a similarity score for each two traces that do not overlap in time. If the score is above a certain threshold, the re-identification agent 234 may associate the two traces.

[0162] Continuing with the above example, the re-identification agent 234 may associate the first trace with the second trace if, for example, the similarity score generated by the Siamese network 242 is at least above a threshold. Assuming the similarity score is above the threshold, the re-identification agent 234 may determine that the first player in the first trace and the second player in the second trace are indeed the same person.

[0163] Fig.11 is a block diagram 1100 illustrating operational aspects discussed above in connection with method 1000 according to an example embodiment.

[0164] As shown, block diagram 1100 may include input video 1102, gesture detection 1104, player tracking 1106, trail collection 1108, gallery creation and pairing 1110, and trail connection 1112. Block diagram 1100 illustrates the general pipeline of method 1000 provided above.

[0165] Given an input video 1102, gesture detection information 1104 (eg, generated by gesture detector 212)

[0166] With the player tracking information 1106 (e.g., generated by one or more of the player tracking agent 126, the automatic editing agent 120, and the camera calibrator 124), the re-identification agent 234 can generate a set of traces 1108. Each set of traces 1108 can include a plurality of short interrupted traces (e.g., traces 1114) of the player. Each trace 1114 can include one or more image patches 1116 contained therein. In view of the traces 1114, the re-identification agent 234 can generate a gallery 1110 for each trace. For example, the gallery 1110 can include those image patches 1118 in a given trace, which include images of the players whose orientations meet a threshold. In other words, the re-identification agent 234 can generate a gallery 1110 for each trace 1114 including the image patches 1118 of each player, so that the number of the player can be visible in each frame. The image patches 1118 can be a subset of the image patches 1116. The re-recognition agent 234 may re-pair each frame to calculate a similarity score via a Siamese network. For example, as shown, the re-recognition agent 234 may match the first frame from trace II with the second frame from trace I and feed the frames to the Siamese network.

[0167] The re-recognition agent 234 can then connect the traces 1112 based on the similarity scores. For example, if the similarity scores of two frames exceed a certain threshold, the re-recognition agent 234 can connect or associate those traces.

[0168] Fig.12 1 is a block diagram illustrating an architecture 1200 of a Siamese network 242 of a re-identification agent 234 according to an example embodiment. As shown, the Siamese network 242 may include two sub-networks 1202, 1204 and one perception network 1205.

[0169] Each of the two sub-networks 1202, 1204 can be configured similarly. For example, sub-network 1202 can include a first convolutional layer 1206, a second convolutional layer 1208, and a third convolutional layer 1210. The first sub-network 1202 can receive the player patch I1 as input and output a feature set learned from the player patch I1 (denoted as f(I1)). Sub-network 1204 can include a first convolutional layer 1216, a second convolutional layer 1218, and a third convolutional layer 1220. The second sub-network 1204 can receive the player patch I2 as input and can output a feature set learned from the player patch I2 (denoted as f(I2)). The outputs from sub-network 1202 and sub-network 1204 can be encoded representations of the corresponding player patches I1, I2. In some embodiments, the outputs from sub-network 1202 and sub-network 1204 can be followed by a flattening operation, which can generate respective feature vectors f(I1) and f(I2), respectively. In some embodiments, each feature vector f(I1) and f(I2) may include 10240 units. In some embodiments, the L2 norm of f(I1) and f(I2) may be calculated and used as input to the perception network 1205.

[0171] Perception network 1205 may include three layers 1222 to 1226. In some embodiments, layer 1222 may include 1024 hidden units. In some embodiments, layer 1224 may include 512 hidden units. In some embodiments, layer 1226 may include 256 hidden units. Perception network 1205 may output a similarity score between image patches I1 and I2.

[0172] Fig.13AA system bus of a computing system architecture 1300 according to an example embodiment is shown. The system 1300 may represent at least a portion of an organization computing system 104. One or more components of the system 1300 may communicate electrically with each other using a bus 1305. The system 1300 may include a processing unit (CPU or processor) 1310 and a system bus 1305, which couples various system components including a system memory 1315 (such as a read-only memory (ROM) 1320 and a random access memory (RAM) 1325) to the processor 1310. The system 1300 may include a cache of high-speed memory that is directly connected, adjacent to the processor 1310, or integrated as part of the processor 1310. The system 1300 may copy data from the memory 1315 and / or the storage device 1330 to the cache 1312 for quick access by the processor 1310. In this way, the cache 1312 may provide a performance improvement that avoids delays incurred by the processor 1310 while waiting for data. These and other modules may control or be configured to control the processor 1310 to perform various actions. Other system memories 1315 may also be used. Memory 1315 may include a variety of different types of memory with different performance characteristics. Processor 1310 may include any general purpose processor and hardware or software modules such as Service I 1332, Service II 1334, and Service III 1336 (configured to control processor 1310) stored in storage device 1330, as well as special purpose processors, where the software instructions are incorporated into the actual processor design. Processor 1310 may essentially be a completely independent computing system, including multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0173] In order to enable a user to interact with the computing device 1300, the input device 1345 can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphic input, a keyboard, a mouse, motion input, voice, etc. The output device 1335 can also be one or more of a variety of output mechanisms known to those skilled in the art. In some cases, a multimodal system can enable a user to provide multiple types of input to communicate with the computing device 1300. The communication interface 1340 can generally control and manage user input and system output. There is no restriction on the operation on any particular hardware arrangement, so the basic features of the present disclosure can be easily replaced with improved hardware or firmware arrangements still under development.

[0174] The storage device 1330 can be a non-volatile memory, or it can be a hard disk or other type of computer 202080017536.8 Division PX20240433|00166-0064-00139 readable medium that can store data accessible to a computer, such as magnetic tape, flash memory card, solid-state storage device, digital versatile disk, cassette, random access memory (RAM) 1325, read-only memory (ROM) 1320, and a mixture thereof.

[0175] The storage device 1330 may include services 1332, 1334, and 1336 for controlling the processor 1310. Other hardware or software modules are also contemplated. The storage device 1330 may be connected to the system bus 1305. In one aspect, a hardware module that performs a particular function may include a software component stored in a computer-readable medium that is associated with the necessary hardware components to perform the function, such as the processor 1310, the bus 1305, the display 1335, etc.

[0176] Fig. 13B A computer system 1350 having a chipset architecture that can represent at least a portion of an organizational computing system 104 is shown. Computer system 1350 can be an example of computer hardware, software, and firmware that can be used to implement the disclosed technology. System 1350 may include a processor 1355, which represents any number of physically and / or logically different resources that can execute software, firmware, and hardware configured to perform the identified calculations. Processor 1355 can communicate with chipset 1360, which can control the input and output of processor 1355. In this example, chipset 1360 outputs information to output 1365, such as a display, and can read and write information to storage device 1370, which can, for example, include magnetic media and solid-state media. Chipset 1360 can also read data from RAM 1375 and write data to RAM 1375. A bridge 1380 for docking with various user interface components 1385 can be provided to dock with chipset 1360. Such user interface components 1385 may include a keyboard, a microphone, touch detection processing circuitry, a pointing device such as a mouse, etc. In general, input to the system 1350 may come from any of a variety of sources, whether machine-generated and / or human-generated.

[0177] Chipset 1360 may also interface with one or more communication interfaces 1390 having different physical interfaces. Such communication interfaces may include interfaces for wired and wireless local area networks, broadband wireless networks, and personal area networks. Some applications of the methods for generating, displaying, and using the GUI disclosed herein may include receiving an ordered data set through a physical interface or being generated by the machine itself by analyzing data stored in memory 1370 or 1375 through processor 1355. In addition, the machine may receive input from a user through user interface component 1385 and perform appropriate functions, such as browsing functions, by using processor 1355 to interpret these inputs.

[0178] It will be appreciated that the example systems 1300 and 1350 may have more than one processor 1310 or be part of a group or cluster of computing devices networked together to provide greater processing power.

[0179] Although the above is directed to the embodiments described in the present disclosure, other and more embodiments may be designed without departing from the basic scope of the present disclosure. For example, various aspects of the present disclosure may be implemented in hardware or software or a combination of hardware and software. An embodiment described in the present disclosure may be implemented as a program product for use with a computer system. The program of the program product defines the functions of the embodiment (including the method described in the present disclosure) and may be included on various computer-readable storage media. Illustrative computer-readable storage media include, but are not limited to: (i) non-writable storage media on which information is permanently stored (e.g., a read-only memory (ROM) device in a computer, such as a CD-ROM disk, flash memory, ROM chip, or any type of solid-state non-volatile memory that can be read by a CD-ROM drive); (ii) writable storage media on which information can be changed (e.g., a floppy disk in a floppy disk drive or hard disk drive or any type of solid-state random access memory). Such a computer-readable storage medium becomes an embodiment of the present disclosure when it carries computer-readable instructions that guide the functions of an embodiment of the present disclosure.

[0180] Those skilled in the art will appreciate that the above examples are exemplary only and not restrictive. After reading this specification and studying the drawings, those skilled in the art will understand that all substitutions, enhancements, equivalents, and improvements are intended to fall within the true spirit and scope of protection of the present disclosure. Therefore, the appended claims are intended to include all such modifications, substitutions, and equivalents that fall within the true spirit and scope of protection taught by the present disclosure.

Claims

1. A computer-implemented method for tracking a player, the computer-implemented method comprising: Receiving, by one or more processors, a plurality of trackable frames of a sports game; generating, by the one or more processors, one or more track slice sets based on the plurality of trackable frames; predicting, by the one or more processors, the movement of a player in each of the one or more track sets based on a playing field of a playing field of the sports game; as well as A graphical representation of the predicted movement of the athlete is output by the one or more processors.

2. The computer-implemented method of claim 1, further comprising: Connecting, by the one or more processors, at least one gap between each of the one or more trace sets, wherein connecting at least one gap comprises augmenting a neighbor metric to include a motion field estimate.

3. The computer-implemented method of claim 2, wherein the motion field estimate corresponds to changes in the player's direction occurring over the plurality of trackable frames.

4. The computer-implemented method of claim 1, wherein the plurality of trackable frames includes body pose information and camera calibration data.

5. The computer-implemented method of claim 4, wherein the one or more trace sets are derived from the body posture information.

6. The computer-implemented method of claim 1, wherein predicting the movement of a player in each trackset of the one or more tracksets comprises generating one or more tracks for each player on the playing field.

7. The computer-implemented method of claim 6, wherein outputting a graphical representation of the predicted movement of the athlete comprises: One or more graphical representations are generated corresponding to the one or more trajectories of each player on the playing field.

8. The computer-implemented method of claim 1 , wherein predicting the movement of the player comprises predicting one or more player trajectories based on ground truth player trajectories using a neural network.

9. A non-transitory computer readable medium comprising one or more sequences of instructions which, when executed by one or more processors, cause a computing system to perform operations comprising: Receiving, by the computer system, a plurality of trackable frames of a sports game; generating, by the computer system, one or more track slice sets based on the plurality of trackable frames; predicting, by the computer system, the movement of a player in each of the one or more track sets based on a playing field of a playing field of the sports game; and A graphical representation of the predicted movement of the athlete is output by the computer system.

10. The non-transitory computer readable medium of claim 9, the operations further comprising: Connecting, by the computer system, at least one gap between each of the one or more trace sets, wherein connecting at least one gap comprises augmenting a neighbor metric to include a motion field estimate.

11. The non-transitory computer readable medium of claim 10, wherein the motion field estimate corresponds to changes in direction of the player occurring over the plurality of trackable frames.

12. The non-transitory computer-readable medium of claim 9, wherein the plurality of trackable frames comprises body pose information and camera calibration data.

13. The non-transitory computer-readable medium of claim 12, wherein the one or more trace sets are derived from the body posture information.

14. The non-transitory computer-readable medium of claim 9, wherein predicting the movement of a player in each of the one or more tracksets comprises generating one or more tracks for each player on the playing field.

15. The non-transitory computer-readable medium of claim 14, wherein outputting the graphical representation of the predicted movement of the athlete comprises: One or more graphical representations are generated corresponding to the one or more trajectories of each player on the playing field.

16. A system comprising processor; as well as a memory containing one or more instruction sequences that, when executed by the processor, cause the system to perform operations comprising: receiving a plurality of trackable frames of a sports game; generating one or more track slice sets based on the plurality of trackable frames; predicting the movement of a player in each of the one or more track sets based on a playing field of a playing field of the sports game; and A graphical representation of the predicted movement of the athlete is output.

17. The system of claim 16, wherein the plurality of trackable frames includes body pose information and camera calibration data.

18. The system of claim 16, wherein predicting the movement of a player in each of the one or more track sets comprises generating one or more tracks for each player on the playing field.

19. The system of claim 18, wherein outputting a graphical representation of the athlete's predicted movement comprises: One or more graphical representations are generated corresponding to the one or more trajectories of each player on the playing field.

20. The system of claim 16, wherein predicting the movement of the player comprises predicting one or more player trajectories based on ground truth player trajectories using a neural network.

Images