End-to-end camera calibration for broadcast video

Abstract

This paper discloses a system and method for calibrating broadcast video sources. The computational system retrieves multiple broadcast video sources comprising multiple video frames. The computational system generates a trained neural network by generating multiple training datasets based on the broadcast video sources and learning to generate a homography matrix for each of the multiple frames through a neural network. The computational system receives a target broadcast video source containing a target motion event. The computational system divides the target broadcast video source into multiple target frames. The computational system generates a target homography matrix for each of the multiple target frames via a neural network. The computational system calibrates the target broadcast video source by warping each target frame with the corresponding target homography matrix.

Description

(Case number 202180018658.3)

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Provisional Application Serial No. 63 / 008,184, filed April 10, 2020, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure generally relates to systems and methods for end-to-end camera calibration of broadcast video action and participants based on, for example, tracking data. Background Technology

[0004] The increasing number of vision-based tracking systems deployed in production requires fast and robust camera calibration. For example, in the field of sports, much of the current work focuses on sports that are easy to extract lines and intersections and have a relatively consistent appearance across the field. Summary of the Invention

[0005] In some embodiments, this document discloses a method for calibrating broadcast video sources. A computing system retrieves multiple broadcast video sources for multiple motion events. Each broadcast video source includes multiple video frames. The computing system generates multiple training datasets based on the broadcast video sources by dividing them into multiple frames and learns to generate a homography matrix for each of the multiple frames via a neural network. The computing system receives a target broadcast video source for a target motion event. The computing system divides the target broadcast video source into multiple target frames. The computing system generates a target homography matrix for each of the multiple target frames via a neural network. The computing system calibrates the target broadcast video source by warping each target frame by the corresponding target homography matrix.

[0006] In some embodiments, a system is disclosed herein. The system includes a processor and a memory. The memory has programming instructions stored thereon that, when executed by the processor, perform one or more operations. One or more operations include retrieving multiple broadcast video sources of multiple motion events. Each broadcast video source includes multiple video frames. One or more operations further include generating a trained neural network by dividing the broadcast video sources into multiple frames to generate multiple training datasets based on the broadcast video sources and by learning to generate a homography matrix for each of the multiple frames through a neural network. One or more operations further include receiving a target broadcast video source of a target motion event. One or more operations further include dividing the target broadcast video source into multiple target frames. One or more operations further include generating a target homography matrix for each of the multiple target frames through a neural network. One or more operations further include calibrating the target broadcast video source by warping each target frame by a corresponding target homography matrix.

[0007] In some embodiments, this document discloses a non-transitory computer-readable medium. The non-transitory computer-readable medium includes one or more sequences of instructions, which, when executed by one or more processors, cause a computing system to perform one or more operations. The computing system retrieves multiple broadcast video sources of multiple motion events. Each broadcast video source includes multiple video frames. The computing system generates a trained neural network by dividing the broadcast video sources into multiple frames to generate multiple training datasets based on the broadcast video sources and by learning to generate a homography matrix for each of the multiple frames through a neural network. The computing system receives a target broadcast video source of a target motion event. The computing system divides the target broadcast video source into multiple target frames. The computing system generates a target homography matrix for each of the multiple target frames via a neural network. The computing system calibrates the target broadcast video source by warping each target frame by the corresponding target homography matrix. Attached Figure Description

[0008] To gain a more detailed understanding of the features described above, a more specific description of the present disclosure can be provided with reference to embodiments, some of which are illustrated in the accompanying drawings. However, it should be noted that the drawings illustrate only typical embodiments of the present disclosure and should not be considered as limiting the scope of the disclosure, as other equally effective embodiments are permissible.

[0009] Figure 1 This is a block diagram illustrating a computing environment according to an example embodiment.

[0010] Figures 2A-2B This is a block diagram illustrating the neural network architecture of a camera calibrator according to an example embodiment.

[0011] Figure 3This is a block diagram illustrating one or more images of a competition venue according to an example embodiment.

[0012] Figure 4 This is a flowchart illustrating a method for generating a fully trained calibration model according to an example embodiment.

[0013] Figure 5 This is a flowchart illustrating a method for calibrating a broadcast camera according to an example embodiment.

[0014] Figure 6A This is a block diagram illustrating a computing device according to an example embodiment.

[0015] Figure 6B This is a block diagram illustrating a computing device according to an example embodiment.

[0016] For ease of understanding, the same reference numerals have been used to denote common, identical elements in the figures where possible. It is contemplated that elements disclosed in one embodiment may be advantageously used in other embodiments without specific description. Detailed Implementation

[0017] Camera calibration is a crucial task in computer vision applications such as tracking systems, simultaneous localization and mapping (SLAM), and augmented reality (AR). Recently, many professional sports leagues have deployed versions of vision-based tracking systems. Furthermore, AR applications used during video broadcasts to enhance audience engagement (e.g., in the NBA) are also being implemented. ® Virtual 3 in NFL ® The concept of "First Down Line" in sports has become commonplace. All these applications require high-quality camera calibration systems. Currently, most of these applications rely on multiple pre-calibrated fixed cameras or real-time feeds directly from the camera's pan-tilt-zoom (PTZ) parameters. However, since broadcast video is the most widely available data source in sports, the ability to calibrate from a single moving camera with unknown and varying camera parameters would significantly expand the scope of athlete tracking data and fan engagement solutions. Calibration of a single moving camera remains a challenging task, as the method must be accurate, fast, and generalizable to a wide range of views and appearances. One or more techniques described in this paper allow computational systems to determine the camera homography of a single moving camera given a frame and motion.

[0018] Current methods for camera calibration primarily follow a framework based on field registration, template matching (i.e., camera pose initialization), and homography refinement. Most of these methods focus on the ease of extracting semantic information (e.g., key court markings), consistent court appearance across the stadium (e.g., grass and white lines), and relatively slow and smooth camera movement. However, these assumptions do not apply to more dynamic sports, such as basketball, where players obscure court markings, court appearance varies across the field, and the camera moves rapidly.

[0019] Furthermore, most existing work consists of multiple independent models, each trained or tuned separately. Therefore, they cannot achieve global optimality for such optimization tasks. This problem also limits the performance of these methods in more challenging scenarios, as errors propagate module-to-module throughout the system.

[0020] One or more techniques described herein relate to novel end-to-end neural networks for camera calibration. By using end-to-end neural networks, this system is able to handle more challenging scenarios involving motion blur, occlusion, and large transforms—scenarios that existing systems simply cannot interpret or resolve. In some embodiments, this system implements region-based semantics instead of lines for camera calibration, thus providing a more robust approach to dynamic environments and those with highly variable appearance features. In some embodiments, this system incorporates a spatial transform network for large transform learning, which helps reduce the number of templates required for calibration purposes. In some embodiments, this system implements an end-to-end architecture for camera calibration, which allows for more efficient joint training and inference of homography.

[0021] Figure 1 This is a block diagram illustrating a computing environment 100 according to an example embodiment. The computing environment 100 may include a camera system 102 communicating via a network 105, an organizational computing system 104, and one or more client devices 108.

[0022] Network 105 can be any suitable type, including a separate connection via the Internet (e.g., a cellular or Wi-Fi network). In some embodiments, network 105 can use direct connections (e.g., Radio Frequency Identification (RFID), Near Field Communication (NFC), Bluetooth™, Bluetooth Low Energy™ (BLE), Wi-Fi™, ZigBee™, Ambient Backscatter Communication (ABC) protocol, USB, WAN, or LAN) to connect terminals, services, and mobile devices. Because the transmitted information may be personal or confidential, encryption or other protection of one or more of these types of connections may be required for security reasons. However, in some embodiments, the information being transmitted may be less personal, and therefore, a network connection may be chosen for convenience rather than security.

[0023] Network 105 may include any type of computer network arrangement for exchanging data or information. For example, network 105 may be the Internet, a private data network, a virtual private network using a public network, and / or other suitable connections that enable components in computing environment 100 to send and receive information between components of environment 100.

[0024] Camera system 102 may be located within field 106. For example, field 106 may be configured to host a motion event involving one or more subjects 112. Camera system 102 may be configured to capture the movement of all subjects (i.e., players) on the playing field, as well as the movement of one or more other related objects (e.g., ball, referee, etc.). In some embodiments, camera system 102 may be an optical-based system using, for example, multiple fixed cameras. For example, a system consisting of six fixed, calibrated cameras may be used to project the three-dimensional positions of the players and the ball onto a two-dimensional top view of the field. In another example, a combination of stationary and non-stationary cameras may be used to capture the movement of all subjects on the playing field, as well as one or more related objects. As those skilled in the art will recognize, such a camera system (e.g., camera system 102) may generate many different camera views of the field (e.g., high sideline view, penalty line view, player huddle view, face-off view, end zone view, etc.). Typically, camera system 102 may be used as a broadcast feed for a given match. Each frame of the broadcast source can be stored in match file 110.

[0025] Camera system 102 can be configured to communicate with organizational computing system 104 via network 105. Organizational computing system 104 can be configured to manage and analyze broadcast sources captured by camera system 102. Organizational computing system 104 may include at least a network client application server 114, a preprocessing engine 116, a data storage 118, and a camera calibrator 120. Both preprocessing engine 116 and camera calibrator 120 may include one or more software modules. One or more software modules may be a set of code or instructions stored on a medium (e.g., the memory of organizational computing system 104) representing a series of machine instructions (e.g., program code) that implement one or more algorithmic steps. Such machine instructions may be actual computer code interpreted by the processor of organizational computing system 104 to implement the instructions, or they may be higher-level encodings of the instructions interpreted to obtain actual computer code. One or more software modules may also include one or more hardware components. One or more aspects of the example algorithm may be executed by the hardware component (e.g., circuitry) itself, rather than as a result of instructions.

[0026] Data storage 118 can be configured to store one or more match files 124. Each match file 124 may include broadcast data for a given match. For example, the broadcast data may be multiple video frames captured by camera system 102.

[0027] Camera calibrator 120 can be configured to calibrate the cameras of camera system 102. For example, camera calibrator 120 can be configured to project players detected in trackable frames onto real-world coordinates for further analysis. Because the cameras in camera system 102 are constantly moving to focus on the ball or key play, such cameras cannot be pre-calibrated. Camera calibrator 120 can be configured to generate a homography matrix that can register the target ground plane field from any frame of broadcast video with a top-view model. For example, camera calibrator 120 can implement a single neural network to find a homography matrix H that can register the target ground plane field from any frame I of broadcast video with a top-view model M. In some embodiments, the standard objective function for calculating homography with point correspondence can be:

[0028]

[0029] in Indicates the position of pixel i in broadcast image I. , It is the corresponding pixel position on the model "image" M. This represents the set of point correspondences between two images I and M.

[0030] Client device 108 can communicate with organizational computing system 104 via network 105. Client device 108 can be operated by a user. For example, client device 108 can be a mobile device, tablet, desktop computer, or any computing system with the capabilities described herein. Users can include, but are not limited to, individuals such as subscribers, customers, potential customers, or consumers of entities associated with organizational computing system 104, such as individuals who have received, will receive, or may receive products, services, or advice from entities associated with organizational computing system 104.

[0031] Client device 108 may include at least application 132. Application 132 may represent a standalone application or a web browser that allows access to websites. Client device 108 may access application 132 to access one or more functions of organizational computing system 104. Client device 108 may communicate via network 105 to request web pages, such as those from network client application server 114 of organizational computing system 104. For example, client device 108 may be configured to execute application 132 to access content managed by network client application server 114. Content displayed to client device 108 may be transferred from network client application server 114 to client device 108 and subsequently processed by application 132 for display via graphical user interface (GUI) of client device 108.

[0032] Figures 2A-2B This is a block diagram illustrating a neural network architecture 200 of a camera calibrator 120 according to an example embodiment. As discussed, the camera calibrator 120 may utilize a single neural network that takes a video frame as input and outputs the homography matrix of that frame. For example, the camera calibrator 120 may utilize a single neural network to perform single-movement camera calibration given unknown camera intrinsic parameters across various sports (e.g., basketball, football, rugby, hockey, etc.). The neural network architecture 200 may include three modules: a semantic segmentation module 202, a camera pose initialization module 204, and a homography refinement module 206. Each of the three modules 202-206 is integrated into a single neural network architecture, as shown in neural network architecture 200. Because all three modules 202-206 are connected, the neural network architecture 200 is capable of end-to-end training.

[0033] The semantic segmentation module 202 can be configured to identify features of the playing field (e.g., a basketball court, a football field, etc.). For example, the semantic segmentation module 202 can be configured to extract key features from the input image I (reference numeral 220) and remove irrelevant information. Such an output can lead to a field-agnostic appearance that can be used to determine the corresponding point. (Refer to number 222). Therefore, the objective function H from the above can be rewritten as:

[0034]

[0035] in A vector representing the eight homography parameters. Indicates having transformation parameters The twist function, This represents measuring two images (in this case, the predicted semantic graph). Any loss function that differs between the overhead model M and the distorted overhead model M.

[0036] The semantic segmentation module 202 can perform region-based segmentation of the playing field by dividing it into one or more regions. By dividing the playing field into one or more regions, the semantic segmentation module 202 can transform the overhead field model M into a multi-channel image. Considering the multi-channel image, the semantic segmentation module 202 can classify each pixel in I into one of the one or more regions. To generate region-based semantic labels for each image, the semantic segmentation module 202 can utilize the associated ground reality homography distortion overhead model, thereby providing ground reality semantic labels for training.

[0037] Figure 3 This is a block diagram illustrating one or more images 302-306 of a basketball court according to an example embodiment. As shown, image 302 may correspond to a top-down view of the basketball court. Semantic segmentation module 202 may divide the basketball court into four regions, thereby generating a 4-channel image. For example, region 308 may correspond to the first channel, region 310 may correspond to the second channel, region 312 may correspond to the third channel, and region 314 may correspond to the fourth channel. In operation, semantic segmentation module 202 may use image 302 to classify each pixel in the input image (e.g., in I) into one of regions 308-314.

[0038] Image 304 can clarify the semantic labels applied to the incoming image. The semantic segmentation module 202 can generate image 304 by using a ground reality homography distortion field model M (e.g., image 302). These images (e.g., images 302 and 304) can then be used to train the semantic segmentation module 202.

[0039] Image 306 can be seen from the top perspective view, illustrating the polygonal region of the field model in the camera view as a scale (fraction) of image 304.

[0040] Return to reference Figure 2A and Figure 2B For segmentation tasks, the semantic segmentation module 202 can implement a Unet-style autoencoder 214 (hereinafter referred to as "Unet 214"). Unet 214 can take image I 220 as input and output... Required semantic graph 222. In some embodiments, cross-entropy loss can be used to train Unet 214. For example:

[0041]

[0042] in It can represent a set of categories. It can represent ground condition labels. It can represent the likelihood that pixel i belongs to category c.

[0043] The camera pose initialization module 204 can be configured to select an appropriate template from a template set using a semantic graph. The camera pose initialization module 204 can use a Siamese network to determine the optimal template for each input semantic image. The Siamese network can be a convolutional encoder that computes a hidden representation of the semantic image, which can be the output of Unet214 or any semantic template image. In some embodiments, the similarity between two images can be the norm L2 between their hidden representations. In some embodiments, each image can be encoded as a 128-bit vector for similarity calculation.

[0044] For a PTZ camera, the projection matrix P can be expressed as:

[0045]

[0046] Q and S are derived from the rotation matrix R, K are the intrinsic parameters of the camera in camera system 102, I is the 3×3 identity matrix, and C is the camera translation. Matrix S describes the rotation from world coordinates to the bottom of the PTZ camera, and Q represents the camera rotation due to translation and tilt. For example, S can be defined as a rotation of approximately 90° around the world x-axis, such that the camera views along the y-axis in the world plane. In other words, the camera is horizontal, and its projection is parallel to the ground.

[0047] In some embodiments, for each image, the camera calibrator 120 may assume a center principle point, square pixels, and no lens distortion. In some embodiments, six parameters may be identified. For example, these six parameters may be focal length, 3D camera position, translation, and tilt angle.

[0048] In some embodiments, the preprocessing engine 116 can initialize the intrinsic camera matrix K, camera position C, and rotation matrix R. Through this initialization, the preprocessing engine 116 can identify the optimal focal length, 3D camera position, and rotation angle. For example, the preprocessing engine 116 can use the Levenberg-Marquardt algorithm to find the optimal focal length, 3D camera position, and rotation angle. Once the preprocessing engine 116 determines K, C, R, and S, it can generate Q. In some embodiments, the preprocessing engine 116 can generate translation and tilt angles for a given Q. For example, the preprocessing engine 116 can generate translation and tilt angles by applying the Rodriguez formula to Q. Therefore, according to the above description, the camera pose initialization module 204 can generate a 6D camera configuration λ (translation, tilt, zoom, and 3D camera position).

[0049] After the preprocessing engine 116 estimates the camera configuration λ for each training image, the preprocessing engine 116 can generate possible camera poses. The dictionary.

[0050] In some embodiments, the preprocessing engine 116 can generate possible camera poses by uniformly sampling from a range of possible camera poses. The dictionary. For example, the preprocessing engine 116 can determine the range of translation, tilt, focal length, and camera position from the training data and uniformly sample the pose from a 6D grid. Even with a small training set, this method is able to cover all camera poses. Furthermore, using a smaller grid simplifies homography refinement because the maximum scale of the required transformation is on the scale of the grid size.

[0051] In some embodiments, the preprocessing engine 116 may use clustering to learn possible camera poses directly from the training data. For example, such a process can be beneficial when the training set has sufficient diversity. For instance, the preprocessing engine 116 can... The camera pose set is treated as a multivariate normal distribution and a Gaussian mixture model (GMM) is applied. In some embodiments, for each component, the mixture weights are... They can be fixed to be equal. In some embodiments, for each distribution, the covariance matrix... It can be fixed. In such an embodiment, The feature scale can be set to the scale of the transformation processed by the homography refinement module 206. Compared with the traditional GMM, the GMM learning algorithm implemented by the preprocessing engine 116 can achieve higher performance given the mixing weights. Covariance Matrix In this case, find the number of components K and the mean of each distribution. Instead of setting the number of components K, the same applies to each component. and This ensures that GMM components are sampled uniformly from the manifold of the training data.

[0052] In some embodiments, the GMM learning algorithm may be:

[0053]

[0054] Because preprocessing engine 116 can determine Therefore, the camera pose initialization module can only update during the maximization step. The preprocessing engine 116 can gradually increase K until a stopping criterion is met. The stopping criterion may aim to generate enough components such that each training example approximates the average of one component in the mixture. The preprocessing engine 116 can utilize all components. Generate camera pose dictionary .

[0055] Considering the camera pose dictionary The camera pose initialization module 204 can calculate the homography of each pose and use it. To distort the overhead field model M. Therefore, the image template set and its corresponding homography matrix It can be determined and used by the camera pose initialization module 204.

[0056] Considering semantic image segmentation and template image set The camera pose initialization module 204 can use a Siamese network to compute each input and template pair. The distance between them. In some embodiments, the targets / labels of each pair can be similar or different. For example, for a mesh-sampled camera pose dictionary, if the template If the pose parameters are the nearest neighbors in the mesh, then the template It may resemble an image. For a GMM-based camera pose dictionary, if the template... The corresponding distribution gives the highest likelihood of the pose parameter λ of the input image, and then the template can be... The images are labeled as similar to the original images. This process generates template similarity labels for each image in the training set.

[0057] Once (after FC1) the input semantic image and template images Encoded, the camera pose initialization module 204 can use the latent representation to compute the L2 distance between the input image and each template. The selection module 210 can find the target camera pose index. Furthermore, its template image can be retrieved using the following formula. and homography As output:

[0058]

[0059] in It can represent the encoding function of a Siamese network.

[0060] In some embodiments, the camera pose initialization module 204 may use a contrastive loss to train a Siamese network. For example...

[0061]

[0062] in Image pairs can be represented The binary similarity label, m can represent the boundary of the contrast loss.

[0063] The homography segmentation module 206 can be configured to refine homography by identifying the relative transformation between the selected template and the input image. For example, the homography segmentation module 206 can implement a spatial transformer network (STN), which allows for the processing of large non-affine transformations and the use of a small camera pose dictionary. For example, given an input image and a selected template, these two images can be stacked and provided as input to the STN. The STN can be used to regress geometric transformation parameters. In some embodiments, residual blocks can be used in the convolutional encoder to protect salient features used for deformation prediction. In some embodiments, ReLU can be used in all hidden layers, while the output layer of the STN can use linear activation.

[0064] To compute the input semantic image and the selected template image Based on the relative transformations between them, the homography segmentation module 206 can stack the images into an n-channel image (e.g., an 8-channel image) to form the input to the STN localization layer. In some embodiments, the output of the localization layer may be a semantic image. Mapping to template Relative homography The parameters (e.g., 8 parameters).

[0065] In some embodiments, the homography segmentation module 206 can initialize the last layer in the localization layer (e.g., FC3) such that all elements in the kernel are zero and biased towards the first n values ​​(e.g., 8 values) of the flattened identity matrix. Therefore, at the start of training, it can be assumed that the input is the same as the template, providing initialization for STN optimization. Thus, the final homography might be... .

[0066] Once H is calculated, the transformer 212 of the homography refinement module 206 can distort the overhead model M to the camera viewpoint and vice versa, allowing the camera calibrator 120 to compute the loss function. For example, the homography refinement module 206 can use dice coefficient loss:

[0067]

[0068] U and V can represent semantic images. It can represent the number of channels. It can represent element-wise multiplication. This can represent the sum of pixel intensities in an image. Here, for example, the intensity of each channel can be the likelihood that a pixel belongs to channel c. In contrast to line-based segmentation, one of the main advantages of using region-based segmentation is its robustness to occlusion and the ability to better utilize (i.e., more efficiently) network capacity, as a larger proportion of image pixels are likely to belong to meaningful categories.

[0069] However, a limitation of loss based on intersection-of-union (IoU) is that the IoU loss can become sensitive to segmentation errors as the proportion of the field of view in the image decreases. For example, if the playing field occupies a small portion of the image, a small transformation can significantly reduce the IoU. Therefore, the homography thinning module 206 uses dice loss on the distorted playing field in both viewpoints—a high-occupancy viewpoint allows for coarse registration, while a low-occupancy viewpoint provides strong constraints for fine-tuning. Thus, the loss function can be defined as:

[0070]

[0071] Where Y can represent a ground-based semantic image. A masking overhead field model can be represented so that the loss is computed only for the region shown in the image. The loss from both viewpoints can be obtained through... Weighted, in which the perspective with the lower occupancy ratio always has a higher weight.

[0072] Because each module 202-206 can use the output of other modules as input, these three modules 202-206 can be connected into a single neural network (i.e., neural network architecture 200). Therefore, the total loss of the network can be transformed into:

[0073]

[0074] in .

[0075] The camera calibrator 120 can incrementally train the entire neural network architecture 200 module by module, allowing the Siamese network and STN to begin training with appropriate inputs. For example, training can begin with a 20-epoch warm-up on UNet; Siamese network training can start from... and Enabled. For example, after another 10 epochs, STN can be started from... and Start. The neural network architecture may continue joint training until convergence.

[0076] Figure 4This is a flowchart illustrating a method 400 for generating a fully trained calibration model according to an example embodiment. Method 400 may begin at step 402.

[0077] In step 402, the organization computing system 104 may retrieve one or more datasets for training. Each dataset may include multiple images captured by the camera system 102 during the competition.

[0078] In some embodiments, the dataset can be created from thirteen basketball games. Those skilled in the art will recognize that more or fewer than thirteen games can be used for training purposes. For example, ten games can be used for training, and the remaining three for testing. Similarly, those skilled in the art will recognize that more or fewer than ten games can be used for training, and more or fewer than three games can be used for testing. The number of games used for training purposes described above is merely exemplary and is not intended to limit the discussion. Different games may have different camera positions, and each game is played on a unique court. Therefore, the appearance of the court for each game may vary from game to game. For each game, 30-60 frames can be selected for each annotation with high camera pose diversity. Professional annotators may have clicked on four to six point correspondences in each image to calculate ground realism homography. These annotations may have already generated 526 images for training and 114 images for testing. In some embodiments, the training data can be further enriched by horizontally flipping the images, thus generating a total of 1052 training examples.

[0079] In some embodiments, the dataset can be created from twenty football matches. For example, the twenty football matches are held in nine different stadiums during the day and at night, and the images may consist of different perspectives and lighting conditions. Therefore, the dataset may include 209 training images collected from 10 matches and 186 test images collected from another 10 matches.

[0080] In step 404, the organization computing system 104 can generate multiple camera pose templates based on one or more datasets. For example, based on one or more retrieved datasets used for training, the camera calibrator 120 can generate camera pose templates for training. In some embodiments, the camera calibrator 120 can generate camera pose templates using the GMM-based methods discussed above, provided that one or more datasets are large and diverse enough. In some embodiments, one or more datasets can be considered large and diverse enough when a complete and relatively clean overhead field image is obtained. In such embodiments, the camera calibrator 120 can set translation, tilt, focal length, and camera position. The standard deviation. In some embodiments, the camera calibrator 120 can also be set for stopping criteria and torsion loss. The threshold.

[0081] Continuing with the first example above, using the basketball dataset, camera calibrator 120 can use a GMM-based method to generate camera pose templates from 1052 training images. In such an example, camera calibrator 120 can adjust translation, tilt, focal length, and camera position. The standard deviations are set to 5°, 5°, 1000 pixels, and 15 feet, respectively. Off-diagonal elements can be set to zero because the camera calibrator 120 assumes those camera configurations are independent of each other. The threshold for the stopping criterion can be set to 0.6, and the clustering algorithm can generate 210 components. For the distortion loss... It can be set to 0.8 because the camera viewpoint may have a lower field of view than the overhead viewpoint.

[0082] In some embodiments, if, for example, one or more datasets have an insufficient number of examples, the camera calibrator 120 can generate a camera pose template using a high grid resolution. In such embodiments, the camera calibrator 120 can set the resolution of translation, tilt, and focal length.

[0083] Continuing with the second example above, using a football dataset, camera calibrator 120 can generate camera pose templates using a high-grid-resolution method. In such an example, camera calibrator 120 can set the resolutions for translation, tilt, and focal length to 5°, 2.5°, and 500 pixels, respectively. In some embodiments, the camera position can be fixed, for example, at 560, 1150, and 186 yards relative to the top left corner of the field. Because the football dataset has insufficient examples for using GMM-based camera pose estimation, camera calibrator 120 can generate 450 templates for camera pose initialization using uniform sampling of the dataset with estimated translation, tilt, and focal length ranges ([-35°, 35°], [5°, 15°], and [1500, 4500] pixels, respectively).

[0084] As those skilled in the art will recognize, although basketball and football have been discussed in the current example, this approach can be extended to video broadcasting of any sport.

[0085] In step 406, the organization computing system 104 can learn how to calibrate a single moving camera based on one or more training datasets. For example, the neural network of the camera calibrator 120 can learn how to calibrate a single moving camera based on one or more training datasets. In some embodiments, each module of the neural network architecture 200 can be trained simultaneously. For example, because each module 202-206 of the neural network architecture 200 uses the outputs of other modules as inputs, these three modules 202-206 can be connected into a single neural network. Therefore, the total loss of the network can become:

[0086]

[0087] in .

[0088] The camera calibrator 120 can incrementally train the entire neural network architecture 200 module by module, allowing the Siamese network and STN to begin training with appropriate inputs. For example, training can begin with a 20-epoch warm-up on UNet; Siamese network training can start from... and Enabled. For example, after another 10 epochs, STN can be started from... and Start. The neural network architecture may continue joint training until convergence.

[0089] In some embodiments, one or more modules 202-206 can be “warmed up” with synthetic data. For example, due to the limited number of training examples in the football dataset referenced above, camera calibrator 120 can use synthetic data to warm up camera pose initialization module 204 and homography refinement module 206. Apart from Unet in semantic segmentation module 202, the remainder of neural network architecture 200 uses semantic images as input, allowing camera calibrator 120 to synthesize any number of semantic images to pre-train parts of the network. Using a specific example, 2000 semantic images can be generated by uniformly sampling translation, tilt, and focal length parameters. For each synthetic image, their ground reality homography is known, and the template assignment can be easily found by downsampling the grid. Therefore, camera pose initialization module 204 and STN can be pre-trained separately. Once camera pose initialization module 204 and homography refinement module 206 are warmed up, camera calibrator 120 can train the neural network using real data.

[0090] In step 408, the organization computing system 104 can output a fully trained prediction model. For example, at the end of the training and testing process, the camera calibrator 120 can have a fully trained neural network architecture 200.

[0091] Figure 5 This is a flowchart illustrating a method 500 for calibrating a broadcast camera according to an example embodiment. Method 500 may begin at step 502.

[0092] In step 502, the organization computing system 104 may receive (or retrieve) a broadcast source of the event. In some embodiments, the broadcast source may be a live source received in real-time (or near real-time) from the camera system 102. In some embodiments, the broadcast source may be a broadcast of a match that has ended. Typically, the broadcast source may include multiple video data frames. Each frame may capture a different camera perspective.

[0093] In step 504, the organization computing system 104 can input each frame into the neural network architecture 200. For example, the camera calibrator 120 can identify the first frame in the received broadcast source and provide that frame to the neural network architecture 200.

[0094] In step 506, the organization computing system 104 can generate a homography matrix H for each frame. For example, the semantic segmentation module 202 can identify the stadium features Y in each frame. The output from the semantic segmentation module 202 can be a semantic graph generated by Unet. Semantic graph This can be provided as input to the camera pose initialization module 204. The camera pose initialization module 204 can use semantic graphs. Select an appropriate template from the template set. The camera pose initialization module 204 can also identify the target camera pose index. Use the selection module 210 to retrieve its template image and homography The camera calibrator 120 can... , and The concatenated items are passed as input to the homography refinement module 206. The homography refinement module 206 can then refine the concatenated items... and Passed to STN to predict the relative homography between the template and the semantic graph Then the homography refinement module 206 can use matrix multiplication (i.e., Generate based on relative homography and The homography matrix H.

[0095] In step 508, the organization computing system 104 can distort each frame using the corresponding homography matrix H for each frame.

[0096] Figure 6AA system bus computing system architecture 600 according to an example embodiment is illustrated. System 600 may represent at least a portion of an organizational computing system 104. One or more components of system 600 may communicate electrically with each other using bus 605. System 600 may include a processing unit (CPU or processor) 610 and a system bus 605 coupling various system components to processor 610, including system memory 615, such as read-only memory (ROM) 620 and random access memory (RAM) 625. System 600 may include a cache of high-speed memory directly connected to, adjacent to, or integrated into processor 610. System 600 may copy data from memory 615 and / or storage device 630 to cache 612 for fast access by processor 610. In this way, cache 612 can provide performance improvements by avoiding delays for processor 610 while waiting for data. These and other modules may control or be configured to control processor 610 to perform various actions. Other system memory 615 may also be used. Memory 615 may include various different types of memory with different performance characteristics. Processor 610 may include any general-purpose processor or hardware or software module configured to control processor 610 (e.g., service 1 632, service 2 634, and service 3 636 stored in storage device 630) and dedicated processors whose software instructions are integrated into the actual processor design. Processor 610 may essentially be a completely independent computing system, containing multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0097] To enable user interaction with computing device 600, input device 645 can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, voice input, etc. Output device 635 can also be one or more of a variety of output mechanisms known to those skilled in the art. In some cases, a multi-mode system allows the user to provide multiple types of input to communicate with computing device 600. Communication interface 640 typically controls and manages user input and system output. There are no limitations on operation on any particular hardware device; therefore, during development, the basic features here can be easily replaced with improved hardware or firmware devices.

[0098] Storage device 630 may be non-volatile memory and may be a hard disk or other type of computer-readable medium that can store data accessible by a computer, such as magnetic tape, flash memory card, solid-state storage device, digital multifunction disk, dark box, random access memory (RAM) 625, read-only memory (ROM) 620 and combinations thereof.

[0099] Storage device 630 may include services 632, 634, and 636 for controlling processor 610. Other hardware or software modules are expected. Storage device 630 may be connected to system bus 605. On the one hand, hardware modules performing a specific function may include software components stored in a computer-readable medium that are connected to necessary hardware components such as processor 610, bus 605, and display 635 to perform that function.

[0100] Figure 6B A computer system 650 is shown having a chipset architecture that can represent at least a portion of an organizational computing system 104. The computer system 650 can be an example of computer hardware, software, and firmware that can be used to implement the disclosed techniques. The system 650 may include a processor 655 representing any number of physically and / or logically distinct resources of software, firmware, and hardware capable of performing identified computations. The processor 655 can communicate with a chipset 660, which can control inputs to and outputs from the processor 655. In this example, the chipset 660 outputs information to an output 665 (e.g., a display), and can read and write information to a storage device 670, which may include, for example, magnetic media and solid-state media. The chipset 660 can also read data from and write data to RAM 675. A bridge 680 may be provided for connection to various user interface components 685 for connection to the chipset 660. Such a user interface component 685 may include a keyboard, microphone, touch detection and processing circuitry, pointing devices such as a mouse, etc. Typically, input to system 650 can come from any of a variety of machine-generated and / or artificially generated sources.

[0101] Chipset 660 can also be connected to one or more communication interfaces 690, which may have different physical interfaces. Such communication interfaces may include interfaces for wired and wireless local area networks, broadband wireless networks, and personal area networks. Some applications of the methods disclosed herein for generating, displaying, and using GUIs may include receiving ordered datasets via physical interfaces, or data stored in memory 670 or 675 may be analyzed by the machine itself via processor 655 to generate applications for generating, displaying, and using the methods disclosed herein. Furthermore, the machine may receive input from a user via user interface component 685 and perform appropriate functions, such as browsing functions, by interpreting this input using processor 655.

[0102] It is understood that the example systems 600 and 650 may have more than one processor 610 or be part of a group or cluster of computing devices networked together to provide greater processing power.

[0103] While the foregoing describes embodiments herein, other and further embodiments may be designed without departing from its basic scope. For example, aspects of this disclosure may be implemented in hardware or software or a combination of hardware and software. The embodiments described herein may be implemented as a program product for use with a computer system. The program of the program product defines the functionality of the embodiments (including the methods described herein) and may be contained on a variety of computer-readable storage media. Illustrative computer-readable storage media include, but are not limited to: (i) non-writable storage media that permanently store information (e.g., read-only memory (ROM) devices within a computer, such as CD-ROM disks readable by a CD-ROM drive, flash memory, ROM chips, or any type of solid-state non-volatile memory); and (ii) writable storage media that store variable information (e.g., floppy disks or hard disk drives in a floppy disk drive or any type of solid-state random access memory). Such computer-readable storage media are embodiments of this disclosure when carrying computer-readable instructions that direct the functionality of the disclosed embodiments.

[0104] Those skilled in the art will understand that the foregoing examples are exemplary and not restrictive. Upon reading the specification and studying the accompanying drawings, all substitutions, enhancements, equivalents, and modifications will be apparent to those skilled in the art and are included within the true nature and scope of this disclosure. Therefore, the appended claims are intended to include all such modifications, substitutions, and equivalents that fall within the true nature and scope of these teachings.

Claims

1. A method for calibrating a broadcast video source, comprising: The computing system receives broadcast video sources of sports events, the broadcast video sources comprising multiple frames; The computing system inputs the plurality of frames into a trained neural network, wherein the trained neural network is configured to generate a corresponding homography matrix for each of the plurality of frames; The computing system calibrates the broadcast video source, wherein calibrating the broadcast video source includes: Based on the corresponding homography matrix, each frame in the plurality of frames is warped to a high occupancy viewpoint and a low occupancy viewpoint, wherein warping each frame in the plurality of frames includes generating one or more images and one or more semantic labels; and Calculate a loss function among the distorted frames, wherein calculating the loss function includes weighting the distorted frames according to the high occupancy view and the low occupancy view; and The computing system further trains the trained neural network using the one or more images, the one or more semantic labels, and the loss function.

2. The method of claim 1, wherein the trained neural network comprises: Semantic segmentation module; Camera pose initialization module; as well as Unigraphy refinement module.

3. The method according to claim 2, wherein the semantic segmentation module is configured to generate a semantic graph.

4. The method of claim 3, wherein the camera pose initialization module is configured to select a template from the template set using the semantic graph to generate the homography matrix.

5. The method of claim 4, wherein the homography refinement module is configured to generate the homography matrix based on the template and the semantic graph.

6. The method according to claim 1, wherein the homography matrix registers the target ground plane surface of any one of the plurality of frames with the top-view model.

7. The method of claim 1, wherein the broadcast video includes a top-down model, and wherein the top-down model includes projecting one or more three-dimensional positions of at least one of one or more players or balls onto a two-dimensional top-down view of the stadium of the sporting event.

8. The method of claim 7, wherein distorting each of the plurality of frames comprises distorting the top-down model using the homography matrix.

9. The method of claim 4, wherein the camera pose initialization module selects the template from the template set using a twin network.

10. A system for calibrating a broadcast video source, comprising: processor; as well as A memory storing programming instructions that, when executed by the processor, perform one or more operations, including: Receive a broadcast video source of a sports event, wherein the broadcast video source comprises multiple frames; The plurality of frames are input into a trained neural network, wherein the trained neural network is configured to generate a corresponding homography matrix for each of the plurality of frames; Calibrate the broadcast video source, wherein calibrating the broadcast video source includes: Based on the corresponding homography matrix, each frame in the plurality of frames is warped to a high occupancy viewpoint and a low occupancy viewpoint, wherein warping each frame in the plurality of frames includes generating one or more images and one or more semantic labels; and Calculate a loss function among the distorted frames, wherein calculating the loss function includes weighting the distorted frames according to high-occupancy and low-occupancy perspectives; and The trained neural network is further trained using the one or more images, the one or more semantic labels, and the loss function.

11. The system of claim 10, wherein the trained neural network comprises: Semantic segmentation module; Camera pose initialization module; as well as Unigraphy refinement module.

12. The system of claim 11, wherein the semantic segmentation module is configured to generate a semantic graph.

13. The system of claim 12, wherein the camera pose initialization module is configured to select a template from the template set using the semantic graph to generate the homography matrix.

14. The system of claim 10, wherein the homography matrix registers the target ground plane surface of any one of the plurality of frames with the top-view model.

15. The system of claim 10, wherein the broadcast video includes a top-view model, and wherein the top-view model includes projecting one or more three-dimensional positions of at least one of one or more players or balls onto a two-dimensional top-view of the stadium of the sporting event.

16. The system of claim 15, wherein distorting each of the plurality of frames comprises distorting the top-view model using the homography matrix.

17. The system of claim 13, wherein the camera pose initialization module selects the template from the template set using a twin network.

18. A non-transitory computer-readable medium comprising one or more sequences of instructions, said sequences of instructions, when executed by one or more processors, causing: The computing system receives broadcast video sources of sports events, the broadcast video sources comprising multiple frames; The computing system inputs the plurality of frames into a trained neural network, wherein the trained neural network is configured to generate a corresponding homography matrix for each of the plurality of frames; The computing system calibrates the broadcast video source, wherein calibrating the broadcast video source includes: Based on the corresponding homography matrix, each frame in the plurality of frames is warped to a high occupancy viewpoint and a low occupancy viewpoint, wherein warping each frame in the plurality of frames includes generating one or more images and one or more semantic labels; and Calculate a loss function among the distorted frames, wherein calculating the loss function includes weighting the distorted frames according to high-occupancy and low-occupancy perspectives; and The computational system further trains the trained neural network using the one or more images, the one or more semantic labels, and the loss function.

19. The non-transitory computer-readable medium of claim 18, wherein the homography matrix registers the target ground plane surface of any one of the plurality of frames with a top-view model.

20. The non-transitory computer-readable medium of claim 18, wherein the broadcast video includes a top-view model, and wherein the top-view model includes one or more three-dimensional positions of at least one of one or more players or balls projected onto a two-dimensional top view of the stadium of the sporting event.

Images