Systems and methods for robotic camera calibration - Patents.com

Abstract

The processor and camera capture data corresponding to the position of the ball and automatically adjust the camera's orientation and zoom level. The processor pre-calibrates the camera so that an initial orientation is known in a world coordinate system (WCS) and different zoom levels are associated with respective parameter values. A first position of the ball in a first image is detected, and a first zoom level of the first image and intrinsic parameter values ​​for the first zoom level are retrieved. A first orientation of the camera is determined based on the pan and tilt relative to the initial orientation. A 3D line through the camera and the ball in the WCS is determined based on the first position, and a first orientation and parameter values ​​for the first zoom level and a 3D position are determined in the WCS along the line based on extrinsic information for the camera.

Description

[Technical field]

[0001] Priority claim This application claims priority to U.S. Provisional Patent Application No. 63 / 200,425, filed March 5, 2021, and U.S. Provisional Patent Application No. 63 / 202,850, filed June 27, 2021. The specifications of the above-identified applications are incorporated herein by reference.

[0002] The present disclosure relates to systems and methods for tracking objects, including golf balls and golf players, during a golf event and determining statistics for use in broadcasting the golf event. [Background technology]

[0003] Detailed statistics of golf shots can be captured during amateur play or during professional tournaments such as PGA Tour events. For example, radar-based tracking systems can be used to capture shot data for tee shots while the ball is in flight. However, many systems currently in use are unable to determine more advanced shot metrics, including, for example, the ball's final resting position after bouncing and rolling. In professional tournaments, these determinations are currently made manually by human operators, using, for example, laser range finders. This is a very labor-intensive task, typically requiring hundreds of operators. In addition to the generation of specific shot data, many operations performed during a successful and visually pleasing broadcast of a golf event require extensive manual input from human operators. Summary of the Invention

[0004] The present disclosure relates to systems, methods, and processing devices for calibration of robotic cameras. The system includes a robotic camera configured to capture data corresponding to a position of a sports ball, the robotic camera further configured to automatically adjust an orientation and zoom level of the robotic camera in response to the captured data or a command; and a processing device coupled to the robotic camera configured to: pre-calibrate the robotic camera such that an initial orientation of the robotic camera is known in a world coordinate system, and associate each of a plurality of different zoom levels used by the robotic camera with a respective intrinsic parameter value; detect a first image position of the sports ball in an image coordinate system of a first image, the first image being captured by the robotic camera at the first orientation; read from the robotic camera a first zoom level associated with the first image and the intrinsic parameter value associated with the first zoom level; determine a first orientation of the robotic camera based on a pan and tilt of the robotic camera relative to the initial orientation; determine a three-dimensional line that passes through the robotic camera and the sports ball in the world coordinate system based on the detected first image position of the sports ball, the intrinsic parameter value of the determined first orientation and the first zoom level; and determine a three-dimensional position of the sports ball in the world coordinate system that lies along the three-dimensional line based on extrinsic information for the robotic camera. [Brief description of the drawings]

[0005] [Figure 1] FIG. 1 shows example layouts 100-115 for tracking units of four example holes.

[0006] [Diagram 2] FIG. 2 illustrates an example diagram 200 for an example golf hole 250 tracking system, in accordance with various example embodiments.

[0007] [Diagram 3]FIG. 3 illustrates an example diagram 300 for a golf course tracking system in accordance with various exemplary embodiments.

[0008] [Figure 4] FIG. 4 illustrates an example tracking unit 400 including a radar system 405 and / or a camera system 410 and / or a lidar system 415, in accordance with various exemplary embodiments.

[0009] [Figure 5a] FIG. 5a shows a first-principle example of MFCW Doppler radar tracking.

[0010] [Figure 5b] FIG. 5b shows a second principle example of MFCW Doppler radar tracking.

[0011] [Figure 5c] FIG. 5c shows a third principle example of MFCW Doppler radar tracking.

[0012] [Figure 6] FIG. 6 illustrates a diagram 600 of example data packets 605-630 containing golf shot specific data that may be transmitted to a broadcast entity and / or other applications during real-time data capture of a golf shot in accordance with various example embodiments.

[0013] [Figure 7] FIG. 7 shows an example image 700 of a club head from a position behind the player.

[0014] [Figure 8a] FIG. 8a shows a shot trajectory overlay, ie a first image 800 from a video feed depicting the shot including a first tracer 805 of the shot, and some associated trajectory data.

[0015] [Figure 8b]FIG. 8b shows a second image 850 from the video feed depicting the shot including a shot trajectory overlay, ie, a second tracer 855 of the shot, and some associated trajectory data.

[0016] [Figure 9] FIG. 9 shows example images 900-915 that can be analyzed by a neural network to detect events related to a player's swing.

[0017] [Figure 10a] FIG. 10a shows an example image 1000 captured by a robotic camera 1005 for determining the orientation of the robotic camera 1005 in world coordinates 1020.

[0018] [Figure 10b] FIG. 10b illustrates a process for determining an orientation of the robotic camera 1005 based on the first image 1000 of FIG. 10a and a second image 1030 captured by the robotic camera 1005 using a different orientation.

[0019] [Figure 10c] FIG. 10c illustrates a process for determining an orientation of the robotic camera 1005 based on the first image 1000 of FIG. 10a and a second image 1040 captured by the second camera 1035.

[0020] [Figure 10d] FIG. 10d illustrates a process for determining an orientation of the robotic camera 1005 based on the first image 1000 of FIG. 10a and a second image 1030 that includes a zoomed-out version of the first image 1000.

[0021] [Figure 11]FIG. 11 shows an example tracking system 1100 including at least one camera 1105 that captures a series of images of the ball, including the ball at rest, and a processing unit for tracking the ball and / or determining the position of the ball at rest, in accordance with various example embodiments.

[0022] [Figure 12] FIG. 12 illustrates an example diagram 1200 including a camera 1105, a golf course terrain 1205, and a corresponding 3D model 1135 superimposed on the golf course terrain 1205 in accordance with various exemplary embodiments.

[0023] [Figure 13] FIG. 13 shows a time series 1300 of ball detection in the (u,v) image plane from a sequence of images and a corresponding time series 1350 of the elevation angle relative to the camera-ball line determined by the non-projector 1115 described in FIG.

[0024] [Figure 14] FIG. 14 shows a plot 1400 including a starting location 1405, a hole location 1410, and lines 1415-1425 that represent potential putt trajectories that result in a successful putt.

[0025] [Figure 15] FIG. 15 shows a putt break fan 1500 as a diagram 1510 of putt launch direction and launch speed combinations that result in a successful putt, and as contour lines 1515, 1520 showing the launch direction and launch speed combinations that result in a ball position with an equal distance to the pin for the second putt.

[0026] [Figure 16a] FIG. 16a illustrates an example method 1600 for tracking the three-dimensional position of a launched ball during play on a hole on a golf course, where the tracking is performed using data from both a first sensor and a second sensor, and the operation of the second sensor is controlled based on 3D position tracking information determined from data captured by the first sensor.

[0027] [Figure 16b] FIG. 16b illustrates a method 1610 for transmitting tracking data for inclusion in a broadcast according to various exemplary embodiments.

[0028] [Figure 16c] FIG. 16c illustrates a method 1630 for identifying a player and associating shot data with a particular player profile associated with a unique player ID, according to various exemplary embodiments.

[0029] [Figure 16d] FIG. 16d illustrates a method 1650 for automatic broadcast feed switching, according to various exemplary embodiments.

[0030] [Figure 16e] FIG. 16e illustrates a method 1660 for automated generation of custom (eg, personalized) broadcast feeds, according to various exemplary embodiments.

[0031] [Fig. 16f] FIG. 16f illustrates a method 1670 for automatic insertion of a tracer into a broadcast feed, according to various exemplary embodiments.

[0032] [Figure 16g] FIG. 16g illustrates a method 1680 for calibrating a robotic camera to a world coordinate system according to various exemplary embodiments.

[0033] [Figure 16h] FIG. 16h illustrates a method 1690 for a warning system for venue spectators, according to various exemplary embodiments.

[0034] [Figure 17a] FIG. 17a illustrates a method 1700 for determining 3D position coordinates of a ball, according to various exemplary embodiments.

[0035] [Figure 17b] FIG. 17b illustrates a method 1720 for determining the bounce and roll of a moving ball based on image data, according to various exemplary embodiments.

[0036] [Figure 17c] FIG. 17c illustrates a method 1740 for determining upcoming putt information according to various exemplary embodiments.

[0037] [Figure 17d] FIG. 17d illustrates a method 1750 for determining topographical parameters of a putting green based on tracked putts according to various exemplary embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0038] Exemplary embodiments may be further understood with reference to the following description and associated accompanying drawings, in which like elements are given the same reference numerals. Exemplary embodiments relate to systems and methods for identifying and tracking objects on a golf course, including golf shots, golf clubs, and golf players. Multiple techniques relating to different aspects of object tracking are described that, when used in combination, can provide complete coverage of relevant moving objects on a golf course. According to some aspects, data generated in accordance with the described techniques can be used to enrich and enhance live broadcast coverage of golf events, e.g., PGA Tour events. Notably, nearly all of the described techniques are performed automatically, requiring little or no manual input from operators on the golf course.

[0039] In a typical round of golf, the statistics registered may only include the number of shots taken by the player on each hole. This registration of strokes is typically done manually. In some professional tournaments, including those sponsored by the PGA Tour, more detailed statistics are recorded, such as the ball's position and lie for each shot. In recent years, ball trajectory data from radar-based tracking systems, such as the TrackMan system, have also been used by the PGA Tour to capture shot data for tee shots. These systems generally measured the three-dimensional position of the ball in flight until impact with the ground. However, the final resting position is generally determined manually by an operator using a surveyor-type laser range finder. Determining the final resting position is very labor intensive, requiring hundreds of operators to track all the shots throughout all the rounds of a typical four-day tournament on the PGA Tour. In some scenarios, volunteers are also tasked with pressing buttons on hand-held devices to capture the times of shots made in the tournament.

[0040] Compared to existing methods for capturing data during professional tournaments, embodiments according to the present disclosure enrich data sets, enhance the captured data, and automate many tasks that are currently performed manually. These solutions automate nearly all operations during professional tournaments and may be applied to any golf course, making the same or similar data available to amateur golfers playing at the local golf course.

[0041] Embodiments according to the present disclosure perform fully automated or semi-automated tracking of all shots on a golf course, assigning each golf shot to a unique player ID. As a result, all shots from the same player are tagged with the same player ID, unique to that player. In addition to tracking the flight, bounce and roll of each golf shot from launch to resting place, various key tags are collected by the tracking system for each shot, including: the number of the golf hole played or other identifier, and data regarding the lie of the ball, the shot number (e.g., the number of shots the player took on this hole, including the current shot, up to the current shot), the time of impact when the shot was hit, the type of club used in the current shot, and environmental conditions such as wind and temperature.

[0042] A system according to the present disclosure can automatically record all shots made by all players during a golf round. Those skilled in the art will appreciate that certain systems can also receive manual input when desired. For example, in tournament play, a camera can be moved to the fairway behind the player preparing the current shot (e.g., the camera is further from the hole than the ball the player is preparing to hit). This camera can be manually and automatically calibrated and positioned, as described in more detail below, and tracking information can be inserted into or derived from images from the camera. Additionally, such manually positioned cameras can also include mobile units that provide additional tracking information to the system. As will be appreciated by those skilled in the art, manual repositioning or targeting of the camera (and any attached radar or other tracking unit) can be seamlessly integrated with automatic tracking and image manipulation, as desired, to create a semi-automatic tracking system.

[0043] Recording every shot during a golf round provides a complete and compelling data set for the player and / or third parties. However, it should be understood that the present solution can be used for only a portion of a golf round, such as several complete golf holes, selected tee shots, etc., if desired. Furthermore, the recording and / or dissemination of this data may be performed only for a selected group of golfers or only for one golfer, and may be offered, for example, as an additional service (e.g., at an additional cost). As described in more detail below, a player identification system may be used to track individual players and associate data related to that player, such as shot data captured during play, with the individual players. Golf ball and / or golf club statistics captured during play may be associated with a particular golfer with a unique player ID. The captured statistics may include any information related to the game of golf, including: carry location, final resting location, trajectory including bounce and roll, ball launch data (speed, angle, direction, spin rate and spin axis), number of shots on the hole, hole number, lie of the ball for each shot, impact time, club used, swing data, etc. Of course, data from such a system may be aggregated and used to evaluate different ball and / or club types under various conditions and / or with different player anatomies or swing types, etc. Such aggregate data may also be used to evaluate course design (or redesign) and / or selection of tee and pin placements.

[0044] In some embodiments, the full course tracking system can be used during professional golf tournaments to enhance the coverage of the event by providing real-time data that can be overlaid on the broadcast feed, including, for example, tracers for shots, advanced shot metrics, landing predictions, and the like. This information can also be made available to players for use in improving their game or to evaluate differences in performance between practice and tournament conditions. However, the same system can also be used during any amateur golf round, whether during a golf tournament, practice round, or any other time. During amateur rounds and practice rounds, golfer groupings and playing schedules may be available to the system, e.g., via a tee reservation system or similar, or this information may not be available. In scenarios where the tracking system is available to golfers during non-scheduled tee times, the system may include some mechanism for obtaining consent from the golfers before tracking their play. This consent may be given before the round, or possibly after the round.

[0045] Exemplary embodiments described below relate to concepts including: a tracking system for full course tracking (tracking of every or nearly every shot taken during an entire tournament of one or more rounds of golf); a tracking unit device usable in the tracking system, including specifications and operations for radar-based, camera-based, lidar-based, and / or other sensors operable to identify and / or track golf shots and / or golf clubs; a tracking unit device and operation specifically configured for tracking the bounce and roll of golf shots; a calibration scheme for the tracking unit and / or other sensors; a broadcast system for broadcasting a video feed of a golf event operable in conjunction with the tracking system, including calibration of broadcast cameras with the tracking system's sensors; a player identification system; an automatic broadcast feed that can be tailored to a particular player; an automatic tracer; automatic detection of player events, including when a player is about to take a shot; calibration and control of robotic cameras based on real-time tracking data provided by the tracking system; a safety / warning system for on-site spectators; a scheme for amateur round tracking; and generation of non-fungible tokens (NFTs). Throughout the description, power saving schemes will be described where, for example, tracking units and / or sensors are triggered to power on based on tracking data provided by different tracking units that suggests, for example, that an object will come into the field of view of a powered-off tracking unit. In some preferred embodiments, the world coordinate system and common time reference used for time synchronization is used by the tracking system across the golf course, for example to enable rapid transfer of data from one sensor to the local coordinate system of another sensor.

[0046] System Overview The present disclosure relates to a tracking system that includes multiple tracking units distributed throughout a golf course. The placement of the tracking units may depend on multiple factors, including, for example, the layout of the hole, the capabilities of the tracking units, the desired tracking coverage of the hole, or other considerations that are described in detail below. It should be understood that the term "hole" as used herein may relate to the entirety of a golf hole, including, for example, the tee box, fairway, rough, green, hazards, bunkers, etc., in accordance with common golf terminology. The actual hole located on the green where the golf ball is intended to ultimately sink and be marked with a flag may be referred to as a "cup" herein, although common golf terminology also refers to this feature as a "hole." Those skilled in the art will appreciate the distinction between a broader definition of "hole" that encompasses the entire golf hole, and a narrower definition of "hole" that encompasses only the cup located on the green and marked with a flag.

[0047] FIG. 1 illustrates example layouts 100-115 for tracking units for four example holes. As illustrated in FIG. 1, the configuration of system components for each of the holes depends on the length and layout of the hole. Typically, each hole has a first tracking unit located near the tee box of the hole, and depending on the length and layout of the hole, additional tracking units distributed at different locations ensure strong tracking coverage at any location where a golf ball may be expected to be located during play of the hole. It may also be advantageous to place tracking units at locations where a high degree of accuracy and / or quality tracking is expected to be obtained. Additional tracking units may be desirable to avoid / minimize occlusion by golfers, trees, or other objects that obstruct the line of sight of the primary tracking unit. Layout 100 illustrates a par-3 hole with a single tracking unit, in this example, a radar unit (Radar 1), located near the tee box of the hole. The radar unit in this example may provide full tracking coverage of the entire par-3 hole. Layout 105 shows a par 4 hole with two tracking units, in this example a first radar unit (radar 1) and a second radar unit (radar 2), located near the tee box of the hole and near the green of the hole, respectively. The two radar units in this example can provide complete tracking coverage of the entire par 4 hole. Layout 110 shows a par 5 hole with two tracking units, in this example a first radar unit (radar 1) and a second radar unit (radar 2), located near the tee box of the hole and near the green of the hole, respectively. The two radar units in this example can provide complete tracking coverage of the entire par 5 hole. Layout 115 shows a par 5 hole with three tracking units, in this example a first radar unit (radar 1), a second radar unit (radar 2), and a third radar unit (radar 3), located near the tee box of the hole, along the fairway of the hole, and near the green of the hole, respectively.The three radar units in this example can provide complete tracking coverage of an entire par 5 hole.

[0048] FIG. 2 illustrates an example diagram 200 for a tracking system of an example golf hole 250, according to various example embodiments. The layout of each golf hole typically includes a teeing area or tee box 255, a fairway 260, a green 265 generally surrounding the fairway 260, and a green 265. The green 265 includes a cup where the golf ball is intended to finish for a player to complete the golf hole 250, which is indicated by a flag 270. In this example, the golf hole 250 further includes a water hazard 275 and a bunker 280. The tracking system of diagram 200 includes a first tracking unit 205 located near the tee box 255 and a second tracking unit 210 located behind the green 265. As mentioned above, for a particular hole, one or more additional tracking units may be distributed at different locations surrounding the hole. The tracking units 205, 210 provide position and / or movement data of a golf ball 215 and / or a golf club being used, which will be described in more detail below. All position data may ultimately be determined in a world coordinate system 220 that is used to generate tracks throughout a golf course, for example, although some sensors may determine some position data in a coordinate system specific to the hole and / or the local coordinate system of the sensor, which are described in more detail below.

[0049] The diagram 200 further illustrates an exemplary path of the ball 215 during play of the hole 250. The ball 215 starts at a first stationary position 215a on the tee box 255 and is hit along a trajectory 215b to a second stationary position 215c on the fairway 260. The ball 215 is then hit along a trajectory 215d to a third stationary position 215e on the green 265. From there, the ball 215 can be putted into a cup marked by a flag 270. A system according to the exemplary embodiments described herein can accurately recognize the ball in motion (215b, 215d) and / or at rest (215a, 215c, 215e). In some embodiments, the first tracking unit 205 may be a tee box (tee area) tracking unit with a first capability selected to track tee shots (e.g., trajectory 215b launched from stationary position 215a) and derive certain information, e.g., launch parameters, from these tee shots. The second tracking unit 210 may be a green (green side) tracking unit with a second capability selected for tracking approach shots (e.g., trajectory 215d launched from stationary position 215c and ending at stationary position 215e) and / or chips and putts, and extraction of specific information from these shots, such as, for example, putt parameters. A third type of tracking unit may comprise a fairway tracking unit with a third capability selected for tracking shots landing on and launched from the fairway 260, while remaining portable for frequent location changes. However, as described in more detail below, multiple different types of tracking units may be used, including various combinations of sensors and capabilities.

[0050] 3 illustrates an example diagram 300 for a golf course tracking system, according to various exemplary embodiments. In diagram 300, only three golf holes 350 are shown, e.g., a first hole 350a (hole 1), a second hole 350b (hole 2), and a third hole 350c (hole 3). However, it should be understood that the principles described in example diagram 300 can be extended to any number of holes and can encompass an entire golf course including any number of holes, e.g., 9, 18, 27, etc. In diagram 300, three players 355 are shown, e.g., a first player 355a is shown on the first hole 350a, a second player 355b is shown on the second hole 350b, and a third player 355c is shown on the third hole 350c. As described in more detail below, in addition to tracking golf shots, the tracking system described herein can accurately identify players and associate those players with golf shots tracked by the system, e.g., using a player ID. The exemplary tracking system shown in Figure 3 is additionally equipped to broadcast a video feed of the golf course, for example, when the golf course is used for professional events such as PGA Tour events. However, it should be understood that the broadcast aspect of the tracking system is optional, and the tracking system may be used for a variety of purposes unrelated to the broadcast of professional events. Certain techniques related to the broadcast of events, including data transmission delivered to the broadcast (e.g., for player / shot information / tracking), broadcast camera calibration, automatic feed switching, etc., are described in more detail below.

[0051] Each hole in FIG. 3 is equipped with multiple tracking units 305, providing tracking coverage of the hole, and connected to a tracking server 320 via wired or wireless connections 310. In this example, multiple broadcast cameras 315 are also located on each hole that are also connected to the tracking server 320 via wired or wireless connections 310. A first hole 350a includes a first tracking unit 305a and a first broadcast camera 315a adjacent to the tee box, and a second tracking unit 305b and a second broadcast camera 315b adjacent to the hole. A second hole 350b includes a third tracking unit 305c and a third broadcast camera 315c adjacent to the tee box, a fourth tracking unit 305d along the fairway, and a fifth tracking unit 305e adjacent to the hole. The third hole 350c includes a sixth tracking unit 305f and a fourth broadcast camera 315d adjacent the tee box, and a seventh tracking unit 305g and a fifth broadcast camera 315e adjacent the hole. It should be understood that the locations and number of tracking units 305 and broadcast cameras 315 shown in Figure 3 are provided for illustrative purposes only, and that any number of tracking units 305 and broadcast cameras 315 may be used depending on the desired tracking coverage / accuracy, the layout of the holes on the course, and / or the desired broadcast capabilities of the broadcasting entity, e.g., the PGA Tour.

[0052] Additional tracking units having similar, reduced or different capabilities than the primary tracking unit 305 shown in FIG. 3 may be implemented for full course tracking, according to various considerations described in detail below. In one example, one or more mobile tracking units used for fairway tracking may have reduced capabilities compared to the tracking units used for tee areas and / or greens. Furthermore, those skilled in the art will appreciate that some very short shots (e.g., misses where blocked by spectators, trees, etc.) may not be captured by the system. To address the possibility of missed shots, the system may have manual correction / additional input capabilities. For example, the system may ask each player to confirm the number of shots upon completion of a hole. In such a situation, there will be no data for missed shots, but the score will remain accurate.

[0053] The tracking process may be performed by a processing unit embedded in each tracking unit 305, by the tracking server 320, or by any of a variety of steps performed in cooperation between the embedded processing unit and the tracking server 320. The tracking server 320 may be, for example, an on-site server or a cloud-based processing service. The tracking server 320 includes processing and storage devices and is coupled to the tracking units 305, the broadcast cameras 315, and optionally other sensors or devices that may be used in the full-course tracking system and / or the broadcast system, which are described in more detail below. The processing device of the tracking server 320 may include one central processor or multiple processors; in some embodiments, the tracking server 320 may handle the tracking of tens or hundreds of objects simultaneously, which requires relatively high computing power, in addition to various additional functions described in more detail below. In some embodiments, the tracking server 320 may also function as or cooperate with a broadcast system providing video coverage of a live event, e.g., a PGA Tour event, such as by outputting / switching video feeds, audio feeds, etc. to present a cohesive viewing experience to the viewer. In other embodiments, the tracking server 320 may provide information to a broadcast entity responsible for the content displayed to the viewer. In this scenario, the tracking server 320 and the broadcast entity may act in coordination such that live data can be inserted into the broadcast in substantially real-time, as described in more detail below.

[0054] The tracking server 320 is responsible for associating each ball (in motion or at rest) with the individual player 355 playing that ball, and each player on the course is associated with a unique player ID. The tracking server 320 is further responsible for transmitting the associated ball tracking information along with the player ID of the player who hit the tracked ball to various consumers. When the tracking system is used for broadcast purposes, the tracking server 320 can transmit the tracking information along with the player ID to consumers such as the broadcaster 325, web application 330 and / or on-site application 335 in coordination with the broadcast camera data. When the tracking system is used primarily for informational purposes, i.e., not associated with a broadcast, the information may be transmitted to an on-site database, for example for later retrieval by the individual players, to individual players via, for example, a mobile application for a mobile device used by the player, or to other interested parties. The player identification system used to identify players and associate them with data captured by the tracking system is described in more detail below.

[0055] Tracking Unit The exemplary tracking unit 305 shown in FIG. 3 may include one or more sensors, e.g., an imaging (e.g., camera-based) system including one or more non-image-based sensors (e.g., microphones, radar and / or lidar systems, etc.) and one or more cameras. The tracking unit 305 may include only non-image-based sensors, only image-based sensors, or a combination of non-image-based and image-based sensors. For example, a camera system may include multiple camera sensors with different characteristics, such as field of view, resolution, orientation, frame rate, wavelength sensitivity, electronically controllable zoom (analog or digital), fixed or movable, fixed or electronically adjustable pan-tilt orientation, etc. The camera sensors may include passive optical receiving sensors, time-of-flight cameras, event-based cameras, etc., and may also include visual, infrared (IR) or near-infrared (NIR) illumination synchronized with the camera frame exposure to increase the signal-to-noise ratio (SNR) of the illuminated scene. Similar to a camera system, a non-image-based sensor (e.g., radar and / or lidar) may include multiple sensors with different characteristics, e.g., different field of view, orientation, resolution, frame rate, wavelength, modulation, etc. For ease of discussion, the disclosed embodiments refer to the non-image-based components of the tracking unit 305 as radar units, however, those skilled in the art will appreciate that such components may be replaced by or include additional non-image-based sensors, as desired.

[0056] The tracking system may include any mixture of different tracking units having different sizes and configurations. Some tracking units may include a combination of radar and camera systems, while other tracking units may include only radar systems or only camera systems. For example, the angular accuracy of the radar unit is typically related to the separation between the various antennas, so a greater separation may produce a higher angular accuracy. In some cases, a larger radar unit may be required to achieve the desired accuracy, while other radar units may be smaller to provide portability or a smaller visual footprint. In one case, the fairway tracking unit may be relatively small and portable so that an operator can change the position and orientation of the tracking unit depending on the various positions of shots in its vicinity, while the tee box tracking unit and the green tracking unit may be larger and substantially fixed in view of the high likelihood or certainty that these tracking units provide good coverage of each particular shot hit on each hole, e.g., tee shots and chips / putts. Additionally, some tracking units may include robotic cameras that can automatically change orientation based on, for example, commands from a tracking server (based on tracking data from a different tracking unit or sensor) or based on tracking data from the same tracking unit. The robotic camera may be a tracking camera that performs optical tracking and changes orientation based on ball detection, or the robotic camera may be a broadcast camera that relies on tracking data obtained from other sensors.

[0057] In addition to orientation, zoom level and crop can be controlled. In some embodiments, additional sensors (e.g., radar or lidar) are included with the robotic camera on a single tracking unit, and all sensors included in the tracking unit can change orientation together. As can be appreciated by those skilled in the art, a radar unit moving with the robotic camera may continue to track the ball within a portion of the radar's field of view (e.g., near its center) where the radar has increased sensitivity, for example, due to the fact that such radars often receive stronger reflected signals from objects moving near the center of their field of view.

[0058] Each tracking unit 305, by itself or in combination with other tracking units 305, provides position and / or motion data of the golf ball and / or golf club used by each player in its field of view. As described in more detail below, some tracking units 305 may also identify players in its field of view and associate ball / club data with the corresponding player. As mentioned above, all position data may be converted to a world coordinate system to identify the location of the ball and / or player on the golf course. Tracking of the ball trajectory and related parameters, e.g., ball spin rate and spin axis, may be performed based on radar data alone, or image-based tracking may be combined with radar-based tracking to determine the three-dimensional location of the ball trajectory. Furthermore, pure image-based tracking, or any other tracking technology such as LIDAR, may be used in some scenarios. Preferably, these applications would employ non-invasive technology that does not require modification of the equipment used by the golfer (e.g., markings on the ball or club), as it would be unlikely to be adopted in any professional tournament. This may be less important in an amateur scenario where players are amenable to using different equipment if it results in lower costs of service, etc. Additionally, club tracking such as club speed, attack angle, club path, face-to-path angle, and impact location on the club face may be implemented using either radar-based tracking, image-based tracking, lidar-based tracking, or any combination of these and other systems.

[0059] FIG. 4 illustrates an example tracking unit 400 including a radar system 405 and / or a camera system 410 and / or a lidar system 415 according to various exemplary embodiments. The tracking unit 400 may include any combination of the aforementioned sensors, as described above. The various sensors are operable to capture data according to their configured parameters. For some sensors, the captured data may be sent directly to a processing unit, such as, for example, the tracking server 320 described above with respect to FIG. 3 or any other data processing unit that processes the data to determine a track in a coordinate system 420, which may be a local coordinate system or a world coordinate system depending on the application. Other sensors include an internal processing unit to perform some or all of the data processing on the captured data. The tracking unit 400 may be located adjacent to a tee box and configured to capture launch parameters of a tee shot 440 of a ball 425 by a player 435, including both launch parameters of the ball 425 and club path parameters of a club 430 used by the player 435. The tracking unit 400 may also be positioned further along the hole and configured primarily for 3D position tracking.

[0060] In a preferred embodiment, an exemplary tracking unit includes a 3D Doppler radar system and a camera system, typically including one or two built-in cameras. The 3D Doppler tracking radar can determine the 3D position of a moving object in XYZ coordinates even when prior information (e.g., launch position) is not available. This means, for example, that a ball suddenly appearing from behind a tree can be precisely located in 3D as soon as line of sight is established by the radar. However, predetermined information regarding launch position can be used if available.

[0061] In this embodiment, the camera is time-synchronized to the radar data and calibrated to the radar's coordinate system. This means that at any given time, the 3D position of the ball in motion determined by the radar can be accurately mapped to the frame captured by the camera that corresponds to the time and position of the image. Similarly, the 3D position determined for a ball in motion determined from the image data can be correlated with the corresponding radar data to more accurately determine, for example, the track. In an alternative embodiment, instead of a 3D Doppler radar, one or more of the tracking units may include a 1D Doppler radar and camera system. In this example, the 1D radar measures only the radial distance or range rate of the ball as it traverses the radar's field of view, while the camera system determines the angular position of the ball as it traverses the camera's field of view. When these respective fields of view overlap, the data may be combined to generate a three-dimensional track of the object, as described in U.S. Pat. No. 10,989,791, which is incorporated herein by reference in its entirety.

[0062] Preferably, each camera is intrinsically and extrinsically calibrated, meaning that both the internal and external parameters of the sensor are known or determined. The internal parameters of the camera may generally include, for example, focal length, lens distortion parameters, and principal point, while the internal parameters of the radar may generally include, for example, phase offset between the receivers. The extrinsic parameters typically constitute the position and orientation of the sensor. Intrinsic and extrinsic calibration for the sensor in the tracking unit (or in some embodiments across different units in the tracking system) allows the angular orientation of each pixel of the image captured by the camera to be known in conjunction with the calibration of the radar system. Knowledge of the distortion parameters for one or more cameras allows for de-warping (correction) of the image / video, if desired, to make it appear more pleasant to the human eye. Various applications of this de-warping function are described in more detail below, particularly with respect to tracking the bounce and roll of golf shots, which are difficult to capture using radar alone.

[0063] When more than one camera is embedded in the tracking unit, the cameras are typically configured with different fields of view (FOV) and operated at different frame rates. For example, a first camera may be configured with 4K resolution, with a horizontal FOV of 50 degrees, operating at 30 fps, to provide a visually pleasing video stream. The first camera may be used, for example, as a broadcast camera or for generating clips of shots. A second camera, for example, a greenside tracking unit looking back towards the fairway, may have a narrower field of view providing higher pixel resolution at a distance, and may be configured to track shots aimed at the green.

[0064] This precise alignment and calibration of the cameras and radar allows for radar tracking and image tracking, which complement and enhance each other. In some scenarios, balls, club heads, and / or people are first detected by the radar, while in other scenarios, these objects / bodies are first detected by one of the cameras. In many scenarios, both sensor types will always have reliable tracking of an object when it is within their respective fields of view. This allows each sensor to be used in different scenarios where each sensor is most accurate, and when available, data from multiple sensors can be combined to increase accuracy compared to measurements based on only one type of sensor.

[0065] The preferred system is further provided with techniques that allow for time synchronization between the tracking units, as described in more detail below, as well as an advanced calibration scheme in which all units are calibrated to the same coordinates (e.g., the world coordinate system). In essence, the combination of sensor data and cross-sensor enhancement possible within a single tracking unit, as described above, is also available across multiple tracking units working in concert, e.g., a first tracking unit tracks an early part of the ball's flight while a second tracking unit takes over a later part of the flight. Those skilled in the art will appreciate that for a portion of the ball's flight that is included in both the fields of view of the first and second tracking units (overlapping field of view area), both data may be combined. Alternatively, the system may identify, for example, a first portion of the overlapping field of view area where the data from the first tracking unit is more accurate, a second portion of the overlapping field of view area where the combination of data from the first and second tracking units provides the most accurate and / or most reliable data, and a third portion of the overlapping field of view area where the data from the second tracking unit is more accurate. The delineation of these first, second and third portions of the overlapping field of view region may be done on a case-by-case basis, taking into account, for example, distance from each first and second tracking unit, occlusion factors, etc. Those skilled in the art will appreciate that the same general principles allow for similar alternations between different modes of use of data from various tracking units when the fields of view of three or more tracking units overlap.

[0066] In some embodiments, real-time 3D tracking using data from the first sensor can be used to control the operation of the second sensor. For example, the system can detect events in the real-time 3D tracking information from the data from the first sensor and use these detected events as a basis for triggering the second sensor to start tracking. These events include, for example: the launch of a shot; the ball passing a certain distance from the tee; the ball moving beyond a predetermined distance from the reference line; and / or other events indicating that the ball has entered or will enter the field of view of the second sensor. The events that trigger a change in the operating state of the second sensor can be selected and configured based on the layout of a given hole. For example, a playing theory can be used that considers the geography of the hole, including items that may block the view / coverage of the first or second sensor, such as the length of the hole. Additionally, the capabilities of each sensor can be taken into account, such as which sensors provide good tracking coverage for a given hole and which do not. The system may be programmed to begin tracking with the second sensor when the ball passes a particular distance from the tee, and / or moves laterally more than a predetermined distance from the reference line, and / or moves closer to the second sensor than the first sensor. In some embodiments, the sensors may have fully or partially overlapping fields of view, although this is not required.

[0067] The detected events may also be used to more efficiently manage power usage for different sensors and subsystems of the tracking system. When an event requiring a particular system capacity is not currently detected in the data from a first sensor or other system components, the power usage of other sensors, such as sensors located further down the hole, may be reduced, for example, by placing various power-drawing components of the item in a sleep or low power mode until the system detects an event requiring full activation of the item or an immediate pending event. This may have great practical importance, since golf course systems are often either battery-operated or generator-powered.

[0068] In a related embodiment, a robotic camera, e.g., a tracking camera and / or a broadcast camera, remotely controlled by a tracking server, is controlled relying on the real-time 3D position tracking information. Using the real-time position data for the ball, the tracking system can control the robotic camera to point it at the current ball position (e.g., track the ball through its flight) and additionally control the zoom and focus to ensure a clear, viewer-friendly image that the viewer is accustomed to (e.g., provide a broadcast video feed) in the same way as achieved by a professional camera operator during a golf tournament. The 3D position of the ball relative to the robotic camera is always known by the tracking system based on real-time data acquired by other sensors in the tracking system. Thus, the optimal crop, orientation, and zoom level for the robotic camera can be automatically controlled by the system based on the real-time data. Optical tracking of the ball in flight may also be implemented using analysis of data from the robotic camera (or any other camera) and used to control the robotic camera to achieve a desired image using any known method. The robotic camera may be used to replace a manual operator of the camera, making the video footage much more appealing to the viewer. It also makes it possible to create close-up footage, for example to depict the lie of the ball. To ensure a stable and viewer-friendly movement of the robotic camera, a specially filtered version of the real-time 3D data from the tracking system can be generated and used to control the robotic camera. Furthermore, knowledge of the characteristics of the robotic camera control, such as delay, maximum angular acceleration, maximum angular velocity, focus delay, etc., can be taken into account and compensated for when controlling the camera. The robotic camera can also be used as a tracking sensor for the positioning of the club and the ball in world coordinates. However, this requires a calibration of the robotic camera, since the field of view is not stationary. These embodiments are described in more detail below.

[0069] For example, in one embodiment, the second sensor may be a camera (e.g., a robotic camera) and the activation command may include parameters to control the optimal crop, orientation, and zoom level of the robotic camera. The zoom and focus of the robotic camera may be continuously updated based on in-flight tracking of the ball by other sensors or other tracking units, even if the ball is not visible or easily found in the image from the camera.

[0070] FIG. 16a illustrates an exemplary method 1600 for tracking a three-dimensional position of a launched ball during play on a hole of a golf course, where the tracking is performed using data from both a first sensor and a second sensor, and the operation of the second sensor is controlled based on 3D position tracking information determined from data captured by the first sensor. In some embodiments, the first sensor is a radar (e.g., a radar of a tee box tracking unit). In other embodiments, the first sensor may be a radar of a different type of tracking unit, e.g., a fairway tracking unit, or may be a camera. In some embodiments, the second sensor is a radar, e.g., of a fairway tracking unit. In other embodiments, the second sensor is a camera. In some embodiments, the second sensor may be co-located with the first sensor, e.g., the second sensor is a camera in the tee box tracking unit and the first sensor is a radar in the tee box tracking unit. In a preferred embodiment, the first and second sensors are calibrated to a world coordinate system.

[0071] In the following, certain processing steps are described as being performed in the tracking system. As mentioned above, the tracking process may be performed by a processing unit embedded in each tracking unit, by a tracking server, or by any of a variety of steps performed in cooperation between the embedded processing unit and the tracking server. As mentioned above, the tracking server may be a physical server on-site at the golf course or a server included in a cloud-based service or any combination of such elements.

[0072] At 1602, the tracking system captures first data using a first sensor, the first data corresponding to a first portion of a trajectory of a launched ball, and calculates a 3D position of the ball in real time as the data is collected. The second sensor is in a reduced or different capacity operating state while the first data is captured by the first sensor. For example, the second sensor may be in a low power state. In another example, the second sensor may be fully powered on but not yet performing data capture and / or tracking functions. In yet another example, the second sensor may be a robotic camera that captures data according to initial camera parameters for, for example, crop, orientation, and zoom level.

[0073] At 1604, the tracking system detects a launched ball event based on the first data. In one embodiment, the 3D position of the ball is used as a basis for determining whether an event is detected relative to the second sensor. For example, an event may be detected if the 3D position of the ball meets one or more position criteria. As discussed above, these events may include: the ball passing a particular distance from the tee; the ball moving more than a predetermined distance from the reference line; and / or other events that indicate that the ball has entered or will enter the field of view of the second sensor. In another example, an event may include the launch of a shot.

[0074] At 1606, the tracking system controls the operational state of the second sensor (e.g., activates the second sensor and adjusts data acquisition parameters of the second sensor) based on the detected event such that the second sensor captures second data corresponding to a second portion of the trajectory of the fired ball. In some embodiments, the activation command may include a command to fully power on and / or begin data capture for 3D position tracking. In other embodiments, the activation command may include parameters for controlling a robotic camera. In still other embodiments, the activation command may power a processing module for tracking the bounce and roll of the shot after initial impact with the ground, as described in further detail below with respect to FIGS. 11-13; 17a-b. In other embodiments, the region of interest may provide an area where the ball is likely to be found so that the second sensor can detect the ball more quickly and using fewer processing resources.

[0075] The method 1600 described above is particularly related to activating a sensor to capture data and / or track a ball in flight based on three-dimensional tracking data of the ball in flight derived from data captured by another sensor, however, the general principles described above are applicable to many other exemplary embodiments described in more detail below.

[0076] For example, in one embodiment, the second sensor may be a camera (e.g., a robotic camera) and the activation command may include parameters to control the optimal crop, orientation, and zoom level of the robotic camera. The zoom and focus of the robotic camera may be continuously updated based on in-flight tracking of the ball by other sensors or other tracking units, even if the ball is not visible or easily found in the image from the camera.

[0077] In another example, detection of an event may relate to detection of a player, e.g., the identity of a player in a video feed, specific movements taken by a player leading up to a shot, etc., and a broadcast feed or some other processing module is triggered based on the event detection. For example, a radar may be triggered based on detection of a player ready to hit a shot from the tee box.

[0078] In yet another example, certain data packets are triggered for transmission to a broadcast (e.g., a broadcast entity) based on real-time 3D position tracking of the ball. Metrics related to the flight of the ball are processed and / or transmitted when certain events are detected, such as launch parameters calculated at launch and transmitted thereafter; peak point parameters transmitted immediately after peak point detection; smooth trajectory parameters calculated at first impact with the ground and transmitted thereafter; bounce and roll and / or final rest position parameters calculated when the ball is detected bouncing and rolling and / or at rest, etc.

[0079] In yet another example, event detection can relate to an estimated landing location of a launched ball. If the landing location where players and / or spectators are located is estimated based on data acquired at the start of the shot, an automatic warning (an automatic "fore" warning) can be triggered so that these persons can protect themselves from errant shots. These and additional embodiments are described in more detail below.

[0080] Radar Tracking In radar technology, tracking of the ball in flight is typically done using a multi-frequency continuous wave (MFCW) Doppler radar operating in the X-band. X-band Doppler radar is a powerful technology for tracking golf balls in flight. Under good conditions, the ball can be tracked to distances of over 300 meters, and the radar can function in all light and weather conditions.

[0081] Figure 5a shows a first principle example of MFCW Doppler radar tracking. The radar wave is transmitted at a particular transmission frequency F TX It is transmitted by the unit at 100 Hz and reflected by the ball at a return frequency F RX The wave received by the unit at has a Doppler shift V proportional to the radial velocity of the ball relative to the tracking unit. R The frequency shift can be detected by mixing the transmitted and received signals and performing a frequency analysis. Various signal processing algorithms can be applied to detect and track the ball and ultimately estimate the trajectory and various data points.

[0082] Figure 5b shows an example of the second principle of MFCW Doppler radar tracking. Angle measurements can be performed by an MFCW Doppler radar that includes multiple receiving antennas. The antennas are arranged so that the wavefront reflected from the ball reaches the two receiving antennas with a time difference that depends on parameters including the direction from the unit to the ball and the distance / direction between the antennas. A system that includes three or more receiving antennas spanning a plane approximately perpendicular to the main beam and tracking direction can perform angle measurements in both horizontal and vertical directions, producing three-dimensional tracking.

[0083] After mixing, the time shift can be observed as a phase shift factor of 2π between the signals in the receive channel. The 2π ambiguity can be resolved by having three or more receive antennas strategically spaced in a two-dimensional grid, as described in U.S. Patent No. 9,958,527, which is incorporated by reference in its entirety. Multiple receive antennas can also increase the signal-to-noise ratio of the received signal when signals from multiple receivers are added in an orderly fashion.

[0084] Figure 5c shows an example of the third principle of MFCW Doppler radar tracking. Range measurements can be performed using multiple transmission frequencies. The wavefronts reflected from the ball, club, or any other item being tracked for each of the two frequencies arrive at a common receiving antenna with different phases that depend on the distance to the ball and the frequency separation. After mixing, the phase difference can be observed as a phase shift factor 2π between the signals in the receiving channel from two or more different transmitter frequencies. The 2π ambiguity can be resolved with two or more frequencies or from a given knowledge of the approximate distance to the measured item. For example, for the ball trajectory, a priori information about the distance to the ball at a particular part of the trajectory may be used, e.g., the range measurement at launch may be assumed to be within a given range interval of a radar located at the tee box.

[0085] As an alternative to transmitting multiple frequencies to determine the range to the ball, a frequency modulated or phase modulated continuous wave (CW) radar, such as an FM-CW radar, can be used, as is well known to those skilled in the art.

[0086] The main advantages of radar tracking technology are its speed, robustness, and maturity; the ability to directly measure, rather than guess, the three-dimensional position of a golf ball at any time during the ball's entire flight; and a vast and growing data record from which detailed aerodynamic models can be constructed that represent the flight of a golf ball.

[0087] Furthermore, the tracking units can be equipped for advanced sensor fusion of radar and camera data and are inherently scalable. Where additional accuracy is important, multiple tracking units may be employed, for example combining a tee area system with a greenside system (as well as additional mid-hole tracking units, if desired) to provide extremely high accuracy in key events (e.g., launch and / or landing) for every shot hit on a hole.

[0088] At every instant that data is captured by the radar (e.g., at approximately 20 ms intervals), a determination can be made of the radial velocity; range; horizontal and vertical angles; 3D position; and any additional information described herein, for all moving objects visible to the radar.

[0089] When a sufficient number of such detections of the golf ball occur consecutively within a time period typically ranging from 100 to 500 ms, the launch of the ball is confirmed and an aerodynamic model is fitted to the data. The aerodynamic model may, for example, include constrained parametric partial difference equations based on Newtonian physics that fit the data in a maximum likelihood sense, with appropriate weighting of the data. If the radar does not directly detect the impact time in the raw radar data, the model may be extrapolated backwards in time to the expected impact location or height. Alternatively, data from other sensors (e.g., microphones) may be used to determine the launch time. Launch parameters may then be extracted by evaluating the model at the estimated time of impact. Launch parameters may include, for example, a timestamp of impact; stationary ball position prior to impact; ball velocity; vertical launch angle; and horizontal launch angle. The aerodynamic model thus serves to smooth the raw radar data and provide a suitable representation of the ball's flight thus far.

[0090] The remaining flight of the ball is tracked, for example, using a nonlinear Kalman filter, increasing the likelihood of tracking the entire flight of the ball until impact with the ground. Furthermore, if line of sight to the ball is occluded for any period of time, the signal from the ball may be reacquired once line of sight is re-established. Once it lands, the bounce can be tracked continuously if visible on radar or by an image-based tracking unit or image-based components of a tracking unit. A detailed description of the methodology for tracking the bounce and roll of the ball and determining its final resting position is provided further below.

[0091] Live data captured by the system may be transmitted continuously during the ball's motion, including data captured by the radar unit, to the tracking server 320 for broadcast after processing at the tracking unit, and / or directly to a third party. Similarly, data received at the tracking server 320 may be continuously processed and transmitted to a third party or broadcast. Alternatively, data may be transmitted only at key points in the shot, such as at launch, at the apex of the carry, at the landing of the carry, or when the ball comes to rest.

[0092] FIG. 6 illustrates a diagram 600 of exemplary data packets 605-630 containing specific data of a golf shot transmitted to a broadcast entity and / or other applications during acquisition of real-time data of the golf shot according to various exemplary embodiments. The data packets 605-630 are illustrated against an exemplary plot of the trajectory of the shot over time, and approximate transmission times of the data packets are provided relative to the launch time of the shot. Additionally, approximate transmission times of the data packets 605-630 are illustrated and differentiated for different types of radar sensors. For example, the approximate time of transmission may vary between radar sensors having different parameters and / or processing capabilities, requiring different amounts of processing time for the data packets, and / or typically tracking different types of shots that have shorter or longer times from launch to rest. For present purposes, any latency required for data transmission, e.g., from the tracking unit 305 to the tracking server 320 or broadcast / application and / or from the tracking server 320 to the broadcast / application, may be ignored. In this embodiment, for clarity, the tracking server 320 is presumed to receive data packets from the tracking units and transmit the data packets to a receiving broadcast entity broadcasting live video of the shot from any number of vantage points. The tracking server 320 may receive data / tracks from a single tracking unit or multiple tracking units. As processing of the raw radar signal occurs at the tracking server 320, data packets 605-630, shown in FIG. 6, are generated by the tracking server 320 and transmitted for broadcast.

[0093] The live video may include a tracer of the shot, determined in substantially real-time using techniques described in more detail below. During the broadcast of the shot, the information contained in the received data packets may be displayed simultaneously with the display of the tracer to augment the broadcast with real-time information for the shot in progress. Those skilled in the art will appreciate that a tracer generally refers to a video feed graphic showing a visual representation of the trajectory of the shot from launch to the position of the ball in the current image, inserted at positions in the image corresponding to the positions in the image occupied by the ball. The tracer graphic may include numerical data shown at various points of the trajectory, e.g., ball speed, peak height, carry, etc.

[0094] The first data packet 605 transmitted from the tracking server to the broadcast includes launch data, e.g., impact time, ball position at rest, ball speed at launch, and both horizontal and vertical launch angles. To process and transmit these parameters to the broadcast entity over some communication medium, a typical tracking unit (combined with a tracking server) may transmit, for example, with a latency of less than about 600 ms (for a tee box tracking unit), about 1000 ms (for a green tracking unit), or about 650 ms (for a fairway tracking unit). That is, the data is processed, included in the image, and ready for broadcast in a manner that requires minimal latency (very close to "real time") from purely live timing, where efficiency of data collection and processing is crucial. The launch data can be displayed on the broadcast as numbers or graphics overlaid on the video feed, or in a variety of other ways that are included in the rendering of the hole / shot in addition to being stored for later manipulation and analysis.

[0095] The second data packet 610 transmitted from the tracking unit to the broadcast includes real-time trajectory data, e.g., the three-dimensional position of the ball, which data is captured and determined. At any time during the flight of the ball, the tracking unit 305 may begin transmitting this real-time trajectory information in data packets 610, and may proceed with transmitting these data packets 610 at predefined intervals, which may be limited by the capabilities of the tracking unit and / or the transmission medium over which the data is communicated. The transmission may end after some predefined time, when the shot reaches some predefined position, e.g., its highest point, or upon transmission of a further data packet containing additional information. In the embodiment of FIG. 6, the transmission of the real-time trajectory data packet 610 is triggered to begin upon transmission of the first data packet 605 containing the launch data. The transmission of the real-time trajectory data packet 610 ends when the third data packet 615 is transmitted, as described below. However, one skilled in the art will appreciate that the real-time trajectory data packet 610 may be configured for transmission for any time during the flight of the shot.

[0096] The nonlinear Kalman filter may be smoothed at every opportunity where data is sent to the broadcast for live rendering of the trajectory. This results in the most accurate posterior estimate of the ball's trajectory according to all data recorded so far. The smoothed trajectory may be further filtered by an automatic regression process before display to improve the viewer experience. The trajectory data may be displayed in a variety of ways, including, for example, as a trajectory plot on a rendering of the hole. The numerical data may also be used directly, for example, as counters of distance, curve, height, etc.

[0097] An optional third data packet 615 transmitted from the tracking unit 305 to the broadcast includes a landing prediction. The landing prediction may include an estimated three-dimensional landing position of the ball based on information collected about the ball's flight so far. The landing prediction may also provide uncertainty information about the prediction. The tracking unit may specify to transmit the third data packet 615 after the launch time, for example, about 1500 ms (for a tee box tracking unit), the highest point of the shot (for a green tracking unit), or about 1500 ms (for a fairway tracking unit). The landing prediction may be displayed on the broadcast in a variety of ways, including, for example, as a location on a rendering of the hole. A fourth data packet 620 transmitted from the tracking unit 305 to the broadcast includes a spin rate. The tracking unit may specify to transmit the fourth data packet 620 after the launch time, for example, about 2500 ms (for a tee box tracking unit). In the green tracking unit and / or fairway tracking unit, spin rate may not be transmitted to the broadcast; spin rate determination is a relatively complex calculation, and for short shots that are generally captured by green and fairway units, this information may be omitted. The spin rate determination may be displayed on the broadcast, for example, as a numerical value or in any desired graphical format.

[0098] The fifth data packet 625 transmitted from the tracking unit 305 to the broadcast includes the live peak, including, for example, the height, range, and side of the peak. The peak side refers to the lateral distance of the ball from a known or assumed target line of the shot, for example, a direction toward the pin on a short hole, or toward the center of the fairway at a particular distance from the tee box on a long hole. Alternatively, the peak side can be determined relative to the initial launch direction of the ball. Regardless of the type of sensor included in the tracking unit 305, the tracking unit can specify that the fifth data packet 625 is transmitted when the tracking unit 305 determines that the peak of the shot has been reached. The live peak determination can be displayed on the broadcast, for example, as a number or graphic overlaid on a rendering of the hole.

[0099] The sixth data packet 630 sent from the tracking unit 305 to the broadcast contains the final measurements of the shot after the ball lands. Once the ball hits the ground, a more complex aerodynamic model than the model used at launch can be fitted to all the collected data to provide a determination of the most likely trajectory of the ball from impact through culmination to landing. The aerodynamic model can capture the spin falloff of the ball, in addition to quantifying the way in which the spin and velocity of the ball affect the instantaneous drag and lift on the ball. The model can further take into account the effects of weather and wind. In one embodiment, the aerodynamically smoothed trajectory can be used to estimate the effective wind speed and direction that will affect the flight of the ball.

[0100] The model is then evaluated in place to extract values ​​for all final data points including: timestamp of impact; ball position at rest before impact (if measured); ball speed; vertical launch angle; horizontal launch angle; peak height, side, and range; estimated spin axis (if not measured); spin rate (if not measured); carry and side (flat); landing angle and velocity (flat); and flight time (flat). The "flat" values ​​for carry and side, landing angle, velocity, and flight time refer to values ​​determined at the same height / level the ball was launched at. Flat values ​​are useful for comparing shots that were not launched / landed at the same location. The aerodynamic model is further used to extract real values, rather than flat values, for flight time; landing angle; landing velocity; range and side by intersecting the trajectory with a known 3D representation of the golf hole layout. The "real" values ​​refer to values ​​determined at actual impact of the ball with the ground taking into account elevation differences. If desired, the aerodynamic model can be summarized in a set of polynomials, allowing for easy navigation, visualization, and manipulation by a third party.

[0101] The sixth data packet 630 may include any desired key metrics, including smoothed trajectory data, and more accurate versions of those metrics determined / predicted and transmitted in previous data packets 605-625. To process and transmit these parameters to a broadcast entity via some communication medium, a typical tracking unit 305 may request or otherwise specify that the sixth data packet 630 be transmitted, for example, after about 300 ms (for a tee box tracking unit), about 350 ms (for a green tracking unit), or about 350 ms (for a fairway tracking unit) relative to the time of impact with the ground. The final shot measurements may be displayed on the broadcast in a variety of ways, including any of the ways described above with respect to data packets 605-625.

[0102] It should be understood that the particular configurations and transmission times of the data packets 605-630 described above are provided for illustrative purposes, and that any more or fewer similar or different data packets may be sent to the broadcast based on any reason, for example, based on a particular request from the broadcaster, the capabilities of the tracking unit, etc. For example, packet transmission architectures specific to different sensors may be configured such that the sensors transmit different data at different times depending on factors such as the processing time of the shot parameter determination, the type of shot typically tracked, etc. Additional data related to the bounce and roll of the shot after landing may be provided, for example, in the sixth data packet 630 along with the final measurement data of the trajectory, or in a separate packet at a different time. Particular techniques for analyzing the bounce and roll of a shot are described in more detail below with respect to FIG. 13.

[0103] 16b illustrates a method 1610 for transmitting tracking data for inclusion in a broadcast, according to various exemplary embodiments. As discussed above, the tracking system may include a radar unit for tracking the launch and / or flight and / or impact of the ball with the ground (and in some embodiments, the radar unit may track the bounce and roll after the initial impact with the ground). The tracking system may also include a camera for tracking the bounce and roll. Data may be acquired for the entire shot trajectory, and the various tracking units may work in coordination with a tracking server to calculate the 3D trajectory of the ball and detect events related to the flight of the ball.

[0104] At 1612, a launch event is detected by the tracking system and launch data for the launched ball is captured using a sensor in a first tracking unit. The tracking unit (and / or tracking server) begins processing the launch data immediately after launch to derive launch parameters.

[0105] At least a first data packet is transmitted to the broadcast entity containing the determined launch parameters, at 1614. The first data packet may be transmitted immediately after the launch parameters are determined at the tracking server (or received at the tracking server after determination at the tracking unit) or at a predetermined time after launch.

[0106] At 1616, a plurality of second data packets are transmitted to the broadcast entity including the real-time 3D trajectory data. In one embodiment, the transmission of the second data packets may be triggered at a predetermined time after launch and end at a predetermined time after start. In another embodiment, the transmission of the second data packets may be triggered at the time of the transmission of the first data packet. In yet another embodiment, the transmission of the second data packets may be triggered when a 3D position event of the ball is detected, e.g., when the ball reaches a particular point on its trajectory, e.g., a predetermined height or distance from the tee. In yet another embodiment, the end of the transmission of the second data packets may be triggered when a 3D position event of the ball is detected, e.g., when the ball reaches a particular point on its trajectory, e.g., a height determined as a percentage of the highest point or predicted highest point.

[0107] At 1618, additional data packets may be transmitted to the broadcast entity while the ball is in flight. For example, a third data packet may include a predicted landing location (e.g., transmitted after transmission of the real-time trajectory data at 1616 has ended), a fourth data packet may include a spin rate (e.g., transmitted after a predetermined time has elapsed after flight or as soon as sufficient data has been acquired and processed to accurately determine the spin rate), and a fifth data packet may include a live apex point (transmitted, e.g., upon detection).

[0108] At 1620, an impact event is detected by the tracking system (e.g., the system detects that the in-flight ball first impacts the ground), and ball trajectory data is accumulated across sensors and / or fitted to an aerodynamic model to determine a final measurement for the ball's trajectory (which may be more accurate than the measurement provided in the previous data transmission).

[0109] At least one sixth data packet is transmitted to the broadcast entity containing the final measurements for the ball's trajectory, at 1622. The sixth data packet may be transmitted immediately after the final measurements are determined or at a predetermined time after impact with the ground.

[0110] At 1624, the ball's landing data, including the bounce and roll and / or final rest position, may be tracked by the tracking unit and / or a further tracking unit, for example a camera in the vicinity of the impact with the ground. The tracking unit (and / or tracking server) processes the bounce and roll during this portion of the ball's motion and can determine 3D position data and further determine a rest position, as described in more detail below with respect to Figures 11-13.

[0111] At least one seventh data packet is transmitted to the broadcast entity containing the landing information, at 1626. The seventh data packet may be transmitted immediately after the landing information is determined or at a predetermined time after the ball comes to rest.

[0112] At 1628, the broadcast entity receives each of the transmitted packets during the ball's flight and after impact / come to rest, and presents the data contained in the data packets on a broadcast video feed. The information can be processed into visually pleasing graphics, etc. The bounce and roll data can be combined with the initial trajectory data to display the ball's complete path of travel from launch to rest.

[0113] Although the method 1610 above involves seven types of data packet transmissions, one skilled in the art will appreciate that a different number and / or type of data packets may be used.

[0114] Certain phenomena can impede the quality of the radar tracking of the ball, for example: shots with particularly low ball speeds; shots that travel nearly perpendicular to the line of sight to the radar unit; strong multipath reflections, primarily from the ground; objects that block the direct line of sight to the radar; and external devices that emit electromagnetic interference. Most phenomena that affect the quality of the radar track can, however, be assisted by augmenting the raw radar data with visual tracking of the ball.

[0115] In a preferred embodiment, one or more tracking units may include a tracking camera capable of image plane tracking or three-dimensional position tracking in addition to the tracking capabilities of the radar itself. The radar may often also be used to point and focus the camera to provide a region of interest for the camera, reducing the calculations required to find the exact pixel location of the ball and further improving the quality of the image-based tracking data. The data from the two sensors is then combined to extract accurate estimates of range and range rate from the radar, with accurate measurements of horizontal and vertical angles from the camera. This fusion of the two technologies provides the best of both worlds; the speed, robustness, weather resistance, and range measurement capabilities of the radar tracking combined with the enhanced angular accuracy of the camera-based tracking.

[0116] The tracking system is inherently scalable, allowing for the addition of any number of tracking units for collaborative tracking and sensor fusion. This may be determined on a case-by-case basis and may be applied where accuracy and / or reliability are paramount. Apart from the obvious benefit of increased accuracy, the additional sensors may equally address many of the issues mentioned above that prevent radar tracking from ensuring that every shot is tracked.

[0117] calibration For the tracking units to output meaningful data, it is paramount that each tracking unit 305 is able to map its measurements into a world coordinate system different from its own local coordinate system. This allows the trajectory to be superimposed, among other things, onto an image or video of the course, and also allows graphics such as tracers to be inserted into a broadcast video feed. The accuracy of a particular data point depends on both the accuracy of the tracking unit's 305 measurements, as well as the accuracy of the calibration of those measurements into world coordinates. The types of measurements so affected include: ball position at rest (before impact); horizontal launch angle; peak reach lateral; and carry lateral (flat).

[0118] The method of calibrating the tracking units to the world coordinate system may be the same across tracking units 305, regardless of whether the individual tracking units 305 are placed on a tee looking down the fairway or on a green looking back up the fairway. Calibration can be done in many different ways. In a preferred embodiment, the GPS location of key reference points on each hole is determined. In addition, the GPS location of each of the tracking units 305 is determined. For each tracking unit 305 that includes a camera system, one or more reference locations are determined in the camera images, allowing the orientation of the tracking unit 305 to be determined in world coordinates. If the tracking unit 305 is moved after the calibration is performed (e.g., because the tracking unit 305 is hit by someone on the course or the structure to which it is attached moves it), the measurements described above may be affected. Changes in the orientation of the tracking unit 305 may be detected by feature matching in images from the built-in camera. Alternatively, the changes in orientation may be determined by motion sensors built into one or more tracking units 305, or in other ways.

[0119] The fairway tracking unit 305 may be calibrated to a world coordinate system in a similar manner, although for portable tracking units it may be preferable to omit calibration to a world coordinate system due to speed. The fairway tracking unit 305 may be calibrated to a broadcast camera, for example, to provide tracer and / or other information to a common coordinate system with the camera, described in more detail below.

[0120] Add another camera for broadcast / filming As mentioned above, some or all of the tracking units 305 may include a camera system. However, one or more additional cameras may be placed on each or some of the holes for the purpose of broadcasting / filming the golf shots. In the following, a camera dedicated for filming will be referred to as a broadcast camera, regardless of whether the camera is actually used for broadcasting or not, as long as the camera's primary purpose is to provide a good viewing experience, either in real time, near real time, or after the fact, and not primarily to provide data for tracking.

[0121] The position and orientation of the broadcast camera can be determined in a manner similar to that used for tracking unit 305. Thus, any ball, in motion or stationary, can be at least approximately located in an image produced by the broadcast camera. In this manner, ball tracking information, such as a tracer or other image illustration, can be inserted into the image from the broadcast camera at a location in the image that corresponds to a location determined based on tracking information from the tracking system.

[0122] In some cases, the camera system present in the tracking unit 305 may be used as a broadcast camera, in which case the same images used for tracking may be used for broadcasting.

[0123] Adding more cameras to complement the tracking unit 305 is a straightforward process. Calibration schemes facilitate the inclusion of other cameras in the tracking and imaging tasks, including time synchronization between the tracking units 305. In one example, a pan, tilt, zoom (PTZ) camera, such as a robotic camera, may be added. With a PTZ camera, the pan, tilt, and zoom can be directly controlled based on the real-time ball trajectory captured by the tracking system, as described in more detail below.

[0124] Player Identification As mentioned above, a unique player ID may be associated with each player on a golf course and can be used to accurately associate each golf shot tracked by the tracking system with the player who took the shot. This player identification function can be implemented in many different ways, either alone or in combination with alternative options, as described below.

[0125] In a first option, the system analyzes the swing characteristics of each golfer in the tournament and generates a unique identifier for each golfer based on unique aspects of the golfer's swing characteristics. For example, the unique swing characteristics may include a combination of biometric data points (e.g., ratios of forearm to upper arm length, height, etc.) and data regarding the biomechanics of each golfer's swing (positioning of forearms and upper arms relative to the club shaft at different points during the swing, degree of twisting of the body relative to the legs, etc.).

[0126] In a second option, the system recognizes the golfer's clothing and / or face. For example, each golfer may be manually identified to the system at the start of each round, and the characteristics of the clothing worn by the golfer may be matched to the golfer in the system for recognition throughout the round. This information may be used by the system on its own or in combination with any other biometric and / or biomechanical characteristics to recognize each golfer for each shot. This option may be used alone or, for example, in combination with any or all of the other options. In a third option, each golfer carries an electronic identifier that allows them to be located by the system. For example, this may be as simple as a smartphone app that transmits a unique identifier as well as GPS coordinates that are recognized by the system, which can then be mapped to world coordinates. This option may be used alone or in combination with, for example, the first or second option.

[0127] In a fourth option, the player identification function may additionally employ theory regarding how players typically play a round of golf. For example, players typically play in groups of 2-4 (e.g., typically 3 golfers on a tour), and players in such groups play each hole together during a round, taking turns with the member of the group furthest from the hole taking the next shot. In another example, after a player takes a shot, that player's next shot occurs from a position close to where the tee shot landed. In yet another example, the system may employ knowledge of the layout of the course. For example, after a group finishes a hole (e.g., hole 1), the system predicts that the group will move to the tee box of the next hole (in this case, hole 2). These game theory methods according to the fourth option may be used in combination with options 1-3 described above to improve player identification and / or reduce the processing load required for player identification. Using this type of competition theory, and by tracking the progress of each group as they progress through the course, player identity tracking may generally only need to distinguish between members of a given group (e.g., the system need only determine which of the three players in a group known to be playing at a particular point on the course is about to hit a particular shot by tracking each group and then knowing where the group is on the course). This additional assistance should be available for all shots during a round, except for the first shot of a group that starts a round (e.g., the group's tee shot on the first hole). Even for these first tee shots, information regarding the identity of the golfer in the group currently on the first tee may be available to the system. This option may be used in combination with any of the three options previously discussed, for example, to narrow the list of options considered by the player identification system when determining the identity of a player detected as preparing to hit.

[0128] With regard to the image-based options discussed above (first and second options), a visual profile can be generated when a player is detected for the first time and associated with a unique player ID. When a player is detected in the video stream, an algorithm can match the detected visual profile with previous visual profiles detected previously (e.g., during the same round or, for non-clothing-based characteristics, in a previous event). For example, a player will rarely change shoes, pants, or hats during a round, and will rarely change the appearance of their upper body clothing during a round. If it starts to rain, the player may put on a rain jacket, or take off a sweater when he gets hot. Using these characteristics, the algorithm can re-identify the player in any camera (given sufficient image quality). This function can be implemented, for example, by a Convolutional Neural Network (CNN). Feature extraction and classification of features associated with the unique player ID can be used to distinguish players. Feature extraction can be done by detecting the color of the clothing, the height, the limb length, the golfer's swing type, or by training an auto-encoder. In the latter case, the feature vectors are provided by the autoencoder but are not necessarily explicit features for human interpretation. When a golfer is first identified, a feature vector v_0 may be associated with the detection and subsequent detections of the player that yield feature vectors v_ni close to the original detection v_0 are interpreted as being the same golfer. The cluster of detections corresponding to a particular golfer can be updated with a new feature vector if the detection is made with high confidence.

[0129] Once the batting order is determined on the first tee, each player's name and / or player ID can be associated with a unique visual profile, and any subsequent matches to this visual profile throughout the round can be tagged with the player ID associated with that player. This identification function can be further improved by utilizing competition theory, such as knowledge regarding which players are playing in the same group during a round, knowledge that these players will play hole 2 next after playing hole 1. For example, for off-tee shots where the image quality may not be sufficient to identify the player, the player identification system can use knowledge from the tracking system, including ball flight tracking as well as the final resting position of the previous shot as determined by the tracking system. Using this data, the system can determine where a given player's next shot will be taken with a high degree of confidence.

[0130] A database of players can be created such that even on the first tee, the system can recognize a player found in the database and associate that player with his or her corresponding Player ID. For example, this type of database can be created for all players on the PGA Tour. For a player participating in a PGA Tour event for the first time, the system can create a new visual Player ID when the player hits his or her first shot on the first tee or prior to this first shot (e.g., during a practice round, warm-up activity, etc.). In essence, automatic Player ID tracking would be provided throughout the round. The system can operate based on logic similar to that employed by a trained human who knows all the visual appearances of the players.

[0131] 16c illustrates a method 1630 for identifying a player and associating shot data with a particular player profile associated with a unique player ID, according to various exemplary embodiments. A system for implementing the method 1630 includes at least a camera and one or more tracking units for capturing shot data of an identified player in an image from the camera.

[0132] At 1632, the player identification system receives the player information for creating a database of visual profiles that includes information for recognizing players associated with the visual profiles. For example, the database may include player visual information, such as swing characteristics (e.g., including a combination of biometric data points that include data regarding the biomechanics of each golfer's swing), clothing features, facial features, or a combination thereof. Each visual profile is further associated with a player ID that can be used by the player identification system to associate shot data with the visual profile. In some embodiments, the database can include PGA Tour players, for example, to track professional events, and in other embodiments, the database can include amateur players.

[0133] At 1634, the player identification system can receive / generate player information for a new player not yet identified in the database or associated with a unique player ID / visual profile. For example, the system can analyze the new player's swing, clothing, or facial features and associate the determined information with a new visual profile. Additionally, clothing features can be updated for any players already associated with a visual profile, such as at the start of a current round. In some embodiments, for example, during amateur player tracking, an application connected to the player identification system and / or tracking system can be installed, for example, on a smartphone or smartwatch. Each golfer can provide a positive indication within the application that can be used to associate a tracking system player ID with the golfer, for example, at the start of a round. Further considerations regarding amateur round tracking are described in more detail below. The database can be populated using either one of 1632 and 1634 or a combination thereof.

[0134] At optional 1636, the player identification system executes algorithms that utilize golf game theory to improve its player identification functions (e.g., all players playing together in a group may be associated with each other for a round of golf such that the system can assume any golfer playing with a recognized member of the group is another member of this group). This may include theories about how players typically play golf; the layout of the course; etc., so that the player identification system only needs to distinguish between members of a given group as the players / groups are tracked around the course in relation to shots hit from specific locations. In some embodiments, a trained Convolutional Neural Network (CNN) is used to execute the game theory and / or player detection / matching functions described below.

[0135] At 1638, a player is detected in a video stream from a first camera during play of a golf round, and a visual profile is detected, e.g., based on clothing, face, swing mechanics, etc. The detected visual profile is matched to a visual profile in a database. In some embodiments, the player's location is determined, e.g., in world coordinates. As described above, a player may be detected based solely on visual characteristics observed in the video stream. An electronic identifier (e.g., transmitted from the player's phone) may be used to improve the match (and in some embodiments may be used in place of image analysis). CNNs may also be used to improve the match between the detected visual profile and a visual profile in a database.

[0136] At optional 1640, any CNN of step 1636 may be updated based on player detections, such as at step 1638. In some embodiments, the CNN is updated after each player detection in the camera feed, while in other embodiments, the CNN is retrained less frequently (e.g., after a number of detections, or after a predetermined time since the previous detection).

[0137] At 1642, a golf shot is detected by a player detected in the video stream. In some embodiments, the first camera used to identify the player is also used to detect the shot, while in other embodiments, the shot is detected by other sensors in the tracking system. In some embodiments, the golf club or golf club type used by the player is also identified and associated with the shot, as described in more detail below.

[0138] In some embodiments, described in more detail below, particularly with respect to amateur round tracking, a player does not need to be detected in the video stream in order for the player to be matched with a shot. For example, a player may have a wearable device associated with the player that is operable to detect swing motion. The wearable device may detect a swing, associate the swing with a timestamp and / or location (e.g., GPS location), and transmit this information to a tracking server. When the tracking system next detects and tracks a shot, the tracking server, either in real time or at a later time (e.g., at the end of the round), may compare the swing detection (and associated timestamp and / or location) based on the timestamp and / or location associated with the swing and shot, to match the swing with the detected shot and associate the shot with the player whose swing information originated from that wearable device. In an alternative embodiment, instead of using a wearable device that detects swing motion, a device that provides time-stamped location information, such as a cell phone, may be used to match the shot tracking recorded in world coordinates with the corresponding location of the device at the time the shot was struck.

[0139] At 1644, the detected player and shot are matched and associated with the player's unique player ID. The match is based on the shot's firing location in world coordinates, as determined by the tracking system, and matches the location determined for the detected player in the video stream from the first camera.

[0140] As described in more detail below, the association of a shot with a unique player ID can be used for a variety of purposes, such as sending the shot's data to the player immediately after processing, accumulating a player's entire round of shot data, accumulating clips of each shot taken by a given player during a round, etc. In further embodiments described in more detail below, the method 1630 described above may be optimized for either professional event tracking or amateur round tracking.

[0141] At option 1646, shots and associated player data are output during play. In one example, for example, in professional round tracking, the player's shot information can be sent to a broadcast. In another example, for example, in amateur round tracking, the shot information can be sent to an application on the player's personal device. The ball tracking data can be transferred to the personal application immediately after it is recorded so that the player can review the data while playing a round of golf. One particularly interesting application for this functionality can be to help a golfer find their ball. For example, when a golfer hits a shot that is not easily found, the personal application can narrow down where the ball is likely to be found, for example, on a golf hole map provided in the application. In another example, if the ball goes into, for example, a water hazard or out of the playing area, the system can precisely determine the area in which it is acceptable to drop a new ball and indicate this area to the golfer via the application. Further examples of the application of the Player ID functionality to both professional and amateur round tracking are described in more detail below.

[0142] In 1648, data on all shots hit by a particular player during a round of golf is accumulated in a database, for example, for presentation to the player at the end of the round or for other uses. The data can be used for a variety of purposes, such as, for example, providing statistics to the player for analysis, generating graphics, etc.

[0143] Automatic club type tagging For a camera placed behind or to the side of the player (e.g., on a tee box overlooking the fairway), an algorithm can be trained to recognize the type of club each player is using for all or some of their shots. In this viewing geometry, the image from the camera would provide a view of the club head from behind. FIG. 7 shows an example image 700 of the club head from a position behind the player. The camera used for club data tracking on the tee shot can operate, for example, at 200 fps with a pixel resolution of 2.5 mm / pixel. These camera specifications may correspond to the minimum camera capabilities required for club trajectory tracking.

[0144] One option for determining club type based on camera data is to train a neural network to recognize different categories of clubs. For example, a neural network may be trained to distinguish between seven different categories of golf clubs, including drivers, woods, hybrids, long irons, short irons, wedges, and putters. More sophisticated training of the neural network allows for more detailed analysis and can determine more specific characteristics of the clubs, such as distinguishing characteristics between specific club types, different clubs within the same club type, etc.

[0145] In an alternative option, an electronic tag that emits a unique identifier when in use (e.g., during a swing motion) can be used to identify the club type. A receiver can receive the signal when the club is in use and match the club's identifier with a player profile associated with the identifier. A receiver could be located, for example, at each tee box. In this option, the club identification system could additionally use data from a tracking system, for example, to confirm that the club transmitting the signal was likely used for the shot.

[0146] On the PGA Tour, the club category for a particular player shot is currently entered manually, so an automated method of determining this would be highly desirable. The club information is associated with the shot and is further associated with the player who hit the shot.

[0147] Broadcast Camera Calibration In order to accurately render the tracer on the broadcast camera feed, the broadcast camera is calibrated to world coordinates, which can be accomplished in the same way as calibrating the tracking unit to world coordinates, as described above.

[0148] For portable units, e.g., fairway tracking units, there is no need to map the shot trajectory to a world coordinate system. In some scenarios, the broadcast camera may be mounted together with the fairway tracking unit. All that is required is a calibration of the broadcast camera to the tracking unit coordinates to depict the tracer in the feed from the broadcast camera mounted together with the fairway tracking unit on the tripod. This baseline calibration may be performed, for example, the morning before each round, establishing a relationship between the intrinsic parameters of the tracking unit's built-in camera and the broadcast camera at different zoom levels. For each stroke, the zoom level is determined using feature matching between images from the two cameras, including, for example, the position of the ball in images from the same time (e.g., the ball's resting position before the shot) and other information that may include tracking data such as the tee position and range of the ball at launch. This allows the broadcast camera to be zoomed independently for each stroke, allowing the tracker to accurately render the tracer (or place any other graphic) at the desired position relative to the ball's position in the image from the broadcast camera, even without the tracker reading the zoom level from the broadcast camera.

[0149] Automatic generation of broadcast feeds As mentioned above, the whole course tracking system allows for automatic tagging of shots, i.e., automatic identification of the golfer who hits each shot. A coordinated tracking and broadcasting system, including the ability to film, track, and insert into video feed graphics, such as tracers (visual representations of the shot's trajectory inserted along the path of the ball), of nearly every shot hit on a golf course, may also make available to each golfer the collection of footage and data of every shot the golfer hits. This allows for the assembly (automatic or semi-automatic) of broadcasts customized for individual consumers or different markets. For example, if a golfer is very popular in a particular region or country, the broadcast feed in that region or country may isolate all shots (or any desired subset) hit by this golfer and automatically insert them into a custom feed. The same may be done in a more personalized manner, where a consumer may indicate, for example, a list of preferred golfers and / or a desire for combinations of preferred golfers and tournament leaders, for inclusion in a custom feed, which may be viewed, for example, through internet-based distribution.

[0150] To make it more efficient to produce these individual feeds, the images and / or footage provided by the standard broadcast cameras may be supplemented with images and / or footage from cameras included in some or all of the tracking units installed around the course. In one embodiment, a first tracking unit installed adjacent to the tee box of each hole includes at least a first camera and potentially additional cameras, in addition to, for example, a radar tracking unit. For certain holes where the green is not clearly visible from the first camera (e.g., holes with a large elevation difference or holes where trees or other obstacles turn to block the view of the first camera before the ball reaches the green), a second tracking unit including a second camera may be positioned behind the green (i.e., on the side of the green opposite the direction of the approach from the fairway to the green). If necessary or desired, additional tracking units including additional cameras may be positioned along the length of the fairway between the tee box and the green to ensure complete coverage of all shots. For example, the initial portion of a tee shot may be shown by a standard broadcast camera (or by a first camera) located behind the tee box, and after the ball is fired, the system may automatically switch to another camera in any of a number of different ways.

[0151] First, based on a theory of the game, including, for example, an analysis of the geography of a given hole (e.g., knowing the length of the hole and the distance to the broadcast camera behind the tee box or any items that may block the view of the first camera), the system may be programmed to switch to a second camera (e.g., located behind the green) when the ball passes a certain distance from the tee and / or moves laterally beyond a predetermined distance from the reference line. Alternatively, the system may match the trajectory of the shot as measured by the first tracking unit with that as measured by the second tracking unit (i.e., the system can determine that the trajectories being tracked simultaneously by the first and second tracking units correspond to the same ball in flight), and when the predetermined criteria are met (e.g., the ball is closer to the second tracking unit than the first tracking unit), the system can automatically switch the broadcast feed to the second camera. This may be useful when one of the tracking units is a moving unit, since the use of a predetermined cutoff point is impractical when the positions of the first and second tracking units change relative to each other from shot to shot.

[0152] Additionally, the system can manage the personalized broadcasting such that, in a given feed, if two desired golfers take shots at or near the same time, the shot from one of the golfers can be identified and buffered for broadcast after the feed of the shot by the other desired golfer is completed. This parameter may be set in the system (e.g., delay the feed of the second shot until a certain amount of time has elapsed after the previous shot has stopped). This automation allows the production of a large number of different high quality personalized feeds without increasing the need for a human operator to manage a significantly increased number of camera angle switches required, etc. The same approach described above can also be used to automatically generate highlights for broadcast / webcast or on-demand services (e.g., generating highlight clips for a particular golfer or group of golfers).

[0153] Additionally, to ensure smooth transitions of announcements / commentary between feeds, a computer-generated commentator may be used to automatically provide relevant information to the viewer, similar to that provided by a human commentator, based on information known to the tracking system. For example, historical data may be used to provide the status of the upcoming shot. Historical data may include, for example, commonly used phrases in each of multiple repeat scenarios, shots performed by the same player in similar shot scenarios, shots performed by other players in similar shot scenarios (from the same day of the same event, from the day before the same event, from previous tournaments, etc.), or general performance metrics accumulated for the player. In another example, current data may be used to provide the status of the upcoming shot, for example, the current hole number, the current shot number, distance to the cup, information about the lie, etc. The automated commentary may be provided in text form (e.g., as subtitles), or may be audible (e.g., computer-generated speech), or may combine voice and text. If the broadcast feed does not include an audio feed, or if it is desirable to augment the audio feed, artificial sounds may be inserted into the broadcast feed, such as sounds of shots (inserted into the feed when a shot is fired or with a delay, natural sounds are attenuated, or are unavailable for some reason), background sounds, etc. Automated commentary may be provided substantially in real time (during a live broadcast) or may be provided in highlight clips generated after the live event, etc.

[0154] It should be understood that the principles described herein for tracking and imaging golf balls and automatic generation of broadcasts can also be applied in other settings, for example, for other sports, especially in scenarios where coverage of multiple different events occurring simultaneously is desired. In one example, these principles can be applied to live broadcasts of tennis tournaments. In another example, these principles can be applied to live broadcasts of American football, where many different American football matches are taking place within the same time period. For video feeds (e.g., highlight reels) generated after the event, these principles can be applied to any sport, such as, for example, baseball, football (soccer), etc.

[0155] 16d illustrates a method 1650 for automatic broadcast feed switching, according to various exemplary embodiments. A system for implementing the method 1650 may include a tracking system including multiple cameras providing broadcast video feeds to a broadcast entity, a tracking unit for 3D position tracking of a fired ball, and a processing unit for detecting events in the tracking data that may trigger a switch from a current broadcast video feed to a new broadcast video feed.

[0156] At 1652, a broadcast video feed is generated based on the video feeds of one or more broadcast cameras according to predefined rules. For example, a third party broadcast may be used, i.e., a video feed controlled by a different entity (the broadcast entity) or selected based on criteria outside the scope of the present embodiment. For present purposes, the present broadcast feed may be assumed to be a broadcast feed of the tee box showing the player moments before hitting his tee shot.

[0157] At 1654, the tracking system detects a launched ball event based on the ball trajectory data. In one embodiment, the 3D position of the ball is used as a basis for determining whether an event has been detected with respect to the current broadcast camera compared to other potential broadcast cameras. For example, an event may be detected if the 3D position of the ball meets one or more position criteria. As discussed above, these events may include: the ball passing a certain distance from the tee; the ball moving more than a predetermined distance from the reference line; and / or other events that indicate that the ball has entered or will enter the field of view of a second camera. In another example, an event may include the launch of a shot.

[0158] At 1656, the broadcast feed switches to a video feed of a second camera based on the detected event such that the second camera captures a second portion of the ball's trajectory. A delay may be implemented after the event detection and before the feed switch. For example, a shot may be detected from a tee box unit and a predetermined delay (e.g., 3 seconds) may be applied before switching to a fairway unit.

[0159] If the second camera is a robotic camera, the camera parameters can be controlled before switching the feed. For example, the optimal crop, orientation, and zoom for the robotic camera can be adjusted to provide a visually pleasing video feed to the broadcast viewers.

[0160] 16e illustrates a method 1660 for automatic generation of a custom (e.g., individualized) broadcast feed, according to various exemplary embodiments. A system for implementing the method 1660 may be similar to the system for implementing the method 1650 described above. Additionally, the system may include a camera (which may be the same as a broadcast camera, for example) that detects a player in the image data, identifies the player and / or its characteristics, and triggers a switch from a current broadcast video feed to a new broadcast video feed.

[0161] At 1662, criteria for a custom broadcast feed are received. The criteria can specify, for example, a priority of players to be displayed in the broadcast feed. For example, a custom feed can be generated based on criteria including: all shots for one or more particular golfers are displayed in the custom feed; all shots for all golfers in a particular country / region are displayed in the custom feed; identifying a hierarchy of particular players and prioritizing the broadcast accordingly; prioritizing particular holes; prioritizing particular types of shots (e.g., very difficult shots), etc. The criteria can further identify features to be overlaid on the broadcast, e.g., particular data points and / or graphics to be displayed in the broadcast feed. The criteria can encompass any manner of prioritization of players, shots, holes, or scenarios, including, for example, criteria based on metrics from the broadcast or social media that indicate that a particular player, combination of players, or shot or hole is the subject of a trend in social media interest or increased betting activity. Highlights can be generated for particularly important or well-executed shots. Shots can be identified as such based on a measure of how good the shot is, e.g., the well-known "strokes" metric. Thus, shots with particularly high or low stroke numbers may be included in the highlight reel. Additionally, all shots from the current leaders (e.g., the top 5 or 10 players) may be included.

[0162] At 1664, the custom broadcast feed is generated according to predefined rules, including custom feed criteria. For example, a third party broadcast may be used in a custom feed broadcast when the custom feed criteria permit it (e.g., when a player of interest is not currently taking a shot). In some embodiments, multiple players of interest are identified in the custom feed criteria, and the video feed is switched between the players, for example, until an event is detected that triggers a switch of the broadcast to a particular camera.

[0163] At 1666, an event is detected that triggers a switch to another camera in the broadcast feed. Events include: a player being detected and identified by a particular camera unit; an identified player located near the launch area (e.g., tee box); an identified player in a hitting stance; etc.; these types of events related to a pending shot to be hit. In some embodiments, a neural network can be used to detect these events related to an upcoming shot, as described in more detail below with respect to FIG. 9. Other types of events can include a determination of the current 3D position of the ball in flight (after launch), which is used to determine whether to switch to another camera, such as a camera with an FOV covering the fairway and / or green that may provide a better field of view than the current broadcast camera, as described above with respect to method 1650 of FIG. 16d.

[0164] At 1668, the broadcast feed is switched from a current camera to another camera based on the event detection. The camera and / or radar unit that detects the event may be different or located separately from the camera being switched to.

[0165] Automatic tracer for every shot FIG. 8a shows a first image 800 from a video feed depicting a shot including a shot trajectory superimposition, i.e., a first tracer 805 of the shot, and some associated trajectory data. In this example, the first image 800 is captured by a tee box tracking unit including a radar and at least one camera. The first tracer 805 can be automatically generated in the image provided by the camera in the tracking unit because, as described above, all pixel orientations in the image are known relative to the radar's coordinate system. A superimposition such as that shown in FIG. 8a can be generated even if the tracking unit is not calibrated to the world coordinate system. FIG. 8b shows a second image 850 from a video feed depicting a shot including a shot trajectory superimposition, i.e., a second tracer 855 of the shot, and some associated trajectory data. In this example, the second image 850 is captured by a green side tracking unit. For example, a similar graphic superimposition including the first tracer 805 or second tracer 855 described above can be inserted into an image from any camera, as long as the camera is calibrated to the radar tracking the shot. The tracer can be inserted within any image, such as a video feed of an external broadcast, even if the radar unit tracking the ball is not located in the same tracking unit as the camera used for the external broadcast, as long as the radar coordinate system of the radar tracking the ball trajectory is calibrated relative to the camera. Additional information such as ball speed, carry and curve can be further tracked and displayed as numerical values ​​such as those shown in Figures 8a-8b.

[0166] Ball position determined based solely on radar tracking may contain some uncertainty until more sophisticated aerodynamic models are fitted to the data. In order to provide a visually pleasing tracer, it is important that the tracer begins in the image at a location that correlates accurately with the location where the ball will be launched. It is therefore preferable to detect the ball's position as accurately as possible in one or more images prior to the launch of the ball.

[0167] A ball identification algorithm can be used to identify the location of the ball. Once a player is identified on the camera and determined to be ready to take a shot (e.g., based on an identified posture), the ball identification algorithm searches for the exact location of the ball in the image. This location can then be used to determine the starting location of the tracer. If this data is not available (e.g., if the ball is obscured by deep rough), the system may need to extrapolate from the initially detected location of the ball in flight to a launch location that can still be corrected using the player's stance as a general marker of the original ball location. Thus, once the tracking system detects a shot and outputs trajectory data, these data can be automatically displayed in the broadcast as a tracer with little or no human input.

[0168] Locating the ball may also use analysis of player motion to detect a player taking a stance over the ball (as is always done before hitting a shot) before the start of a swing, with the ball located near the club head, and a neural network may be trained to search for and recognize the ball by locating the vicinity of the club head when the player is taking their stance before hitting the ball.

[0169] 16f illustrates a method 1670 for automatic insertion of a tracer into a broadcast feed, according to various exemplary embodiments. A system for implementing the method of 1670 includes a radar tracking unit providing a 3D position track of the launched ball, and a broadcast camera providing a video feed of the ball launch, the broadcast camera (or a broadcast entity upstream from the actual camera) configured to insert 3D graphics into the broadcast feed, the 3D graphics including at least the tracer. However, in other embodiments, a tracking camera (separate or the same as the broadcast camera) can be used to provide the 3D position track.

[0170] At 1672, the radar coordinate system of the radar tracking the ball trajectory is calibrated to the broadcast camera. The broadcast camera may be calibrated to the radar in any manner previously discussed. For example, calibration of the broadcast camera to the tracking unit coordinates is only required for a mobile and / or robotic tracking unit (including one or both of a tracking unit and a broadcast camera (co-mounted)) to pull a tracer from the broadcast camera into the feed. For a camera tracking unit, a relationship between the tracking unit's camera intrinsic parameters and the broadcast camera may be established for different zoom levels, as described above.

[0171] In optional 1674, the location of the ball is precisely detected in one or more broadcast images prior to the launch of the ball. This optional step may improve the rendering of the tracer starting point in the image by determining a location in the image that precisely correlates with the location where the ball will be launched. A ball identification algorithm may be used to locate the exact location of the ball in the image and then determine the location of the tracer starting point. In another embodiment, the player's stance may be used to identify the location of the ball, and / or a neural network may be trained to locate and recognize the ball by searching for a location in the vicinity of the club head when the player is taking their stance prior to hitting the ball.

[0172] At 1676, a tracer is generated for the broadcast video feed based on the trajectory data captured by the tracking unit. If information about the exact hole location before launch is not available (e.g., the ball is obscured by deep rough), the system may need to extrapolate from the first detected position of the ball in flight to a launch position that can still be corrected using the player's stance as a general marker of the original ball location. The broadcast camera can zoom to any zoom level independently for each stroke without requiring the tracking unit to read the zoom level from the broadcast camera (in a scenario where a baseline calibration has been performed and intrinsic parameters have been mapped to different zoom levels, the current parameters can be determined from similar detections of the tracking and broadcast cameras). In some embodiments, a first tracer can be generated for a first video feed from a first broadcast camera (e.g., located on a tee box) to follow the ball during a first portion of its trajectory (e.g., its initial trajectory), while a second tracer can be generated for a second video feed from a second broadcast camera (e.g., located on a fairway or green) to follow the ball during a second portion of its trajectory (e.g., its descent).

[0173] The live video can be automatically augmented with graphics such as, for example, tracers and lower third graphics (e.g., ball speed, peak point, carry, curve, spin rate, etc.) Additionally, other graphics can be automatically added to display, for example, the player name on this hole, shot number, hole number, current score, and other lower third graphics seen in live sports broadcasts.

[0174] In this way, a fully automated broadcast can be generated. Video of each shot can be clipped and trimmed to accurately include the relevant period based on ball tracking. Additionally, each clip can be tagged with the player name (or player ID), the shot number on this hole, the hole number, etc. Clips can be saved with and / or without tracer information and / or other graphics. This database of clips essentially contains every shot of every player in any tournament.

[0175] Automatic event detection A live video stream from a tracking unit's camera can be used to train a neural network for the specific application of a particular golf scenario (e.g., a tournament scenario). The network can be applied to any camera stream, and is particularly useful for a tracking unit's camera located behind a tee box. In this case, the network outputs detections of events such as, for example, when a player and / or other person steps up to the tee, when a player is preparing to hit the ball, and when a shot has been hit. These detections can be used to automate event generation that is useful for on-site staff, broadcasters, and betting applications. Detected events can include: player on tee; player preparing to hit a shot (addressing the ball); player taking a swing; shot hit; tee box cleared.

[0176] FIG. 9 shows example images 900-915 that can be analyzed by a neural network to detect events related to a player's swing. At 900, a player is detected in the image prior to hitting a shot. The neural network has not yet detected a shot-related event. At 905, the player is detected approaching the ball. At 910, the player is detected taking a swing. At 915, the player is detected hitting the ball.

[0177] When developing an algorithm or network to recognize a golfer, the neural network can consider various factors in detecting an event. One factor to consider is that generally, only a golfer holds a golf club and at the same time places himself in the address position to hit a golf shot. The specific angle of the golf shaft at address is also unique and can be used. Furthermore, the relative position of the ball and the club head is a good indicator that can be used to identify the player who is addressing the ball and about to start the swing. Using techniques similar to those used to identify individual golfers, people traveling with the golfer may be excluded from such identification. For example, caddies and their clothing may be identified and stored in the system such that such individuals are not identified as golfers even when holding a club and / or standing on the ball.

[0178] Detection of events can also be used to more efficiently manage power usage for different sensors and subsystems of the tracking system. When an event requiring a particular system capability is not currently detected in the tracking unit's field of view or in data from other system components, the power usage of that item can be reduced, for example, by placing various power-drawing components of the item in a sleep or low power mode until the system detects an event requiring full activation of the item or an immediate pending event. This can be of great practical importance since golf course systems are often either battery-powered or generator-powered.

[0179] In a further aspect, detection of events for an upcoming shot, such as a player's approach to the ball or the formation of a posture on the ball, can be used to predict when the upcoming shot will occur. This can have great practical importance for live betting applications, such as betting on the distance of an upcoming driver shot or the distance from the pin. For example, if the system estimates that an upcoming shot will be executed within, for example, 5 seconds, the live betting window for that upcoming shot can be closed depending on the estimated shot time. Existing methods typically rely on a manual trigger to close the betting window.

[0180] For live betting applications, the tracking system may be designed to be tamper-proof or to detect tampering, e.g., so that a manual operator cannot tamper with the sensors to gain a betting advantage, e.g., by delaying the live feed of the shots so that bets are placed after the actual shots are taken.

[0181] Controlling a robot camera from real-time 3D tracking data The tracking system tracks the ball in motion with very low latency and constantly updates the ball's predicted future trajectory, allowing the system to control other cameras in real time. Using real-time position data for the ball, the tracking system can control a robotic camera to point the camera at the current ball position (tracking the ball through its flight) and additionally control zoom and focus to ensure a clear, viewer-friendly image that viewers are accustomed to in the same manner as achieved by professional camera operators during golf tournaments. The robotic camera described herein may correspond, for example, to the broadcast camera 315 described in FIG. 3. However, the robotic camera may also be used primarily for tracking purposes rather than as a broadcast camera.

[0182] The 3D position of the ball relative to the robot camera is always known by the tracking system based on real-time data acquired by other sensors of the tracking system. Thus, the optimal crop, orientation, and zoom level for the robot camera can be automatically controlled by the system based on the real-time data. Optical tracking of the ball in flight may also be performed using analysis of data from the robot camera (or any other camera) and used to control the robot camera to achieve the desired image using any known method. An advantage of the proposed system is that the zoom and focus of the robot camera can be continuously updated based on in-flight tracking of the ball by other sensors or other tracking units, even if the ball is not visible or easily found in the image from a particular camera (e.g., when a white ball disappears against the background of white clouds). That is, controlling the aim and focus of the robot camera based on the three-dimensional tracking data may be used to enhance the optical tracking using the image from the robot camera, which ensures that the sharpest possible image of the ball is captured, increasing the likelihood of detection of the ball in the image. Without knowledge of the ball's 3D position relative to the robotic camera, this detection is often not possible or is delayed as the optical tracking system attempts to reacquire the unlocated ball (e.g., when the camera is focused at a distance that does not correspond to the ball's current position, the ball is lost against a similarly colored background, or the line of sight is occluded).

[0183] In some scenarios, the robotic camera may be located in a tracking unit that includes additional sensors, e.g., radar or lidar, and the entire tracking unit changes orientation based on the tracking capabilities of the camera. This change in orientation can improve the tracking of the additional sensors in addition to tracking the robotic camera. For example, some shots may have unexpected trajectories (e.g., poor shots) that are not fully covered by the initial FOV of the sensor. When these shots are detected by the tracking camera, the orientation of the entire tracking unit including the additional sensors can be changed to provide, for example, improved sensitivity to radar.

[0184] Robotic cameras may be used to replace manual camera operators, making video footage much more appealing to viewers, and also allowing for the creation of close-up footage to depict, for example, the lie of the ball. To ensure a stable and viewer-friendly movement of the robotic camera, a specially filtered version of the real-time 3D data from the tracking system can be generated and used to control the robotic camera. Furthermore, knowledge of the characteristics of the robotic camera control, such as delay, maximum angular acceleration, maximum angular velocity, focus delay, etc., can be taken into account and compensated for when controlling the camera. The robotic camera can also be used as a tracking sensor for the positioning of the club and ball in world coordinates. However, this requires a calibration of the robotic camera, since the field of view is not stationary.

[0185] Robot camera calibration In order to use a robotic camera as a tracking sensor, the robotic camera needs to be calibrated. Calibration includes two parts: extrinsic calibration and intrinsic calibration. As mentioned above, the extrinsic parameters essentially include two parts: the determination of the camera's position in world coordinates and its orientation in world coordinates. The determination of the robotic camera's position is done in a manner similar to that employed for the tracking unit and / or broadcast camera described above. What remains is the determination of the orientation in world coordinates and the intrinsic parameters.

[0186] 10a shows an example image 1000 captured by a robotic camera 1005 for determining the orientation of the robotic camera 1005 in world coordinates 1020. The image 1000 is shown in a two-dimensional (u,v) coordinate system local to the camera 1005. A ball 1010 is located at pixel coordinates (u,v) of the image. b 1015 is located in image 1000. Pixel (u,v) in world coordinates 1020 b There are various ways to determine the position of 1015.

[0187] According to the first option, pixel (u,v) bThe position of 1015 can be determined in world coordinates 1020 by pre-calibrating the intrinsic parameters (e.g., focal length, principal point, lens distortion, etc.) of each different zoom level used by the robotic camera 1005. The orientation of the robotic camera 1005 in world coordinates 1020 is determined by using pan and tilt offsets of the orientation of the camera 1005 when looking at the ball 1010, for example, for a given orientation of the robotic camera 1005 calibrated to world coordinates in a manner similar to that described above for the tracked unit cameras and / or broadcast cameras, determined from the camera's pan / tilt sensor or from a tracking-based control signal applied to the camera orientation. That is, the system determines based on the pan / tilt information. A three-dimensional line described by a vector R and a point p in world coordinates, where p is the position of the camera. This vector R extends in a direction from the camera (e.g., from the camera's focal point, point p) towards the ball (or any other object whose position is to be determined). Knowing this line from the camera in three dimensions, the three-dimensional position of the ball (or other object) on the line described by vector R and point p can be obtained by combining this line data with the ball's distance information (radar or other information in world coordinates or any other coordinates that can be translated into a coordinate system relative to the camera) or by determining where the line described by vector R and point p intersects with the golf course surface as displayed in a three-dimensional model of the golf course. This allows the three-dimensional position of the ball along the line described by vector R and point p to be determined in three dimensions in world coordinates. Those skilled in the art will appreciate that by identifying the intersection point of the line described by vector R and point p and the three-dimensional model of the golf course surface, it is possible to determine the position of only balls at rest on the golf course surface. Further options described below explain different ways of identifying this vector R such that this information can be used together with distance information or three-dimensional surface model information to allow for the location of the ball in three dimensions in world coordinates based on the pixel location of the image from the camera.

[0188] According to the second option, the pixel position (u,v) b 1015 first calculates the position (u,v) of a pixel in the image 1000 in the local (u,v) coordinates in world coordinates 1020. b By detecting 1015, it can then be determined, without changing the zoom level, by changing the orientation of the robot camera 1005 so that it faces a predefined reference position 1025, which corresponds to the same image (u,v) where the ball 1010 is located in the image 1000. b Ensure that the sensor is located within the range of

[0189] Figure 10b shows a process for determining the orientation of a robot camera 1005 based on the first image 1000 of Figure 10a and a second image 1030 captured by the camera 1005 using a different orientation. Similar to Figure 10a, the ball 1010 is located at pixel coordinates (u,v). b The camera 1005 is positioned in the first image 1000 in 1015. The camera 1005 is positioned such that a reference position 1025 (whose position is known in world coordinates) corresponds in the second image 1030 to the coordinates (u,v) in the first image 1000. b Same coordinates (u,v) as 1015 b 1015. This enables the system to identify a line described by the three-dimensional vector R from the camera to the ball in the first image 1000.

[0190] The pan and tilt offsets between the two scenarios are then used to calculate vector R, the 3D orientation of the camera in world coordinates 1020 when looking at the ball 1010 compared to a reference position 1025. The obvious advantage of this method is that the intrinsic parameters of the camera 1005 do not affect the accuracy of the determination.

[0191] According to the third option, pixel (u,v) b The location of 1015 is initially the pixel (u,v) of the first image 1000 in local (u,v) coordinates. bThe position of the robot camera 1005 and the tracking unit camera 1035 can be determined in world coordinates 1020 by detecting the position of the robot camera 1015 and then detecting the same position in a second image 1040 from the tracking unit camera 1035. In a preferred embodiment, the robot camera 1005 and the tracking unit camera 1035 are co-located, although this is not required.

[0192] FIG. 10c shows a process for determining the orientation of the robotic camera 1005 (three-dimensional vector R) based on the first image 1000 of FIG. 10a and a second image 1040 captured by a second camera 1035. The second camera 1035 may be a tracking unit camera calibrated to the world coordinate system 1020. The first image 1000 from the robotic camera 1005 may be located within the second image 1040 from the tracking unit camera 1035. One option for locating the first image 1000 within the second image 1040 is to perform feature matching between the two images 1000, 1040 using identified features 1045 whose positions are known in world coordinates (e.g., trees, bunkers, ponds, golf flags, green contours, and other unique features that may be identified in the images). Since the position of the second image 1040 is known in world coordinates 1020, it can be transferred to the first image 1000 so that the combination of the vector R, the position p of the camera in world coordinates and the distance information allows the determination of the world coordinate position of the ball 1010 in the robot image 1000 using distance and / or a three-dimensional model of the golf course, as described above.

[0193] According to the fourth option, pixel (u,v) b The location of 1015 is initially determined by pixel (u,v) in the first image 1000 in local (u,v) coordinates. b By detecting the position of 1015 and then zooming out the first image 1000 (e.g., generating a second image 1050 that includes a zoomed-out version of the first image 1000), at least two reference points 1060, e.g., 1060a and 1060b, can be determined in world coordinates 1020 so as to be visible in the image 1050.

[0194] FIG 10d illustrates a process for determining an orientation of a robotic camera 1005 based on the first image 1000 of FIG 10a and a second image 1030 that includes a zoomed-out version of the first image 1000. As in FIG 10a, the ball 1010 is aligned with pixel coordinates (u,v). b 1015 is located in the first image 1000. The camera 1005 zooms out while the coordinates (u,v) of the second image 1030 are changed so that the reference point 1060 is visible in the second image 1030. b The pixel location in 1015 is the coordinate (u,v) of the first image 1000. b The reference point 1060 is the same as 1015. The reference point 1060 is detected at several pixel locations 1055, e.g. 1055a and 1055b, in the second image 1050 while tracking the position 1015 of the ball position in the image 1000. Assuming slight lens distortion, it is only necessary to determine the focal length. The focal length is determined by correlating the reference point location 1055 with a predefined angle in world coordinates 1020 of the two corresponding reference locations 1060. Thereafter, the position of the ball 1015 in the zoomed out image 1050 can be determined in world coordinates 1020.

[0195] 16g illustrates a method 1680 for calibrating a robotic camera to a world coordinate system, according to various exemplary embodiments. A system for performing the method 1680 may include only the robotic camera and a processing unit for controlling its operation. In some embodiments, the system may include additional cameras.

[0196] At 1682, the robotic camera is provided with uncalibrated intrinsic parameters, for example, the robotic camera has either been recently powered on or has recently had its orientation changed relative to a previous orientation.

[0197] At 1684, a first image (e.g., a calibration image) is captured with the robot camera and the ball is detected in the image. The (u,v) coordinates of the ball are determined in the camera coordinate system.

[0198] At 1686, a second image is received from either the robotic camera (using a different orientation and / or a different zoom level) or a second camera that simultaneously captures a second image of the same ball.

[0199] At 1688, an orientation of the robotic camera is determined based on the first and second images.

[0200] Safety / warning systems for on-site spectators Real-time ball flight tracking, as described above, can predict the landing point while the ball is in the air, with real-time updates. This landing point prediction can be used, for example, in an automated "forehand," to warn spectators of an approaching errant shot. For example, spectators in the affected area can be given appropriate instructions, such as seeking cover or protecting their heads.

[0201] The alarm system may include, for example, wired or wireless georeferenced speakers that may be triggered by the tracking system. The golf course is preferably divided into different spectator zones. Foreground alerts may be triggered in one or more spectator zones whenever an errant shot approaches these zones with sufficient probability. The alarm system may be a personal application on a smartphone, tracking the location of each spectator (or any spectator who has opted in to the service). A special alert, such as a vibration / sound or similar, may be triggered by the smartphone application when an errant shot is likely to land in a zone where the spectator is located.

[0202] FIG. 16h illustrates a method 1690 for a warning system for venue spectators, according to various exemplary embodiments.

[0203] At 1692, trajectory data is captured using one or more first sensors.

[0204] At 1694, the landing point is predicted based on the trajectory data captured so far. The landing point prediction may be performed initially as early as possible in the ball's flight and then refined as more trajectory data is captured. The landing point prediction may include a specific point and the warning region may have a predetermined uncertainty radius around the point or may include a region based on the actual estimated uncertainty.

[0205] At 1696, an alert is triggered and delivered to any interested parties, for example, within an alert radius. In one example, a speaker located near the predicted landing location can be activated to provide an audible alert. In another example, the alert can be sent to a spectator's device located at the landing site.

[0206] Amateur Round Tracking All of the features of the system described above can be implemented on any golf course. All ball trajectory tracking is completely independent of shot quality and is applicable to amateurs as well as professionals. Furthermore, amateur rounds may be conducted in a manner similar to the customs and rules governing professional golf tournaments, and thus the competition theory previously discussed for professional events may be similar to amateur rounds. For example, a group of 1-4 players will typically play hole 1 first, then hole 2, etc. Thus, the detailed player ID tracking described above may be implemented in a substantially similar manner.

[0207] One aspect of professional event tracking that may not be available for amateur rounds is pre-stored information regarding the identities of players in a given group. In order to associate tracking of a golf round with a particular golfer, several additional features can be used.

[0208] In one embodiment, an application connected to the tracking system can be installed, for example, on a smartphone or smartwatch. Each golfer can provide a positive indication in the application that can be used to associate a tracking system player ID to the golfer, for example, at the start of a round. There can be a special area around the first tee, for example, a small, marked area on the ground with a single-person space where a player can sync their smartphone with the tracking system. For example, a user can press a "sync" button in the application while in the special area to associate the player's name and history with a unique player ID in the tracking system. In another embodiment, tracking and association of player IDs can also be done via some kind of transmitter, for example, a smartphone or smartwatch with location capabilities.

[0209] In another embodiment, a device worn by the player, e.g., a smart watch, can be used to detect swing motion by the player. The device may be associated with the player and may include an accelerometer (or other sensor capable of detecting such motion) and processing logic for detecting changes in acceleration corresponding to a golf swing. The device may have GPS capability (or be coupled to another nearby device having GPS capability, e.g., a smartphone) so that the approximate location of the player during the swing can also be detected. The device may also be operable to associate the swing motion with a timestamp. Thus, when the tracking system detects a shot (and the associated time of the shot), the shot can be matched with the detected swing that matches the timing and location of the shot. There is no requirement, particularly in amateur round tracking, that the match between the swing and the shot be performed in real time. Thus, the swing detected by the device (and associated timestamp) can be matched in time with a shot detected later by the tracking system, e.g., the end of the round. This is particularly useful in scenarios where, for example, the device worn by the player has limited capabilities in terms of wireless coverage, processing power, etc. The device can store the swing detection along with the associated timestamp until a later time when the swing data can be associated with the tracking data. In one embodiment, a player associated with the device can review shot information associated with detected swings at the end of a round and verify accuracy of the match. In an alternative embodiment, the player carries a positioning device, such as a mobile phone. The device records a path of the device's world coordinates with associated timestamps, which can then be matched to the determined launch location of the shot tracked by the tracking system. The positioning device does not have to be worn by the player, but can be in the player's bag, near where the player will be launching the ball at all times.

[0210] A player's image profile may be stored in the tracking system for future golf rounds or may be created before every golf round. Tracking systems for each of multiple courses could potentially share these image profiles in some circumstances. After each round is played, the captured data may be transferred to a personal application and each round can be reviewed with all tracking data for later analysis across the golf rounds.

[0211] In one embodiment, the ball tracking data may be transferred to a personal application immediately after it is recorded, so that the player can review the data while playing a round of golf. One particularly interesting application for this functionality may be to help a golfer find his or her ball. For example, when a golfer hits a shot that is not easily found, the personal application may narrow down where the ball is likely to be found, for example, on a golf hole map provided in the application or another map application (e.g., Apple Maps or Google Maps). This functionality may be particularly useful when trying to find a ball in deep rough, where narrowing the search area significantly increases the chances of finding the ball and also reduces the time required to find the ball.

[0212] When a golfer hits a shot that falls into a hazard, depending on the rules of golf for the particular hazard, the golfer may be given the option to drop a new ball according to where the previous shot crossed the hazard line. Because the system has track of the ball trajectory and therefore knows exactly where the ball crossed the hazard line, the system can determine exactly the area in which it is permissible to drop the new ball. This can resolve the sometimes contentious decision of where the ball may legally be dropped, saving the golfer time and restraint while achieving a fairer outcome. The identification of the permissible drop areas may be shown in a software application that shows the layout of holes with the permissible drop areas.

[0213] Creation of Non-Fungible Tokens (NFTs) The proposed system can automatically capture and track any special events that occur during a golf round, which may be a hole-in-one occurring or any other specific unique golf shot or event. The system can be configured to automatically detect such moments and create video clips that may include ball tracer, club and / or ball trajectory information. The creation of these clips can be created in such a way that only one digital copy can exist and thereby can be used as a non-fungible token (NFT).

[0214] Introduction to camera-less applications For example, for tracking shots hit from a tee box or onto a green, a tracking system may include radar, a combination of radar and cameras, or a combination of several additional sensors. These different sensors may track different and / or overlapping parts of the flight of the golf ball. Each sensor acquires object data consisting of measurements of the ball one or more times in a coordinate system specific to the device, and when calibrated to each other, the data may be projected into a highly consistent world coordinate system for further processing and data fusion. Doppler radar capable of angle and absolute range measurements can accurately track the ball in three dimensions; radar and camera equipped tracking units can use image tracking to enhance angle measurements for more accurate tracking; and systems including multiple cameras with overlapping fields of view can use stereoscopic vision to determine the ball position in three dimensions.

[0215] Sensor calibration is important for accurate measurements. Sensor calibration requires the determination of various parameters, both internal and external to the sensor. Intrinsic parameters may include focal length, lens distortion parameters, and principal point for cameras, and phase offsets for radars. Extrinsic parameters typically configure the position and orientation of the sensor.

[0216] Known systems based on Doppler radar tracking have drawbacks, for example, when the ball moves slowly or rolls along the ground; its radial velocity v relative to the radar r is low when the ball moves at a particular angle relative to the radar-to-ball direction; and when the ball is stationary and Doppler radar tracking fails completely. Known systems that rely on stereoscopic vision to track the ball can determine the three-dimensional trajectory of the ball even at low speeds, but without a detailed model of the golf course they cannot generally provide metrics relevant to understanding the short game of golf.

[0217] The internal or intrinsic parameters of a sensor may be generally defined as parameters related to the internal operation of the sensor. The intrinsic parameters of a camera are the parameters necessary to connect the pixel coordinates of an image point to the corresponding coordinates of the camera reference frame and include focal length, principal point, and lens distortion parameters. The intrinsic parameters of a camera may be measured in a production setup and stored in the system, and methods for determining the intrinsic parameters in a laboratory setting are believed to be known to those skilled in the art.

[0218] The external or extrinsic parameters of a sensor may be generally defined as parameters related to the positioning of the sensor (e.g., position and orientation) relative to world coordinates. Knowledge of the external parameters of a sensor allows for mapping of measured data from the sensor's coordinate system to a world coordinate system so that it can be compared to data from other sensors, or to the position of a three-dimensional model of a portion of a golf course.

[0219] The extrinsic parameters can be determined in several ways, as will be appreciated by those skilled in the art. A preferred method of calibration is to measure the GPS location of the camera and several reference points on the golf course within the camera's field of view. A calibration image is captured with the camera and the positions in the image plane of the measured reference points can be determined by human annotation or automatic detection means to calculate the extrinsic parameters. Easily recognizable features that can be used in the calibration include, for example, the location of the pin (hole), the edge of a bunker, or objects temporarily or permanently placed on the course used for calibration. The GPS locations of the reference points can be supplemented by measurements of the distance and / or height difference between the reference points and the camera, for example, using a laser range finder or similar equipment.

[0220] A tracking camera for tracking bounce and roll and / or rest position according to this embodiment is typically installed behind the green and mounted on a TV tower or similar structure, with the camera view from that position including the green and surrounding areas. Cameras may also be placed along the fairway on long par 5 holes to capture the ball's landing position on the fairway. The tracking function according to this embodiment may be implemented using a single camera, and does not require multiple cameras utilizing, for example, stereoscopic imaging technology.

[0221] In one embodiment, the lie of the ball is determined based on images from a single camera of the ball at rest and a 3D model of the terrain at the ball's rest position.

[0222] In some embodiments, a system and method is provided that tracks and provides bounce and chip shot tracking metrics using a single camera and an accurate 3D model of a portion of a golf course, where the single camera is calibrated to the portion of the golf course covered by the 3D model by capturing a series of images with the camera, detecting the ball in the images, unprojecting the pixel locations of the ball detection to generate a line in 3D space from the camera to the ball location, determining a start line and an end line corresponding to the start and end points of the trajectory, intersecting the 3D model with the start line and end line, applying the physical model and determining the ball's track.

[0223] In one embodiment, a determination is made whether a ball detected in a sequence of images is bouncing or rolling.

[0224] In one embodiment, 3D tracking of the ball's roll is provided by detecting the ball in a series of images and unprojecting the pixel locations of the ball detections, thereby generating a line in 3D space from the camera to the ball location, and intersecting the 3D model with the line to determine the ball's track.

[0225] In one embodiment, the stimp on the green is determined from the putt track and terrain slope in a 3D model.

[0226] In one embodiment, a putt breaking fan is determined for a given initial ball position, and the putt breaking fan is compared to the measured 3D putt trajectory to determine the sensitivity of the ball launch parameters to a successful putt. The putt breaking fan is described in more detail below.

[0227] In one embodiment, radar tracking of the golf ball is used to guide the ball detector in time to the appropriate image and the appropriate search area within the image.

[0228] In one embodiment, an image is captured from a camera and it is determined whether the image includes a person and / or if the person is a golfer in an address position, and based on this detection, the ball detector is directed in time to the appropriate image and the appropriate search area within the image.

[0229] In one embodiment, the extrinsic calibration parameters of the camera relative to the 3D model are updated based on the tracking of the jump.

[0230] 3D surface model of a golf course or part of a golf course In accordance with various exemplary embodiments described herein, a three-dimensional model of an entire golf course, or a particular portion of a golf course (e.g., one or more greens and the areas immediately surrounding those greens), can be used to improve and enhance tracking of shots within the areas covered by the 3D model.

[0231] A 3D model is a representation of a golf course (or a portion of a golf course) stored in computer memory and can be used to render 3D graphics of the golf course. The 3D model of a particular golf hole preferably covers a portion of the area within the field of view of a single camera used to track the golf ball according to the present embodiment. Thus, the 3D model covers at least the terrain around the green where a shot from the tee or fairway is likely to land. The model may also cover some or all of the fairway and surrounding rough and / or semi-rough, or may cover the entire hole. To cover an entire golf course where multiple cameras are used, multiple 3D models may be used, each covering a portion of the course, or one 3D model covering the entire course. Those skilled in the art will appreciate that any reference to world coordinates may apply to a coordinate system common to all holes on a golf course, or to the area of ​​a single hole on a golf course. The use of a world coordinate system allows measurements from different sensors to be fused and allows measurements of the ball position to be mapped to the 3D model. World coordinates do not need to be common to all sensors covering a round of golf. That is, as long as all sensors tracking the flight of any shot are able to transform their data into a common coordinate system, the desired results will be obtained. Thus, if a group of sensors covers hole number 1 and only hole number 2, it is not necessary for those sensors to be able to transform data from a sensor on hole number 2 into a common coordinate system. Thus, use of the term world coordinate system only suggests that this system is relative to the physical environment of the sensors, and does not imply that this coordinate system needs to be common across all sensors operating on a golf course.

[0232] The 3D model includes at least a surface model that represents the elevation of the terrain at a given location on the course. The surface may be represented as a mesh of triangles, as a spline surface, or another representation known to those skilled in the art. The surface may also be represented by a coarse model that provides a coarse representation of the elevation of the course in combination with one or more fine models that provide a more detailed surface map of some or all of the features represented by the 3D model, the fine models representing offsets to the coarse model. A coarse model may be provided for a given geographic area, while a fine model may be provided only for a portion of the area covered by the coarse model, for example, where play most likely occurs or where additional accuracy of the surface is desired. These areas may include the fairways and surrounding semi-roughs, bunkers, greens and areas surrounding the greens. This scheme of including a coarse model and one or more fine models may be advantageous by avoiding the creation and processing of a detailed model of the entire course.

[0233] In a preferred embodiment, the 3D representation of the golf course is provided in world coordinates so that GPS positions can be easily mapped onto the 3D representation. The 3D representation may also be provided in a local coordinate system and associated with a known mapping to world coordinates. The ability to map the GPS, and / or world coordinate system, to the 3D model allows the camera extrinsic parameters to be associated with both the golf course and the 3D model, thus allowing the position of the ball on the ground to be determined by capturing an image with the camera, detecting the pixel position of the ball in the image, applying the intrinsic and extrinsic camera parameters to unproject the detection to determine a camera-ball line in a coordinate system, such as the world coordinate system, and determining the intersection of the camera-ball line with the 3D model.

[0234] In one embodiment, the 3D model includes a representation of the types of terrain present on the portion of the golf course represented by the model; for example, each triangle in the mesh may have a terrain type associated with it, or the terrain types may be represented in a fine elevation model. The terrain types may include at least the following features: fairway, rough, semi-rough, bunker, green, and fringe, but many more features may be included for a more detailed representation of the various terrains found on a golf course. These features may be used to improve tracking of the trajectory of the ball as it impacts these terrains, as described below. The model may also include representations of non-terrain features, such as trees, bushes, and buildings, which may provide realism to any graphical rendering based on the 3D model, although these non-terrain features are generally not used for any tracking purposes.

[0235] The 3D model can be obtained, for example, by drone scanning using either lidar, optimometry, or a combination thereof, or by another method known to those skilled in the art. The scan can result in a point cloud from which a 3D surface model can be created using techniques known to those skilled in the art. The relationship between the local coordinate system of the course and world coordinates can be established by mapping a number of fixed points identifiable on the 3D model with precise GPS measurements. For example, the PGA Tour has detailed maps of the golf courses used for play on the tour.

[0236] Ball position determination system using a non-projector FIG. 11 illustrates an exemplary tracking system 1100 including at least one camera 1105 that captures a series of images of the ball, including the ball at rest, and a processing unit for tracking the ball and / or determining the position of the ball at rest, according to various exemplary embodiments. The processing unit includes a ball detector 1110 that determines the position of the ball in an image coordinate system, an unprojection module (unprojector) 1115 using intrinsic and extrinsic camera parameters 1130 to determine a camera-ball line in 3D space from the camera 1105 to the ball, and an intersection module (intersector) 1120 using a 3D model 1135 of a portion of a golf course to determine an intersection point of the camera-ball line with the 3D model 1135. The processing unit further includes an output generator 1125 for outputting the determined information for use, for example, during broadcast. FIG. 11 is described below in conjunction with FIG. 12.

[0237] 12 shows an example diagram 1200 including a camera 1105, a golf course terrain 1205, and a corresponding 3D model 1135 superimposed on the golf course terrain 1205, in accordance with various example embodiments. The diagram 1200 includes a golf ball trajectory 1210, including an impact location 1215 and a final resting location 1220 of the ball, and an example camera-ball line 1225 that intersects with the 3D model 1135 at the ball's final resting location 1220.

[0238] The camera 1105 has a sensor and lens selected to provide a field of view (FOV) of a portion of the golf course, e.g., the green and its surroundings including the fairway, toward which tracking is performed using the tracking system 1000, toward the direction where a shot is most likely to be hit. Alternatively, if the camera 1105 is positioned to capture a shot landing on the fairway, the FOV of the camera 1105 may cover the portion of the fairway where a shot from a tee or an approach shot to a green is most likely to land, e.g., the terrain 1205 shown in FIG. 12. The resolution of the camera 1105 is selected to provide enough pixels of the ball for the ball detector 1110 to operate efficiently. For example, the minimum cross-section of the ball seen in a captured image should be in the range of 3-10 pixels. The camera captures a series of images 1190 (...,i (n-1) ,i (n), i (n+1),... ) for a particular frame rate f s In a preferred embodiment, the captured images i (n) is at least the time t n The metadata may also include the exposure time for the image, the crop area of ​​the sensor that captured the image, and potentially other parameters.

[0239] Once captured, each image from the sequence of images 1190 captured by the camera is passed to a ball detector 1110, which includes one or more algorithms for determining whether a ball is present in a particular image 1195 and the pixel location (u,v) of the ball. The algorithms may include a convolutional deep neural network (CNN) trained to detect golf balls in images. If a ball is detected in the image, the ball detector 1110 sends the (u,v) coordinates, and possibly the image itself and some of the metadata, to a non-projector 1115.

[0240] The nonprojector 1115 receives information from the ball detector 1110 including at least the (u,v) coordinates of the ball and reads the stored intrinsic and extrinsic calibration parameters 1130 from memory or external storage. With this information, as shown in FIG. 12, the nonprojector 1115 determines a camera-ball line 1225 in 3D space including a straight line passing through the focal point of the camera 1105 in the direction of the ball based on the (u,v) coordinates of the image plane and the intrinsic and extrinsic parameters 1130 of the camera. If the captured image does not cover the entire field of view of the camera 1105, the nonprojector 1115 also uses information about the crop region of the image to determine the camera-ball line 1225. The crop region may be part of the image metadata passed from the camera 1105 or ball detector 1110, or if constant for an entire series of images, it may be stored in the system 1100 during calibration or system initialization and made accessible to the nonprojector 1115 in a manner similar to the stored calibration parameters 1130. The deprojector 1115 sends a representation of the camera-ball line 1125 to the intersection module 1120 .

[0241] The intersection module 1120 executes an intersection algorithm based on a camera-ball line 1225 and a 3D model 1135 of at least a portion of the golf course that overlaps with the camera's FOV. The intersection module 1120 receives the camera-ball line 1225 from the non-projector 1115, reads the 3D model 1135 from memory or external storage, and determines an intersection 1220 between the camera-ball line 1225 and the 3D model 1135. A 3D point representation of the intersection 1220 is used to determine the ball position on the course. When the ball is at rest, this intersection 1220, which may be represented as a 3D point in space, is the ball position on the course and the height of the terrain at the ball position. The method of determining the intersection may depend on the representation of the 3D model, for example, whether the 3D model is a triangle mesh, a spline surface, or any other surface representation. In some examples, the method may be iterative, or the intersection may be formulated as a solution to an optimization problem and solved using a numerical solver.

[0242] In scenarios where the ground is uneven or has small hills, or if the ball is located on the edge of a bunker, there may be multiple intersections between the 3D model 1135 and the camera-ball line 1225. For example, additional intersections 1230 and 1235 between the 3D model 1135 and the camera-ball line 1225 are shown in FIG. 12. In this case, the intersection module 1120 may determine the ball's position to be the intersection 1220 between the camera-ball line 1225 and the 3D model that is closest to the camera 1105.

[0243] The intersection module 1120 passes the 3D coordinates of the intersection 1220 to an output module 1125, which connects to any output means needed to use the ball position information, such as a database 1140 that stores information of shots hit during play; a 3D graphics rendering engine 1145 that can depict the ball position for a television or online audience, such as by a top view of the course; calculation and / or input into a graphical illustration of the distance from a static ball position to a tee position (for the actual carry, including bounce and roll); or calculation and / or input into a graphical illustration of the distance from the ball position to a flag or other interesting feature of the golf course, such as a bunker, pond, or green.

[0244] The system 1100 may include a radar 1150 and a radar tracker 1155 that can measure the ball position as a function of time, where the radar 1150 is calibrated to the same coordinate system as the tracking camera 1105, preferably calibrated to the world coordinate system 1240, and the radar 1150 is time-synchronized to the tracking camera 1105 so that each image can be correlated to the ball position measured by the radar. In this scenario, the ball position detected by the radar 1150 can be mapped to the image plane of the camera 1105 using a projector 1160 to narrow the search window of the ball detector 1110. This is particularly advantageous when the ball enters the field of view of the camera 1105, including, such as, the green and the area surrounding the green for the green-side camera 1105, because in some circumstances the ball detector 1110 only needs to be activated when the ball has previously been detected by the radar 1150.

[0245] The radar 1150 may be, for example, a continuous wave Doppler radar that radiates microwaves at X-band frequencies (10 GHz) with a power of up to 500 milliwatts EIRP (equivalent isotropically radiated power), and thus may comply with FCC and CE regulations for near field international radiators. However, in other jurisdictions, other power levels and frequencies may be used according to local regulations. In an exemplary embodiment, the microwaves are radiated at higher frequencies, for example, between 5 and 125 GHz. For more accurate measurements at lower object velocities, frequencies of 20 GHz or higher may be used. Any type of continuous wave (CW) Doppler radar may be used, including phase or frequency modulated CW radar, multi-frequency CW radar, or single frequency CW radar. It will be appreciated that other tracking devices, such as lidars, may be used with radiation in either the visible or invisible frequency ranges. Current pulse radar systems are limited in their ability to track objects near the radar device. However, the distance an object must be from these pulse radar systems has decreased over time and is expected to continue to decrease. Thus, these types of radars may soon be useful for these operations, and their use in the inventive systems described below is contemplated.

[0246] Any type of radar capable of tracking objects in three dimensions may be used. A MFCW Doppler radar operating anywhere in the X-band or permitted bands may have multiple receiving antennas spaced in a two-dimensional array to measure the phase difference between receivers, thereby deriving a direction to the ball in the local radar coordinate system using knowledge of the wavelength of the transmitted wave, and to transmit multiple frequencies to measure the phase difference, thereby deriving a range to the ball from knowledge of the wavelength of the transmitted wave.

[0247] The radar tracker 1155 is responsible for detecting moving objects of interest in the raw sensor data and, where possible, stitching together successive detections of the same moving object into a "track". A track is thus one or more detections of a moving object at successive times. Tracks may be smoothed to remove the effects of noise, and tracks may be represented in a way that allows positions along the track to be interpolated between measured positions or extrapolated both forward and backward in time beyond the measured positions.

[0248] To calibrate the radar 1150 to world coordinates, or any common coordinate system used for the camera 1105 and the 3D model 1135, the position and orientation of the radar needs to be determined. For this purpose, the radar unit may be equipped with a camera with known intrinsic parameters as described above and with a known relative position and orientation to the local radar coordinate system. The radar extrinsic parameters may then be determined as in determining the camera extrinsic parameters.

[0249] In a preferred embodiment, the radar 1150 and camera 1105 are built into one physical hardware unit with a known camera-to-radar calibration, which can be determined when the unit is manufactured. Thus, once the unit is calibrated to the golf course using the techniques described above or similar methods, both the radar 1150 and camera 1105 are calibrated to world coordinates and measurements can be referenced in the same coordinate system to a 3D model 1135 of a portion of the golf course.

[0250] In a preferred embodiment, the track of the golf ball in flight is determined using the position determined by the radar 1150 in combination with the detection of the ball detector 1110 based on images captured by the camera 1105. Thus, a more accurate ball track can be obtained. If a Kalman filter is used to smooth the radar position over time, the ball detection from the camera 1105 can be used as an input to the Kalman filter to increase the accuracy of the position determination. Using the enhanced radar trajectory, a more accurate ball landing position can be determined than a position determined using only radar measurements. The ball landing position can be determined as the intersection 1220 between the ball track and the 3D model 1135 of the portion of the golf course in a manner similar to the manner of determining the intersection between the camera-ball line and the 3D model, as described above.

[0251] In a preferred embodiment, the ball detector 1110 may be guided to reduce the search area to the vicinity of the expected ball location in the image, thereby reducing the calculations required to detect the ball. The ball location from the radar may be projected into the image through knowledge of the internal and external parameters of each sensor, or the ball detector may include a memory of the location of the ball detected in a previous image, so that for each new image received from the camera, the search area of ​​the ball may be determined using previous information about where the ball was detected in one or more previous images. This information may be stored in the ball detector 1110 as an array of (u,v) coordinates of the previous frame. The ball detector 1110 may alternatively include an array of previous images, so that when processing a new image, image information of the search area from several consecutive images is fed simultaneously to the neural network, improving the detection speed and accuracy of the neural network.

[0252] Additionally, the ball detector 1110 may be equipped to handle several balls within a search area. In a golf round or tournament, under normal conditions, only one ball at a time is moving on a particular hole, while the others are stationary. By using a neural network to detect multiple balls within a search area and using the detection information stored from previous images, it is possible to determine which ball detections represent stationary balls and which ball detections represent moving balls. Alternatively, a neural network trained to use an area of ​​several consecutive images for ball detection can determine, as part of its output, which detections represent stationary or moving balls. This allows the search window to be updated to follow a moving ball and not lose track of the ball, even when there are multiple balls in the image. The method described herein allows for the use of a limited search window after an initial ball detection is made when there is no radar in the system or when the radar tracker 1155 is no longer able to track the ball.

[0253] The ball detector 1110 determines when the moving ball comes to rest by comparing the (u,v) coordinates of the ball detection in two or more successive images. After this determination is made, the ball detector 1110 signals the unprojector 1115, the interceptor 1120, and the output module 1125 to perform the appropriate calculations and generate outputs as described above. If a previous radar track for a golf shot has been determined, the rest position of the stroke thus determined is connected to the radar measurements and preferably labeled with an identifier that is also used to label the radar track.

[0254] FIG. 17a illustrates a method 1700 for determining 3D position coordinates of a ball, according to various exemplary embodiments. A system for implementing the method 1700 includes at least a camera and a processing device for detecting the presence of a ball in an image, unprojecting the image coordinates to 3D coordinates, determining the camera-line ball in the 3D coordinates, and determining an intersection point of the camera-line ball. Various of the steps described above may be enhanced if radar data is additionally available, as described above. The intersection point of the 3D model and the camera-line ball determines the position of the ball in 3D coordinates. In some embodiments, the system determines whether the ball is stationary, and the position in the 3D coordinates is the stationary position.

[0255] At 1702, a camera captures a series of images. The camera has a sensor and lens selected to provide a field of view (FOV) of the portion of the golf course over which tracking will occur, and a resolution selected to provide enough pixels of the ball for the ball detector to operate effectively. The camera captures a series of images at a particular frame rate f s Let us consider a set of images (...,i (n-1) ,i (n) ,i (n+1), ...). The time at which the image was captured, t n Metadata can be included with the image, including such things as exposure time, crop area, and potentially other parameters.

[0256] At 1704, a processing device, e.g., a ball detection module thereof, in turn receives images from the camera and executes a ball detection algorithm on the images to determine whether a ball is in the image and the pixel location of the (u,v) coordinates of the detected ball. In some embodiments, the ball detector can use the radar data, e.g., to narrow the image area to be searched or to trigger the execution of the ball detection algorithm in time. The radar and camera may be incorporated into one physical hardware unit or may be located separately, as long as the radar and camera are commonly calibrated, e.g., to world coordinates. In some embodiments, the ball detector may provide ball detections to the radar / processing device to improve radar tracking and to improve the accuracy of subsequent ball detections. In one embodiment, the ball detector determines when a moving ball comes to rest by comparing the (u,v) coordinates of the ball detections in two or more consecutive images.

[0257] The ball detector can narrow the search area based on previous information, e.g., an array of (u,v) coordinates for ball detections in a previous frame. The ball detection algorithm may include a convolutional deep neural network (CNN) trained to detect golf balls in images, which can receive image information for the search area from an array of previous images to increase the detection speed and accuracy of the CNN. Additionally, the ball detection algorithm can be equipped to handle several balls in the search area, using logic to determine which ball detections represent a stationary ball and which represent a moving ball. In another embodiment, the neural network is trained to determine which detections represent a stationary or moving ball, allowing the search window to be updated without losing track of the ball.

[0258] At 1706, a processing unit, e.g., a non-projection module, reads the intrinsic and extrinsic calibration parameters of the camera and executes an algorithm using the (u,v) coordinates received from the ball detector to determine a camera-ball line that includes a straight line passing through the focal point of the camera in 3D space, e.g., the orientation of the ball in world coordinates. The processing unit may further use metadata associated with the images, e.g., a crop region, to determine the camera-ball line. The crop region may also be stored in the system if it is held constant for the entire series of images.

[0259] At 1708, a processing device, e.g., an intersection module, executes an intersection algorithm based on the camera-ball line and a 3D model of at least a portion of the golf course that overlaps with the camera's FOV to determine an intersection point of the camera-ball line with the 3D model. The intersection point is determined based on a representation of the 3D model, and in some embodiments, the process of determining the intersection point may be iterative or may use a numerical solver to solve an optimization problem. The camera and / or radar and / or 3D model are calibrated to a common coordinate system, which may be, for example, world coordinates. The 3D model is a representation of the golf course (or a portion of the golf course) stored in computer memory, and includes a surface model (e.g., a mesh of triangles, a spline surface, etc.) that represents the height and / or type of terrain at a given location on the course. The 3D model may be provided in world coordinates, such as for ease of mapping GPS locations, etc. to the 3D model. The 3D model may also be provided in a local coordinate system and have a mapping to world coordinates.

[0260] At 1710, a processing device, e.g., an output module, outputs the 3D position of the ball to some output means, e.g., a database, a 3D graphics rendering engine, a graphical depiction of the ball's position relative to other course features, etc.

[0261] When the ball is determined to be stationary, additional determinations may be made, as described in more detail below, such as the type of terrain on which the ball is currently located.

[0262] Bounce and Roll Tracking In another aspect of the present disclosure, camera-based tracking may be used to track and determine the bounce and roll of the ball. The determination of the bounce is preferably performed in the non-projector 1115 described above, but may also be performed in the ball detector 1110 based solely on the (u,v) coordinates. In one exemplary method, a series of images are processed to determine the presence of a bounce, as described below.

[0263] Figure 17b shows a method 1720 for determining the bounce and roll of a moving ball based on image data. A system for implementing the method 1720 may be similar to the systems described above in Figures 11 and 17a, including at least a camera and a processing unit including a ball detector and a projector. The method 1720 may be used, for example, to track chip shots, approach shots, or any other shot that exhibits a bouncing motion.

[0264] At 1722, the camera captures a series of images, similar to 1702 above. At 1724, similar to 1704, a processing device, e.g., its ball detection module, receives the images from the camera and determines whether a ball is present in the images and the pixel location of the (u,v) coordinates of the detected ball.

[0265] At 1726, after the series of images has been processed, the ball detector (or in some embodiments, the non-projector) generates a time series of ball detections in the (u,v) image plane from the series of images. At 1728, the non-projector converts the (u,v) coordinates into a series of camera-ball lines using the intrinsic and extrinsic camera calibrations available to the non-projector. The elevation angle of each line, i.e., the angle between the camera-ball line and the ground plane, is calculated, resulting in a time series of vertical projection angles from the camera.

[0266] Figure 13 shows a time series 1300 of ball detections in the (u,v) image plane from a sequence of images and a corresponding time series 1350 of the elevation angle relative to the camera-ball line determined by the projection lens 1115 described in Figure 11. The time series 1350 is analyzed for minima, which represent detections where the projection angle from the camera to the ball is steepest, shown at points 1352 and 1354 in the time series 1350, and which may correspond to a collision between the ball and the ground.

[0267] At 1730, the camera-ball lines of these minima in the time series 1350, which correspond to bounces, are passed to an intersection module to determine the 3D position of the bounce, as described above with respect to Figures 11 and 17a. Because the ball may be assumed to be on or very close to the ground for these detections, the 3D ball position may be accurately determined by the method used to determine the ball's rest position. The accuracy of the bounce detection may be improved to sub-frame accuracy by extrapolating either the (u,v) coordinates or the vertical projection angle, both before and after the time series minimum, to determine the most likely sub-frame time, vertical projection angle and camera-ball line of the bounce.

[0268] In systems that include radar, it is sometimes possible to track the bounce in the radar as a separate track and to track the signal from the rolling ball. In these cases, such information can be used to augment the bounce and roll tracking performed using only a camera. The radar measures the radial velocity of the ball and the distance to the ball, and because the radial velocity of the ball typically exhibits discontinuities as the ball bounces, very precise timing of the bounce can be determined by detecting the start and end points of the radar track that represents the bounce.

[0269] In 1732, bouncing is distinguished from rolling, where the time series of ball detections in the (u,v) plane 1300 is used in other ways herein, as described below.

[0270] A physical model of bouncing can be used to better distinguish between bouncing and rolling of the ball. In an exemplary method, given two consecutive minima in the (u,v) time series 1300 of pixel location detection, the position of the ball at each minimum may be determined from the projection as described above under the assumption that the ball hits the ground at each minimum, and the time of each assumed impact with the ground may be determined as the time of the image closest to each minimum, or alternatively, may be determined using an extrapolation of the (u,v) detection as described above. From the determined 3D position and the time of the start and end of the bouncing, an approximate 3D track of the ball may be determined based on the physical model 1165. A simple model may assume that the ball is only influenced by gravity, and therefore assumes that drag and lift forces (Magnus forces) are irrelevant at the low speeds and short distances associated with bouncing. The track determined from the physical model 1165 is projected onto the image plane of the camera 1105 and compared to the ball detection between the two (u,v) time series minima. A good correspondence indicates that the ball likely bounced during the minimum.

[0271] Detections in images of the (u,v) time series that are not determined to be bounces are classified as being part of a ball roll. Because the ball can be assumed to be on the ground while rolling, a track corresponding to the roll can be created by unprojecting the (u,v) positions and then determining the intersection between the camera-ball line and a 3D model of the course.

[0272] At 1734, this track is passed to the Output Module 1125 for output to the user, for overlaying on a broadcast, or for association with the main ball track by a common identifier and storage in a database. Thus, the ball track determined using the methods described above provides important information regarding bounce that is passed to the Output Module 1125.

[0273] Determining the terrain relative to the lie of the ball at rest In another embodiment, the ball's lie or terrain features in / on which the ball will be located for the next stroke can be determined. Two exemplary methods are described, or a combination of the two methods can be used. In one method, when the ball is determined to be stationary, the ball's position determined in the intersection module 1120 can be compared to information in the 3D model 1135 of the golf course to determine what terrain the ball is on. This can be, for example, fairway, semi-rough, rough, fringe, bunker, or green. In another method, the ball detection in the image can be classified as one of fairway, semi-rough, rough, fringe, bunker, or green based on how much of the ball is visible to the camera to characterize the length of grass, or a classification of the background of a small crop of the image centered on the ball. The latter determination can be made using a neural network trained to detect the terrain type of the lie based on a small crop of the image including the ball and the terrain.

[0274] If the ball is not visible in the image, the lie may be determined when the next stroke is filmed by capturing an image, detecting the golfer in the image, detecting the club in the image as the golfer addresses the ball, estimating a direction to the club head by projection, determining the approximate location of the golf club head, and identifying a terrain type as the terrain associated with this location in the 3D model of the course.

[0275] Chip shots and putts In another aspect, chip shots and putts may be tracked by the system by employing techniques similar to those described above for bounce and roll tracking. Tracking chips and putts is generally not possible with radar-only systems because shots are hit in any direction, either on the green or from the area immediately surrounding the green onto the green, and because the shots are often slow, resulting in low radial velocities measured by radar. Chip shots generally follow the arc of the initial shot and include one or more bounces and rolls. The arc of the initial shot can be tracked by tracking the bounce using the non-projection and intersection techniques described above. The bounce and roll of subsequent shots can be tracked similarly using the techniques already described.

[0276] A putt trajectory can be subdivided into up to three segments characterized by the physics of the moving ball: a bouncing segment; a sliding segment; and a rolling segment. During the bouncing segment, the ball exhibits a small bounce and the ball's speed decreases each time it bounces on the ground. During the sliding segment, the ball does not bounce, but the ball's rotation does not match the speed at which the ball travels on the ground, causing the ball to slide. In this segment, the ball loses speed, but the spin resulting from friction between the ball and the ground increases. During the rolling segment, the ball loses speed due to friction. Generally, the velocity decrease as a function of time is greatest during the bouncing segment and least during the rolling segment. Not all putts exhibit bouncing and sliding segments. These aspects of a putt can be measured and logic related to the principles discussed above can be used to identify putting segments in the tracking data. Further details regarding putting segments and the corresponding speed decrease are provided in U.S. Pat. No. 10,444,339, which is incorporated herein by reference in its entirety.

[0277] Putts can be tracked in a similar manner to chip shots. When tracking a putt trajectory using the presently disclosed camera and 3D model system, it is often useful to assume that the ball is on the ground during the entire trajectory, thereby reducing the putt tracking to a rolling track. Even if bounce tracking is not performed, the velocity of the ball during the putt trajectory can be matched to a 3D model to determine the bounce, slide and roll segments of the trajectory.

[0278] To limit power consumption and / or computational load on the system, the system 1100 can capture images at a low frequency (such as 1 fps) and perform person detection 1170. If person detection 1170 is successful (a person is detected in the image), address detection 1175 can be performed to determine whether a shot is about to be hit. If an address is detected, the system 1100 can switch the camera 1105 to a higher frame rate as specified above and begin ball detection 1110 in the area of ​​the image around the detected address. Prior knowledge of the position of the ball in the image, e.g., determined by the exemplary method for determining the resting position of the ball, can be used to determine whether person detection 1170 is likely to result in address detection 1175. For example, if a person is detected in the vicinity of the ball, then address is likely to occur. If no ball is detected in the image near the person detection, then address is likely not to occur. It is advantageous to implement the person detector 1170 and address detector 1175 as a convolutional neural network trained to detect people and detect golfers in address positions.

[0279] In another embodiment, a putting break fan can be determined from the rest position of the ball before the putt and a 3D model of the golf course. The 3D putt break fan displays different putt trajectories from the initial position of the ball, with changes in launch speed and launch direction that should result in a successful putt attempt. The expected successful putt attempt is determined as the ball trajectory that intersects the position of the hole at a speed that will cause the ball to fall into the cup. Generally, this depends on whether the ball falls more than half the ball diameter due to gravity when it is over the hole. Thus, a putt trajectory that intersects the center of the hole may result in a successful putt at a higher speed than a putt trajectory that has less impact on the hole. Using knowledge of the stimp of the green and the 3D model of the course, a simulated putt trajectory of the ball can be determined assuming a particular launch speed and launch direction of the putt. A method for determining the putt break fan is to simulate many putts with different launch speeds and launch directions and observe which launch conditions result in successful putts.

[0280] FIG. 14 shows a plot 1400 including a starting location 1405, a hole location 1410, and lines 1415-1425 representing possible putt trajectories that result in a successful putt. Line 1415 represents the highest launch rate that will result in a successful putt, line 1425 represents the lowest launch rate that will result in a successful putt, and line 1420 represents a launch rate between the highest and lowest launch rates that will result in a successful putt. Lines 1415 and 1425 correspond to the boundaries of the so-called putt break fan. Often, line 1420 represents a putt trajectory where the ball passes through the hole location and comes to rest at 2 feet if the cup is not present to interrupt the ball in its path. Putts between lines 1415 and 1420 are more likely to result in a short follow-up putt of less than 2 feet if the putt is not made, while putt trajectories between lines 1420 and 1425 will result in a follow-up putt of more than 2 feet.

[0281] FIG. 15 shows a putt break fan 1500 as a diagram 1510 of putt launch direction and launch speed combinations that result in a successful putt, and as contour lines 1515, 1520 showing the launch direction and launch speed combinations that result in a ball position with an equal distance to the pin for the second putt.

[0282] A method is provided for determining the tolerance of success of a putt. Once a successful putt has been hit and tracked by the system and the outcome determined, the tolerance to success may be determined by simulating small changes to launch speed and launch direction and the resulting simulated results. Similarly, for unsuccessful putts, it may be determined how much the launch conditions need to be changed in order for the putt to be successful. This information may be output as a two-dimensional diagram showing the combinations of launch speed and launch direction for a successful putt from the initial position of the putt.

[0283] In some situations, a golfer may not attempt to sink the putt into the hole, but instead attempt to execute a putt that places the ball in a more favorable position, perhaps closer to the pin, to increase the probability of the next putt being successful. The results of such a lay-up putt can be analyzed in a similar manner by simulated putting. An isocurve on a plot of launch velocity versus launch direction can indicate launch conditions that will result in a similar distance from the pin for the next shot.

[0284] In another embodiment, a reverse calculation is performed to determine the stimp of the green from the 3D ball track and a 3D model of the green. As will be understood by those skilled in the art, stimp is a measure of the green that describes the resistance applied to the movement of the ball by the surface (i.e., how much the surface quality slows the ball down while rolling, such as against the effect of gravity on a hill). Stimp can be determined by calculating the velocity decay, taking into account any elevation changes along the ball track, to measure the extent to which the ball is slowed down by the resistance of the green. Stimp varies across the green and is often direction-dependent, due to factors such as sunlight, irrigation, drainage, etc. However, based on the tracking of several putts, a 2D map of stimp across the green can be determined for further analysis of putt trajectory and resistance to successful putts.

[0285] In yet another embodiment, a camera-based track representing a known bounce can be used to update the calibration between the camera and the 3D model of the course. The accuracy of this calibration is important to accurately determine the camera-ball line when unprojected, and therefore to determine the correct ball position in the intersection module.

[0286] As described above, the bounce can be characterized by a physical model based on the 3D positions of the start and end of the bounce, as well as the time of the bounce. By projecting the ball position of the bounce onto the image plane, the projected ball pixel position and the pixel position of the ball detection can be compared. If the calibration from the camera to the 3D model is slightly changed, a new physical model of the bounce will result, which can be projected onto the image plane and compared to the ball detection. Thus, the calibration offset can be determined as the change to the calibration that best matches the projection of the physical model ball position to the detected ball position.

[0287] Similarly, if the distance to the ball can be determined from the radar at the start or end time of the bounce, the range determined by the radar may be compared to the distance to the ball determined in the intersection module, and the calibration of the camera to the 3D model may be updated to obtain a range from the camera to the ball in the intersection module that best matches the range to the ball measured by the radar. Such updates may be applied after several detections to more accurately determine the calibration update to apply, or an adaptive filter may be used to remove noise on the detection of the calibration offset. Conversely, if the calibration of the camera to the 3D model is more trusted, the radar range measurements may be updated to reflect the position of the bounce determined from the camera and ball detection in the 3D model.

[0288] 17c illustrates a method 1740 for determining information of an upcoming putt, according to various exemplary embodiments. A system for implementing the method 1740 may be similar to the systems described above in FIGS. 11 and 17a-b, including at least a camera and a processing unit including a ball detector, a non-projector, and a crossing module. The method 1740 may be used, for example, to generate an analysis of an upcoming putt.

[0289] At 1742, as above, the camera captures a series of images, the processor determines whether a ball is in the images and the pixel location of the (u,v) coordinates of the detected ball, the processor determines a camera-ball line, and the processor determines the intersection point of the camera-ball line with the 3D model.

[0290] The 3D position of the ball is the rest position of the ball on the green. Based on this rest position, information about the upcoming putt can be determined.

[0291] In 1744, knowledge of the stimp of the green and a 3D model of the course can be used to determine a simulated putt trajectory for the ball, assuming a particular launch speed and launch direction of the putt. Many putts are simulated with different launch speeds and launch directions to observe which launch conditions result in successful putts.

[0292] In 1746, various simulated launch conditions are mapped onto a Pat-Blake-Fan diagram.

[0293] 17d illustrates a method 1750 for determining topographical parameters of a putting green based on tracked putts, according to various exemplary embodiments. A system for implementing the method 1750 may be similar to the systems described above in FIGS. 11 and 17a-c, including at least a camera and a processing unit including a ball detector, a non-projector, and an intersection module. The method 1750 may be used, for example, to analyze the condition of a green based on putts made.

[0294] At 1752, as above, the camera captures a series of images, the processor determines whether a ball is in the images and the (u,v) coordinate pixel location of the detected ball, the processor determines the camera-ball line, and the processor determines the intersection point of the camera-ball line with the 3D model. However, in this step, the camera performs this position determination function to track the current putt.

[0295] Parameters of the putt are extracted, including velocity decay of the tracked putt, at 1754. The velocity decay may be characterized for the entire putt or multiple segments of the putt.

[0296] At 1756, the parameters of the putt are refined based on the elevation difference encountered by the putted ball along its path. This step is preferably performed based on a 3D model of the green. However, the elevation difference may also be tracked directly by a tracking device.

[0297] In 1758, the stimp of the green is determined. As mentioned above, stimp varies across the green and is often direction dependent due to factors such as sunlight, irrigation, etc. However, based on the tracking of several putts, a 2D map of stimp across the green can be determined for further analysis of putt trajectory and resistance to successful putts.

[0298] The present disclosure relates to a system that includes: a database configured to store profiles including visual profiles of a plurality of sports players, each visual profile including identification information of a player, each profile including an associated player ID; a camera configured to capture a video stream including images of the sports players; a tracking device configured to capture data corresponding to a trajectory of a sports ball fired by the sports player; and a processing device coupled to the database, the camera, and the tracking device, configured to: detect a first sports player in images from the video stream and determine visual characteristics of the first detected sports player; match the determined visual characteristics with a first visual profile and an associated first player ID of a first profile stored in the database; and associate a first trajectory associated with a first sports shot with the first player ID.

[0299] The visual profile includes information about the characteristics of the athlete's ball striking swing. The characteristics of the ball striking swing include biometric data points combined with the biomechanics of the ball striking swing. The athlete's biometric data points include height, limb length, and parameters dependent thereon. The biomechanics of the ball striking swing include the position of the limbs relative to the ball striking implement being swung, or the degree of twisting of the body relative to the legs. The visual profile includes information about the athlete's facial features or clothing features.

[0300] The clothing characteristics of the athlete are visually detected by the camera or a further camera at the start of the sporting event. A new profile and an associated visual profile are generated for a new athlete not yet identified in the database by analyzing the new athlete's ball-striking swing characteristics, facial characteristics, or clothing characteristics. The profile is generated in the database when the processing device receives an indication from the new athlete to generate a new profile and the camera or a further camera captures a video stream of the new athlete to analyze the new athlete's ball-striking swing characteristics, facial characteristics, or clothing characteristics. An electronic identifier carried by the first athlete and associated with the first athlete is positioned by the processing device to improve matching of the determined visual characteristics to the first visual profile. The electronic identifier includes a device that transmits GPS coordinates to the processing device. The athlete is a golfer, and the processing device is further configured to execute an algorithm utilizing golf theory to improve matching of the determined visual characteristics of the first detected athlete with the first visual profile and the associated first player ID by narrowing a list of profiles in the database that are considered when matching the determined visual characteristics of the first detected athlete with the first visual profile.

[0301] The golf theory includes a theory related to golfer groupings, golf hole layouts, or golf course layouts. The golf theory includes a theory related to a first estimated current lie of a sports ball associated with a first detected sports player, determined based on data from the tracking device. The algorithm includes a neural network trained to detect and match golfers to a visual profile. The neural network is updated throughout play based on changes in the golfer's visual characteristics. The neural network is updated after each sports player is detected and matched to a visual profile, after a number of sports players are detected and matched to a visual profile, or after a predetermined time since a previous player detection. The processing device is further configured to output the first trajectory parameters and the associated player ID to a broadcast. The processing device is further configured to output the first trajectory parameters to a personal device of the first sports player associated with the player ID prior to a next shot of the first sports player. The first trajectory parameters include a second measured or estimated lie location of the second sports ball after the second sports ball comes to rest, such that an application on the personal device can provide information to indicate the lie location and aid in locating the second sports ball.

[0302] The sportsperson is a golfer, and the first trajectory parameters include an indication that the ball hit by the first golfer stopped at a location in a first area of ​​a golf course on which the sportsperson is playing, the first area having a special rule associated therewith, and the parameters further include an indication of a second area in which the first golfer may take a next shot based on the location and the special rule. The processing device is further configured to accumulate a summary including the parameters for all trajectories of all shots hit by a first one of the sportspersons during the sporting event. The sportsperson is a golfer, and the processing device is further configured to: detect a type of golf club used by the first sportsperson for a first shot hit by the first sportsperson; and associate the detected golf club type with the first trajectory. The type of golf club is detected in the video stream using a neural network trained to recognize a plurality of different types of golf clubs. The type of golf club is detected based on a signal emitted from an electronic tag attached to the golf club when the golf club is being used. The processing device is further configured to output the first trajectory parameters to a database such that the database can associate the first trajectory parameters with the first visual profile.

[0303] The disclosure also relates to a method including detecting a first sports player in an image from a video stream captured by a camera, determining visual characteristics of the first detected sports player, matching the determined visual characteristics with a first visual profile and an associated first player ID of a first profile stored in a database, the database storing profiles including visual profiles of a plurality of sports players, each visual profile including a player identification information, each profile including an associated player ID, and associating a first trajectory associated with a first sports ball with the first player ID, the first trajectory being determined from data corresponding to trajectories of sports balls projected by the sports player captured by a tracking device.

[0304] The present disclosure also relates to a processor coupled to a database, a camera, and a tracking device configured to perform the following operations: detecting a first sports player in an image from a video stream captured by the camera; determining visual characteristics of the first detected sports player, matching the determined visual characteristics with a first visual profile and an associated first player ID of a first profile stored in a database, the database storing profiles including visual profiles of a plurality of sports players, each visual profile including a player identification information, each profile including an associated player ID; and associating a first trajectory associated with a first sports ball with the first player ID, the first trajectory being determined from data corresponding to a trajectory of a sports ball launched by the sports player captured by the tracking device.

[0305] The present disclosure also relates to a system that includes: a database configured to store profiles of each of a plurality of athletes, each profile including identification information of one of the players and a player ID associated with one of the athletes; a tracking device configured to capture shot data corresponding to a trajectory of a sports ball fired by the athletes; a motion sensor device configured to capture motion data corresponding to a swing motion of a player or a ball-striking equipment; and a processing device coupled to the database, the tracking device, and the motion sensor device, configured to: detect a first swing of a first sports player from the motion data captured by the motion sensor device, the first sports player being associated with a first player ID; associate the first swing of the first sports player with a first timestamp and a first location detected for the first swing; and associate a first trajectory of the sports ball from the shot data captured by the tracking device corresponding to a first shot corresponding to the first swing based on the first timestamp and the first location.

[0306] The system stores swing data corresponding to the first swing, the first timestamp, the first location and subsequent detection of an additional swing having a further timestamp and a further location corresponding to the additional swing, the first swing and the additional swing being matched to a trajectory of the first swing and the additional swing at the end of the first athlete's play based on the corresponding first timestamp and first location and the further timestamp and further location, respectively. The motion sensor device or a further device associated with the motion sensor device has GPS capability for determining a location of the first swing. An electronic identifier carried by and associated with the first athlete is detected and a location of the first swing is determined. The athlete is a golfer, and the processing unit is further configured to execute an algorithm utilizing golf theory to improve the match between the first swing and the first trajectory by narrowing a list of profiles in the database that are considered in making the match.

[0307] The golf theory includes a theory related to a grouping of golfers, a golf hole layout, or a golf course layout. The golf theory includes a theory related to an estimated current lie of the first sports player's ball determined based on the data from the tracking device. The processing device is further configured to output the first trajectory parameters to a personal device of the first sports player associated with the player ID prior to a next shot of the first sports player. The first trajectory parameters include an estimated lie location for one of the corresponding sports balls after the corresponding sports ball comes to rest such that an application on the personal device can provide data facilitating locating the corresponding sports ball. The sports players are golfers playing on a golf course, and the first trajectory parameters include an indication that the ball hit by the first golfer came to rest at a location within a first area of ​​the golf course, the first area having a special rule associated therewith, and the parameters further include an indication of a second area in which the first golfer may take a next shot according to the location and the special rule.

[0308] The processing device is further configured to accumulate a summary including parameters for all trajectories of all shots hit by the first athlete during the sporting event. The athlete is a golfer, and the processing device is further configured to: detect a type of golf club used by the first athlete for the first golf shot; and associate the detected type of golf club with the first trajectory. The type of golf club is detected based on a signal emitted from an electronic tag attached to the golf club when the club is used.

[0309] The disclosure also relates to a method that includes detecting a first swing of a first athlete associated with a first player ID from motion data captured by a motion sensor device, where the motion sensor device captures motion data corresponding to a swinging motion of a player or ball-striking equipment, and a database stores profiles for each of a plurality of athletes, each profile including an identification of one of the players and a player ID associated with one of the athletes; associating the first swing of the first athlete with a timestamp and a location; and associating a first trajectory of a sportsball with the first swing based on the timestamp and the location, where the first trajectory is determined from data corresponding to a trajectory of a sportsball launched by the athlete captured by a tracking device.

[0310] The disclosure also relates to a processor coupled to the database, the motion sensor device, and the tracking device configured to perform the following operations: detecting a first swing of a first athlete associated with a first player ID from motion data captured by the motion sensor device, where the motion sensor device captures motion data corresponding to a swing motion of a player or ball-striking equipment, and the database stores profiles for each of a plurality of athletes, each profile including an identification of one of the players and a player ID associated with one of the athletes; and associating the first swing of the first athlete with a timestamp and a location; and associating a first trajectory of a sportsball with the first swing based on the timestamp and the location, where the first trajectory is determined from data corresponding to a trajectory of a sportsball launched by the athlete captured by the tracking device.

[0311] The present disclosure also relates to a system that includes: a database configured to store a profile for each of a plurality of athletes, each profile including an identification of one of the players and a player ID associated with one of the athletes; a tracking device configured to capture shot data corresponding to a trajectory of a sportsball launched by the athletes; a positioning device associated with a first of the plurality of athletes associated with a first player ID, the positioning device configured to capture position data approximating a position of the positioning device at a given time; and a processing device coupled to the database, the tracking device, and the positioning device configured to: detect a first trajectory of the first sportsball within the shot data; determine a first location at which the first sportsball was launched and a first time at which the first sportsball was launched; receive position data including positions of the positioning device over a duration that includes the first time, each position associated with a timestamp; determine that a first time associated with the first position and the first trajectory matches a first location and a first timestamp of the position data; and associate the first trajectory of the first sportsball with the first player ID.

[0312] Matching the first position and first time with the first location and first timestamp includes: determining the first position and the first location in a world coordinate system; and determining a correspondence between the first position and the first location within a predetermined distance. The system stores position data of a first round of golf for the first athlete, and trajectories detected in the shot data are respectively matched with the position data at the end of the first round of golf. The position determining device has GPS capabilities for determining the position data. The athlete is a golfer, and the processing unit is further configured to run an algorithm utilizing golf theory to improve the match between the first position and first time with the first location and first timestamp by narrowing a list of profiles in the database that are considered in making the match.

[0313] The golf theory includes a theory related to a grouping of golfers, a golf hole layout, or a golf course layout. The golf theory includes a theory related to an estimated current lie of the first sportsman's ball determined based on the data from the tracking device. The processing device is further configured to output the first trajectory parameters to a personal device of the first sportsman associated with the player ID prior to a next shot of the first sportsman. The first trajectory parameters include an estimated lie location for the sports ball corresponding to the first trajectory after the sports ball corresponding to the first trajectory comes to rest such that an application on the personal device can provide data facilitating finding the sports ball corresponding to the first trajectory. The sportsman is a golfer playing on a golf course, and the first trajectory parameters include an indication that the ball hit by the first golfer came to rest at a location within a first area of ​​the golf course, the first area having a special rule associated therewith, and the parameters further include an indication of a second area in which the first golfer may take a next shot according to the location and the special rule.

[0314] The processing device is further configured to accumulate a summary including parameters for all trajectories of all shots hit by the first athlete during the sporting event. The athlete is a golfer, and the processing device is further configured to: detect a type of golf club used by the first athlete for a first golf shot corresponding to the first trajectory; and associate the detected type of golf club with the first trajectory. The type of golf club is detected based on a signal emitted from an electronic tag attached to the golf club when the club is used.

[0315] The disclosure also relates to a method that includes: detecting a first trajectory of a first sports ball in shot data captured by a tracking device corresponding to a trajectory of a sports ball launched by the sports player; determining a first location where the first sports ball was launched and a first time where the first sports ball was launched; receiving position data from a position determining device associated with a first sports player associated with a first player ID, where a database stores a profile for each of a plurality of sports players, each profile including an identification of one of the players and a player ID associated with one of the sports players, the position data approximating a position of the position determining device at a given time, the position data including positions of the position determining device over a duration that includes the first time, each position being associated with a timestamp; determining that a first time associated with the first position and the first trajectory matches the first location and the first timestamp of the position data; and associating the first trajectory of the first sports ball with the first player ID.

[0316] The disclosure also relates to a processor coupled to the database, the tracking device, and the position determination device configured to perform operations including: detecting a first trajectory of a first sports ball in shot data captured by the tracking device corresponding to a trajectory of a sports ball launched by the sports player, determining a first location at which the first sports ball was launched and a first time at which the first sports ball was launched; receiving position data from a position determination device associated with a first sports player associated with a first player ID, where the database stores a profile for each of a plurality of sports players, each profile including an identification of one of the players and a player ID associated with one of the sports players, the position data approximating a position of the position determination device at a given time, the position data including positions of the position determination device over a duration that includes the first time, each position being associated with a timestamp; determining that a first time associated with the first position and the first trajectory matches the first location and the first timestamp of the position data; and associating the first trajectory of the first sports ball with the first player ID.

[0317] The present disclosure also relates to a system that includes: a robotic camera configured to capture data corresponding to a position of a sports ball, the robotic camera further configured to automatically adjust an orientation and zoom level of the robotic camera in response to the captured data or a command; and a processing device coupled to the robotic camera configured to: pre-calibrate the robotic camera such that an initial orientation of the robotic camera is known in a world coordinate system, and associate each of a plurality of different zoom levels used by the robotic camera with a respective intrinsic parameter value; detect a first image position of the sports ball in an image coordinate system of a first image, the first image being captured by the robotic camera at the first orientation; read from the robotic camera a first zoom level associated with the first image and an intrinsic parameter value associated with the first zoom level; determine a first orientation of the robotic camera based on a pan and tilt of the robotic camera relative to the initial orientation; determine a three-dimensional line that passes through the robotic camera and the sports ball in the world coordinate system based on the detected first image position of the sports ball, the intrinsic parameter value of the determined first orientation and the first zoom level; and determine a three-dimensional position of the sports ball in the world coordinate system that lies along the three-dimensional line based on extrinsic information for the robotic camera.

[0318] At a first position of the image, the sports ball is on a surface of the sports playing area, the extrinsic information for the robotic camera includes information received from a three-dimensional model of at least a portion of the sports playing area encompassing the three-dimensional position of the sports ball, and the processing unit determines the three-dimensional position of the sports ball by locating an intersection of a line with the three-dimensional model. The system further includes a pan and tilt sensor affixed to the robotic camera, and the processing unit is further configured to read a pan and tilt of the robotic camera relative to an initial orientation from the pan and tilt sensor.

[0319] The pan and tilt of the robotic camera are determined from a control signal used to adjust the robotic camera to a first orientation, the control signal being generated in response to tracking data of the sports ball captured prior to capture of the first image. The robotic camera is further configured to automatically adjust the crop in response to the captured data or an external command. The intrinsic parameters include focal length, principal point, and lens distortion. The extrinsic information for the robotic camera includes information from a sensor regarding the distance of the sports ball from the robotic camera.

[0320] The disclosure also relates to a method that includes: pre-calibrating a robotic camera such that an initial orientation of the robotic camera is known within a world coordinate system, and associating each of a plurality of different zoom levels used by the robotic camera with a respective intrinsic parameter value, the robotic camera being configured to capture data corresponding to a position of a sports ball, the robotic camera being further configured to automatically adjust the orientation and zoom level in response to the captured data or a command; detecting a first image position of the sports ball in an image coordinate system of a first image, the first image being captured by the robotic camera at the first orientation. , detecting; reading a first zoom level from the robotic camera and associated intrinsic parameter values ​​for the first zoom level of the robotic camera corresponding to the first image; determining a first orientation of the robotic camera based on a pan and tilt of the robotic camera relative to an initial orientation; determining a three-dimensional line passing through the robotic camera and the sports ball in a world coordinate system based on the detected first image position of the sports ball, the determined first orientation and the intrinsic parameter values ​​of the first zoom level; and determining a three-dimensional position of the sports ball in the world coordinate system located along the three-dimensional line based on extrinsic information for the robotic camera.

[0321] The present disclosure also relates to a processor coupled to a robotic camera configured to perform operations including: pre-calibrating the robotic camera such that an initial orientation of the robotic camera is known in a world coordinate system and each of different zoom levels used by the robotic camera is associated with a respective intrinsic parameter value, the robotic camera being configured to capture data corresponding to a position of a sports ball, the robotic camera being further configured to automatically adjust the orientation and zoom level in response to the captured data or a command; detecting a first image position of the sports ball in an image coordinate system of the first image, the first image being associated with the first orientation. detecting a first image position of the sports ball captured by the robotic camera in a first orientation; reading a first zoom level from the robotic camera and associated intrinsic parameter values ​​for the first zoom level corresponding to the first image; determining a first orientation of the robotic camera based on a pan and tilt of the robotic camera relative to an initial orientation; determining a three-dimensional line passing through the robotic camera and the sports ball in a world coordinate system based on the detected first image position of the sports ball, the determined first orientation and the intrinsic parameter values ​​for the first zoom level; and determining a three-dimensional position of the sports ball in the world coordinate system located along the three-dimensional line based on extrinsic information for the robotic camera.

[0322] The disclosure also relates to a system that includes: a robotic camera having a predetermined position in a world coordinate system, the robotic camera configured to capture data corresponding to a position of a sports ball, the robotic camera further configured to automatically adjust an orientation and zoom level in response to the captured data or a command; and a processing device coupled to the robotic camera configured to: detect a first image position of the sports ball in an image coordinate system of a first image, the first image being captured with the robotic camera in a first orientation; detect a reference point in the image coordinate system in the first image or in a further image captured after an initial adjustment of the robotic camera's orientation to bring a reference point into its field of view, the three-dimensional position of the reference point being known in the world coordinate system; adjust an orientation of the robotic camera to a second orientation such that the reference point is located at the first image position in the second image; determine an orientation difference between the first orientation and the second orientation; determine a three-dimensional line that passes through the robotic camera and the sports ball in the world coordinate system based on the three-dimensional position of the reference point relative to the robotic camera and the orientation difference between the first orientation and the second orientation; and determine a three-dimensional position of the sports ball in the world coordinate system that is located along the three-dimensional line.

[0323] In the first image position, the sports ball is on a surface of the sports playing area and a three-dimensional position of the sports ball is determined based on identifying an intersection between a line and a three-dimensional model of at least a portion of the sports playing area that encompasses the three-dimensional position of the sports ball.

[0324] The second image is captured without adjusting the zoom level relative to the first image. The system further comprises a pan and tilt sensor affixed to the robotic camera, and the processing unit is further configured to read an orientation difference between the first orientation and the second orientation from the pan and tilt sensor.

[0325] The disclosure also relates to a method that includes: detecting a first image position of a sports ball in an image coordinate system in a first image, the first image being captured by a robotic camera at a first orientation, the robotic camera having a predetermined position in a world coordinate system, the robotic camera being configured to capture data corresponding to a position of the sports ball, the robotic camera being further configured to automatically adjust an orientation and zoom level in response to the captured data or a command; detecting a reference point in the image coordinate system in the first image or in a further image captured after an initial adjustment of the robotic camera orientation to bring the reference point into its field of view, the three-dimensional position of the reference point being known in the world coordinate system; adjusting an orientation of the robotic camera to a second orientation such that the reference point is located at the first image position in the second image; determining an orientation difference between the first orientation and the second orientation; determining a three-dimensional line that passes through the robotic camera and the sports ball in the world coordinate system based on the three-dimensional position of the reference point relative to the robotic camera and the orientation difference between the first orientation and the second orientation; and determining a three-dimensional position of the sports ball in the world coordinate system that lies along the line.

[0326] The present disclosure also relates to a robotic camera configured to perform operations including: detecting a first image position of a sports ball within an image coordinate system in a first image, the first image being captured by the robotic camera at a first orientation, the robotic camera having a predetermined position in a world coordinate system, the robotic camera configured to capture data corresponding to the position of the sports ball, the robotic camera further configured to automatically adjust an orientation and zoom level in response to the captured data or a command; detecting a first image position of a sports ball within an image coordinate system in the first image or the robotic camera to capture a reference point within its field of view. detecting a reference point in a further image captured after an initial adjustment of the robot camera's orientation, the three-dimensional position of the reference point being known in the world coordinate system; adjusting the orientation of the robot camera to a second orientation such that the reference point is located at the first image position in the second image; determining an orientation difference between the first orientation and the second orientation; determining a three-dimensional line that passes through the robot camera and the sports ball in the world coordinate system based on the three-dimensional position of the reference point relative to the robot camera and the orientation difference between the first orientation and the second orientation; and determining a three-dimensional position of the sports ball in the world coordinate system that is located along the three-dimensional line.

[0327] The disclosure also relates to a system that includes: a first camera, the first camera being a robotic camera configured to capture first data corresponding to a position of a sports ball, the robotic camera further configured to automatically adjust an orientation and zoom level in response to the captured data or a command; a second camera, calibrated to a world coordinate system, configured to capture second data corresponding to a position of the sports ball; and a processing device coupled to the robotic camera and the second camera, configured to: detect a first image position of the sports ball in an image coordinate system of a first image, the first image being captured at a first orientation of the robotic camera; detect at least a first feature in the first image; detect the first feature in a second image captured by the second camera; determine a three-dimensional position of the first feature in the world coordinate system based on the second camera calibration; perform feature matching between the first and second images based on the first feature and identify the first image position in the second image; determine a three-dimensional line that passes through the robotic camera and the sports ball in the world coordinate system based on the three-dimensional position of the first feature in the world coordinate system and the first image position of the sports ball; and determine a three-dimensional position of the sports ball in the world coordinate system that lies along the three-dimensional line.

[0328] In the first image location, the sports ball is on a surface of a sports playing area and a three-dimensional position of the sports ball is determined based on an intersection between a line and a three-dimensional model of at least a portion of the sports playing area that encompasses the three-dimensional position of the sports ball. The second camera is co-located with the first camera. The sports playing area is a golf course and the first features include trees, bunkers, a pond, a golf flag, and a green.

[0329] The disclosure also relates to a method that includes: detecting a first image position of a sports ball in an image coordinate system in a first image, the first image being captured with a first orientation of a first camera, the first camera being a robotic camera configured to capture first data corresponding to a position of the sports ball, the robotic camera being further configured to automatically adjust an orientation and zoom level in response to the captured data or a command; detecting at least a first feature in the first image; detecting a first feature in a second image captured by a second camera, the second camera being calibrated to a world coordinate system and configured to capture second data corresponding to a position of the sports ball; determining a three-dimensional position of the first feature in the world coordinate system based on the second camera calibration; performing feature matching between the first and second images based on the first feature and locating the first image position in the second image; determining a three-dimensional line that passes through the robotic camera and the sports ball in the world coordinate system based on the three-dimensional position of the first feature in the world coordinate system and the first image position of the sports ball; and determining a three-dimensional position of the sports ball in the world coordinate system that lies along the three-dimensional line.

[0330] The present disclosure also relates to a processing device coupled to a first camera and a second camera configured to perform the following operations: detecting a first image position of a sports ball within an image coordinate system in a first image, the first image being captured at a first orientation of the first camera, the first camera being a robotic camera configured to capture first data corresponding to a position of the sports ball, the robotic camera being further configured to automatically adjust an orientation and zoom level in response to the captured data or a command; detecting at least a first feature in the first image; detecting the first feature in a second image captured by the second camera. detecting, wherein the second camera is calibrated to a world coordinate system and configured to capture second data corresponding to a position of the sports ball; determining a three dimensional position of a first feature in the world coordinate system based on the second camera calibration; performing feature matching between the first and second images based on the first feature and locating a first image position in the second image; determining a three dimensional line that passes through the robotic camera and the sports ball in the world coordinate system based on the three dimensional position of the first feature in the world coordinate system and the first image position of the sports ball; and determining a three dimensional position of the sports ball in the world coordinate system that lies along the three dimensional line.

[0331] The present disclosure also relates to a system that includes: a robotic camera configured to capture data corresponding to a position of a sports ball, the robotic camera further configured to automatically adjust an orientation and a zoom level in response to the captured data or a command; and a processing device coupled to the robotic camera configured to: detect an image position of the sports ball in an image coordinate system in a plurality of images, a first image position of the sports ball in a first image captured at a first orientation of the robotic camera; adjust a zoom level of the robotic camera to a second zoom level such that a first reference point and a second reference point are located within a field of view of the robotic camera, and the image position of the sports ball is detected when the zoom level is adjusted. detect a second image location of the first reference point and a third image location of the second reference point in the image coordinate system in the second image; determine a fourth image location of the sports ball in the second image; determine an angular location of the fourth image location relative to the second image location and the third image location by correlating the second image location and the third image location to the first reference point and the second reference point relative to the robotic camera at a predetermined angle; determine a three dimensional line that passes through the robotic camera and the sports ball in the world coordinate system based on the first image location, the second zoom level and the angular location of the first image location relative to the second and third image locations; and determine a three dimensional location of the sports ball in the world coordinate system that lies along the three dimensional line.

[0332] In a first image position, the sports ball is on a surface of the sports playing area and a three-dimensional position of the sports ball is determined by identifying an intersection between a line and a three-dimensional model of at least a portion of the sports playing area that encompasses the three-dimensional position of the sports ball. A second image is captured without adjusting the orientation relative to the first image. If the sports ball is not visible in the second image, a fourth image position of the sports ball in the second image is estimated based on tracking the image position of the sports ball while the zoom level is adjusted.

[0333] The disclosure also relates to a method that includes: detecting an image position of a sports ball in a plurality of images in an image coordinate system, where a first image position of the sports ball in a first image is captured at a first orientation of a robotic camera, the robotic camera configured to capture data corresponding to a position of the sports ball, the robotic camera further configured to automatically adjust an orientation and a zoom level in response to the captured data or a command; adjusting a zoom level of the robotic camera to a second zoom level such that the first reference point and a second reference point are located within a field of view of the robotic camera, where the image position of the sports ball is tracked while the zoom level is adjusted; adjusting the second image position of the sports ball in a plurality of images in an image coordinate system, where a first image position of the sports ball in a first image is captured at a first orientation of a robotic camera, the robotic camera configured to capture data corresponding to a position of the sports ball, the robotic camera further configured to automatically adjust an orientation and a zoom level in response to the captured data or a command. determining a second image location of the first reference point and a third image location of the second reference point in an image coordinate system within the robotic camera; determining a fourth image location of the sports ball in the second image; determining an angular location of the fourth image location relative to the second image location and the third image location by correlating the second image location and the third image location to the first reference point and the second reference point relative to the robotic camera at a predetermined angle; determining a three-dimensional line that passes through the robotic camera and the sports ball in the world coordinate system based on the first image location, the second zoom level, and the angular location of the first image location relative to the second and third image locations; and determining a three-dimensional position of the sports ball in the world coordinate system that lies along the three-dimensional line.

[0334] The present disclosure also relates to a processor coupled to a robotic camera configured to perform operations including: detecting an image position of a sports ball in an image coordinate system in a plurality of images, where a first image position of the sports ball in a first image is captured at a first orientation of the robotic camera, the robotic camera configured to capture data corresponding to the position of the sports ball, the robotic camera further configured to automatically adjust an orientation and a zoom level in response to the captured data or a command; adjusting a zoom level of the robotic camera to a second zoom level such that the first reference point and a second reference point are located within a field of view of the robotic camera, where the image position of the sports ball is tracked while the zoom level is adjusted. detecting a second image location of the first reference point and a third image location of the second reference point in the image coordinate system in the second image; determining a fourth image location of the sports ball in the second image; determining an angular location of the fourth image location relative to the second image location and the third image location by correlating the second image location and the third image location to the first reference point and the second reference point relative to the robotic camera at a predetermined angle; determining a three-dimensional line that passes through the robotic camera and the sports ball in the world coordinate system based on the first image location, the second zoom level, and the angular location of the first image location relative to the second and third image locations; and determining a three-dimensional location of the sports ball in the world coordinate system that lies along the three-dimensional line.

[0335] The present disclosure also relates to a system that includes: a first sensor having a sensor field of view configured to capture data corresponding to a trajectory of a sports ball; a first broadcast camera having a first field of view configured to capture a first video stream including a first image of a first portion of the trajectory of the sports ball; and a processing device coupled to the first sensor and the first broadcast camera that is configured to: calibrate the first sensor to the first broadcast camera such that pixel orientations in the first image are known relative to a coordinate system of the first sensor, the calibration further including a relationship between parameters of the first sensor and the first broadcast camera at different zoom levels; generate a broadcast video feed using the first video stream including first images of the first portion of the sports ball trajectory as the sports ball traverses the first field of view; determine trajectory parameters based on the data of the sports ball trajectory as the sports ball traverses the sensor field of view; and insert a tracer indicative of a path of the sports ball into the broadcast video feed, the tracer being generated based on the calibration between the first sensor and the first broadcast camera.

[0336] The first sensor is a tracking camera. The processing unit is further configured to determine a current zoom level of the first broadcast camera by feature matching between images from the tracking camera and the first broadcast camera. The tracer is inserted into the broadcast video feed regardless of the current zoom level of the first broadcast camera based on the calibration of the different zoom levels. The position of the sports ball is detected in one or more of the first images prior to launch to improve rendering of the starting point of the tracer. A ball identification algorithm searches for the position of the sports ball in the first images. A player's stance is detected prior to launch to locate the sports ball. The system of claim 28 further includes a second broadcast camera having a second field of view configured to capture a second video stream including a second image of a second portion of the sports ball's trajectory, and the processing device is further configured to: calibrate the first sensor to the second broadcast camera such that pixel orientations in the second image are known relative to a coordinate system of the first sensor, the calibration further including a relationship between parameters of the first sensor and the second broadcast camera for different zoom levels; switch the broadcast video feed to the second video stream including the second image of the second portion of the sports ball's trajectory as the sports ball traverses the second field of view; and further insert a tracer into the broadcast video feed based on the calibration between the first sensor and the second broadcast camera.

[0337] The disclosure also relates to a method that includes: calibrating a first sensor to a first broadcast camera, the first sensor having a sensor field of view and configured to capture data corresponding to a trajectory of a sports ball, the first broadcast camera having a first field of view configured to capture a first video stream including a first image of a first portion of the trajectory of the sports ball, the first sensor and the first broadcast camera being calibrated such that pixel orientations in the first image are known relative to a coordinate system of the first sensor, the calibration further including a relationship between parameters of the first sensor and the first broadcast camera at different zoom levels; generating a broadcast video feed using the first video stream including the first image of the first portion of the trajectory of the sports ball as the sports ball traverses the first field of view; determining trajectory parameters based on the data of the trajectory of the sports ball as the sports ball traverses the sensor field of view; and inserting a tracer indicative of a path of the sports ball into the broadcast video feed, the tracer being generated based on a calibration between the first sensor and the first broadcast camera.

[0338] The present disclosure also relates to a processor coupled to a first sensor and a first broadcast camera configured to perform operations including: calibrating the first sensor to the first broadcast camera, the first sensor having a sensor field of view and configured to capture data corresponding to a trajectory of a sports ball, the first broadcast camera having a first field of view configured to capture a first video stream including a first image of a first portion of the trajectory of the sports ball, the first sensor and the first broadcast camera being calibrated such that pixel orientations in the first image are known relative to a coordinate system of the first sensor, the calibration further including a relationship between parameters of the first sensor and the first broadcast camera at different zoom levels; generating a broadcast video feed using the first video stream including the first image of the first portion of the trajectory of the sports ball as the sports ball traverses the first field of view; determining trajectory parameters based on the data of the trajectory of the sports ball as the sports ball traverses the sensor field of view; and inserting a tracer indicative of a path of the sports ball into the broadcast video feed, the tracer being generated based on a calibration between the first sensor and the first broadcast camera.

[0339] The disclosure also relates to a system that includes: a tracking camera having a first field of view configured to capture images in a camera coordinate system of a sports ball bouncing and rolling after an initial trajectory, the tracking camera having intrinsic and extrinsic calibration parameters associated therewith; a storage device including the intrinsic and extrinsic calibration parameters and a three-dimensional (3D) model of at least a portion of a sports playing area overlapping with the first field of view; and a processing device coupled to the tracking camera that is configured to: perform ball detection to detect a pixel location of the sports ball in the image; determine a camera-ball line, including a straight line passing through the camera, an orientation of the sports ball in the 3D coordinate system based on the intrinsic and extrinsic calibration parameters; determine an intersection point of the camera-ball line and the 3D model based on the 3D model; and output the intersection point as a 3D position of the sports ball in the image.

[0340] The processing device is further configured to: determine that the sports ball is stationary by comparing pixel locations of the sports ball in successive images; and output the intersection point as the 3D stationary location of the sports ball in the images. The captured images are associated with metadata including a time the image was captured, an exposure time, and a crop area. The system further includes a tracking radar having a second field of view at least partially overlapping with the first field of view configured to capture radar data of the sports ball, and the processing device is further configured to rely on the radar data to detect the sports ball to narrow the area searched in the images and the number of images searched in time. Detection of the sports ball using the radar data in combination with the images is used by the processing device to improve accuracy of the 3D location determination of the sports ball. The processing device is further configured to detect the sports ball using a neural network trained to detect sports balls in the images.

[0341] The neural network uses information on the search area from a previous image to improve accuracy of the detection of the sports ball in the current image. The processor is further configured to determine a camera-ball line based on a crop region of the image. The sports playing area is a golf course, and the 3D model of the portion of the golf course includes a surface model representing a terrain height at a given location on the golf course. The surface model further represents a terrain type at the predetermined location on the golf course. The processor is further configured to determine a terrain type on which the sports ball is to be detected based on the 3D model. The surface model includes a triangular mesh or a spline surface. The intersection points are determined using an iterative process or using a numerical solver to solve an optimization problem. The 3D location of the sports ball in the image is output to a database, a 3D graphics rendering engine, or a graphical representation of the 3D location of the sports ball. The 3D model is provided in a 3D coordinate system of the camera-ball line.

[0342] The disclosure relates to a method that includes: performing ball detection to detect a pixel location of a sports ball in an image captured by a tracking camera, the tracking camera having a first field of view configured to capture images in a camera coordinate system of the sports ball bouncing and rolling after an initial trajectory, the tracking camera having intrinsic and extrinsic calibration parameters associated therewith stored in a memory; determining a camera-ball line including a straight line passing through the camera, an orientation of the sports ball in a 3D coordinate system based on the intrinsic and extrinsic calibration parameters; determining an intersection point of the camera-ball line with a three-dimensional (3D) model based on the memory stored in the memory, the 3D model including at least a portion of a sports playing area that overlaps with the first field of view; and outputting the intersection point as a 3D position of the sports ball in the image.

[0343] The present disclosure also relates to a processor coupled to a tracking camera and a storage device configured to perform operations including: performing ball detection to detect a pixel location of a sports ball in an image captured by the tracking camera, the tracking camera having a first field of view configured to capture images of the sports ball bouncing and rolling in a camera coordinate system after an initial trajectory, the tracking camera having intrinsic and extrinsic calibration parameters associated therewith stored in the storage device; determining a camera-ball line including a straight line passing through the camera, an orientation of the sports ball in a 3D coordinate system based on the intrinsic and extrinsic calibration parameters; determining an intersection point of the camera-ball line with a three-dimensional (3D) model based on the storage device, the 3D model including at least a portion of a sports playing area that overlaps with the first field of view; and outputting the intersection point as a 3D position of the sports ball in the image.

[0344] The disclosure also relates to a system that includes: a tracking camera having a first field of view configured to capture images in an image plane of a camera coordinate system of a sports ball bouncing and rolling after an initial trajectory, the tracking camera having intrinsic and extrinsic calibration parameters associated therewith; a storage device including the intrinsic and extrinsic calibration parameters; and a processing device coupled to the tracking camera and the storage device that is configured to: perform ball detections to detect pixel locations in the image plane of the sports ball in each series of images; generate a first time series of ball detections in the image plane; determine, for each sports ball detection in the first time series based on the intrinsic and extrinsic calibration parameters, a camera-ball line that includes a straight line passing through the camera in a direction of the sports ball in a 3D coordinate system; generate a second time series of elevation angles for each camera-ball line; and identify a bounce of the sports ball captured in the series of images based on a minimum value of the second time series.

[0345] The storage device further stores a three-dimensional (3D) model of at least a portion of the sports playing area overlapping with the first field of view, and the processing device is further configured to: determine, for each identified bounce, an intersection point of the 3D model with the camera-ball line based on the 3D model; and output the intersection point as a 3D position of the bounce of the sports ball. The system further includes a tracking radar having a second field of view at least partially overlapping with the first field of view configured to capture radar data of the sports ball, and the processing device is further configured to identify the bounce dependent on the radar data by determining a velocity discontinuity in the radar data. The processing device is further configured to: distinguish between bounces and rolls in the second time series based on a physical model of the bounce; and classify each minimum in the second time series not determined as a bounce as a roll. The 3D position of the sports ball in the image is output to a database, a 3D graphics rendering engine, or a graphical representation of the 3D position of the sports ball.

[0346] The disclosure also relates to a method that includes: performing ball detection to detect a pixel location in an image plane of a sports ball in each of a series of images captured by a tracking camera, the tracking camera having a first field of view configured to capture images in the image plane of a camera coordinate system of the sports ball bouncing and rolling after an initial trajectory, the tracking camera having intrinsic and extrinsic calibration parameters associated therewith stored in a memory; generating a first time series of ball detections in the image plane; determining, for each ball detection in the first time series based on the intrinsic and extrinsic calibration parameters, a camera-ball line that includes a straight line passing through the camera in the direction of the sports ball in a 3D coordinate system; generating a second time series of elevation angles for each camera-ball line; and identifying a bounce of the sports ball captured in the series of images based on a minimum of the second time series.

[0347] The disclosure also relates to a processor coupled to a tracking camera and storage device configured to perform operations including: performing ball detection to detect a pixel location in an image plane of a sports ball in each series of images captured by the tracking camera, the tracking camera having a first field of view configured to capture images of the sports ball bouncing and rolling in the image plane of a camera coordinate system after an initial trajectory, the tracking camera having intrinsic and extrinsic calibration parameters associated therewith stored in storage device; generating a first time series of ball detections in the image plane; determining, for each ball detection in the first time series based on the intrinsic and extrinsic calibration parameters, a camera-ball line that includes a straight line passing through the camera in the direction of the sports ball in a 3D coordinate system; generating a second time series of elevation angles for each camera-ball line; and identifying the bounce of the sports ball captured in the series of images based on a minimum of the second time series.

[0348] The present disclosure also relates to a system that includes: a first sensor having a sensor field of view configured to capture data corresponding to a trajectory of a sports ball; a first broadcast camera having a first field of view configured to capture a first video stream including a first image of a first portion of the trajectory of the sports ball; a second broadcast camera having a second field of view configured to capture a second video stream including a second image of a second portion of the trajectory of the sports ball; and a processing device coupled to the first sensor and the first and second broadcast cameras configured to: generate a broadcast video feed using the first video stream including the first image of the first portion of the sports ball trajectory as the sports ball trajectory crosses the first field of view; determine trajectory parameters based on the data of the sports ball trajectory as the sports ball crosses the sensor field of view; detect a sports ball event based on the trajectory parameters; and switch the broadcast video feed to use the second video stream depending on the detected event such that the second broadcast camera captures a second image of the second portion of the sports ball trajectory as the sports ball crosses the second field of view.

[0349] The event includes parameters of the position criteria indicating that the sports ball has entered or will enter the second field of view. The position criteria include the sports ball moving more than a first distance from the launch position or the launched ball moving more than a second distance from the reference line. The event includes the launch of the sports ball. The second broadcast camera includes a robotic camera, and an operational state of the second camera is adjusted from a first state associated with the first camera parameters to a second state associated with the second camera parameters. The first and second camera parameters include crop, orientation, and zoom levels for the robotic camera in the first and second states, respectively. The processing unit is further configured to provide a region of interest to the second broadcast camera in an area where the sports ball is likely to be found. The second field of view at least partially overlaps with the first field of view.

[0350] The second field of view does not overlap with the first field of view. The broadcast video feed is switched to use the second video stream after a predetermined delay following detection of the event. The first sensor and the first broadcast camera are co-located on the same tracking unit. The first sensor and the first broadcast camera are the same device including a tracking camera additionally used for broadcast. The first sensor, the first broadcast camera and the second broadcast camera are calibrated to each other such that a tracer mapping the trajectory of the sports ball in the broadcast video feed can be applied to the first video stream and the second video stream.

[0351] The disclosure also relates to a method that includes: generating a broadcast video feed using a first video stream including a first image of a first portion of a trajectory of a sports ball, the first video stream captured by a first broadcast camera having a first field of view, the first video stream including a first image of the first portion of the trajectory of the sports ball as the sports ball traverses the first field of view; determining trajectory parameters based on sports ball trajectory data captured by a first sensor having a sensor field of view as the sports ball traverses a sensor field of view; detecting a sports ball event based on the trajectory parameters; and switching the broadcast video feed to use a second video stream including a second image of a second portion of the trajectory of the sports ball, the second video stream captured by a second broadcast camera having a second field of view, the broadcast video feed being switched depending on the detected event such that the second broadcast camera captures a second image of the second portion of the sports ball trajectory as the sports ball traverses the second field of view.

[0352] The present disclosure also relates to a processor coupled to a first sensor and a first and second broadcast camera configured to perform operations including: generating a broadcast video feed using a first video stream including a first image of a first portion of a trajectory of a sports ball, the first video stream being captured by a first broadcast camera having a first field of view, the first video stream including a first image of the first portion of the trajectory of the sports ball as the sports ball traverses the first field of view; determining trajectory parameters based on sports ball trajectory data captured by a first sensor having a sensor field of view as the sports ball traverses the sensor field of view; detecting a sports ball event based on the trajectory parameters; and switching the broadcast video feed to use a second video stream including a second image of a second portion of the trajectory of the sports ball, the second video stream being captured by a second broadcast camera having a second field of view, the broadcast video feed being switched depending on the detected event such that the second broadcast camera captures a second image of the second portion of the sports ball trajectory as the sports ball traverses the second field of view.

[0353] The present disclosure also relates to a system that includes: a first sensor having a sensor field of view configured to capture data corresponding to a trajectory of a sports ball and a characteristic of a player; a first broadcast camera configured to capture a first video stream including a first image; a second broadcast camera configured to capture a second video stream including a second image; and a processing device coupled to the first sensor and the first and second broadcast cameras configured to: receive criteria for a custom broadcast video feed, the criteria related to prioritization of players or locations represented in the custom broadcast video feed; generate a first portion of the custom broadcast video feed using the first broadcast camera; detect an event related to a preferred player or location, the event based on a trajectory parameter of a movement or location of the preferred player or one eye of a sports ball associated with the preferred player; and switch the custom broadcast video feed to use the second video stream depending on the detected event, such that the custom broadcast video feed includes a second image of the preferred player or the trajectory of a first sports ball associated with the preferred player.

[0354] The event includes detection and identification of a priority player in a second image of the second broadcast camera. The event further includes a position or orientation of the priority player detected in the second image. The position or orientation of the priority player includes the priority player approaching the sports ball, standing on the sports ball, swinging a club, or hitting the sports ball. The neural network is used to detect the event. The event includes parameters of a position criterion indicating that a sports ball launched by the priority player may be captured in the second image of the second broadcast camera. The position criterion includes the sports ball moving more than a first distance from a launch position, or the sports ball moving more than a second distance from a reference line, or the event includes a launch of the sports ball. The second broadcast camera includes a robotic camera, and an operating state of the second camera is adjusted from a first state using the first camera parameters to a second state using the second camera parameters.

[0355] The first and second camera parameters include crop, orientation, and zoom level for the robotic camera. The robotic camera is calibrated to the first sensor. The first sensor, the first broadcast camera, and the second broadcast camera are calibrated to each other such that a tracer that maps the trajectory of the sports ball in the custom broadcast video feed can be applied to the first video stream and the second video stream. Criteria for the custom broadcast video feed include: all shots of a preferred player are displayed in the custom broadcast video feed; all shots of all players from a particular country / region are displayed in the custom broadcast video feed; or a particular player demographic is identified and prioritized in determining which shots to display in the custom broadcast video feed. When multiple preferred players are determined to be taking shots at the same time, the video stream of one of the preferred players is buffered for broadcast after the video stream of another one of the preferred players.

[0356] The disclosure also relates to a method that includes: receiving criteria for a custom broadcast video feed, criteria regarding prioritization of players or locations to be displayed in the custom broadcast video feed; generating a first portion of the custom broadcast video feed using a first broadcast camera configured to capture a first video stream including a first image; detecting an event related to a preferred player or position, the event based on a movement or position of the preferred player or a trajectory parameter of a first sports ball associated with the preferred player, the event being detected by a first sensor having a sensor field of view configured to capture data corresponding to a sports ball trajectory or player characteristic; and switching the custom broadcast video feed to use a second video stream including a second image captured by a second broadcast camera, the custom broadcast video feed being switched depending on the detected event, such that the custom broadcast video feed includes a second image of the preferred player or the trajectory of the first sports ball associated with the preferred player.

[0357] The present disclosure also relates to a processor coupled to a first sensor and a first and second broadcast camera configured to perform the following operations: receiving criteria for a custom broadcast video feed, the criteria related to prioritizing players or locations to be represented in the custom broadcast video feed; generating a first portion of the custom broadcast video feed using the first broadcast camera configured to capture a first video stream including a first image; detecting an event related to a preferred player or position, the event based on a movement or position of the preferred player or a trajectory parameter of a first sports ball associated with the preferred player, the event being detected by a first sensor having a sensor field of view configured to capture data corresponding to a sports ball trajectory or player characteristic; and switching the custom broadcast video feed to use a second video stream including a second image captured by a second broadcast camera, the custom broadcast video feed being switched depending on the detected event, such that the custom broadcast video feed includes a second image of the preferred player or a trajectory of a first sports ball associated with the preferred player.

[0358] The present disclosure also relates to a system that includes: a first sensor having a first field of view configured to capture first data corresponding to a first portion of a trajectory of the sports ball including the launch of the sports ball; a second sensor having a second field of view configured to capture second data corresponding to a second portion of the trajectory of the sports ball; and a processing device coupled to the first and second sensors configured to: determine a three-dimensional (3D) position track of the sports ball based on first data of the first portion of the sports ball trajectory as the sports ball traverses the first field of view; detect an event of the sports ball based on one or more parameters obtained from the 3D position track; and adjust an operational state of the second sensor depending on the detected event such that the second sensor captures second data of the second portion of the sports ball trajectory as the sports ball traverses a second field of view and the processing device determines the 3D position track of the second portion of the sports ball trajectory.

[0359] The event includes parameters related to the position of the sports ball relative to the second field of view. An operational state of the second sensor is adjusted to capture second data when the sports ball is determined to have traveled more than a first distance from the launch location or when the sports ball is determined to have traveled a second distance from the baseline. An operational state of the second sensor is adjusted from a low power state to a full power state. An operational state of the second sensor is adjusted from a non-tracking state to a tracking state. The second sensor includes a robotic camera, and an operational state of the second sensor is adjusted from a first state using first camera parameters to a second tracking state using second camera parameters. The first and second camera parameters include crop, orientation, and zoom level for the robotic camera. An operational state of the second sensor is adjusted from a first state in which bounce and roll are not tracked using a dedicated bounce and roll tracking module to a second state in which bounce and roll are tracked using a dedicated bounce and roll tracking module.

[0360] The processing device is further configured to provide a region of interest to a second sensor that specifies an area where the sports ball is likely to be found. The second field of view at least partially overlaps with the first field of view. The second field of view does not overlap with the first field of view. The processing device determines a 3D position track for the first and second portions of the sports ball trajectory in a world coordinate system. The first and second sensors are 3D Doppler radar. The first and second sensors are calibrated to the world coordinate system based on GPS locations of the reference points and GPS locations of the first and second sensors. The first sensor is the 3D Doppler radar and the second sensor is a tracking camera. The second sensor is calibrated to the world coordinate system based on the GPS locations of the reference points, and the processing device is further configured to identify one or more reference points in the image and determine an orientation of the second sensor based thereon. The first sensor and the second sensor are co-located, and the orientation of the second sensor determined from the feature matching of the image and the orientation of the first sensor are determined based on the orientation of the second sensor.

[0361] The disclosure also relates to a method that includes: determining a three-dimensional (3D) position track of a sports ball based on first data of a first portion of the sports ball's trajectory as the sports ball traverses a first field of view, including the launch of the sports ball, from a first sensor having a first field of view configured to capture first data corresponding to a first portion of the sports ball's trajectory as the sports ball traverses a first field of view, detecting a sports ball event based on one or more parameters obtained from the 3D position track; and adjusting an operating state of a second sensor depending on the detected event, the second sensor having a second field of view configured to capture second data corresponding to a second portion of the sports ball's trajectory as the sports ball traverses a second field of view; and determining the 3D position track for the second portion of the sports ball trajectory.

[0362] The disclosure also relates to a processor coupled to first and second sensors configured to perform operations including: determining a three-dimensional (3D) position track of the sports ball based on first data of a first portion of the sports ball's trajectory as the sports ball traverses the first field of view, from a first sensor having a first field of view configured to capture first data corresponding to a first portion of the sports ball's trajectory including the launch of the sports ball; detecting an event of the sports ball based on one or more parameters obtained from the 3D position track; and adjusting an operating state of a second sensor dependent on the detected event, the second sensor having a second field of view configured to capture second data corresponding to a second portion of the sports ball's trajectory as the sports ball traverses the second field of view; and determining the 3D position track for the second portion of the sports ball trajectory.

[0363] The present disclosure also relates to a system that includes: a tracking camera having a first field of view configured to capture an image of a golf ball located on a green in a camera coordinate system, the tracking camera having intrinsic and extrinsic calibration parameters associated therewith; a storage device including the intrinsic and extrinsic calibration parameters and a three-dimensional (3D) model of at least a portion of a golf course overlapping the first field of view; and a processing device coupled to the tracking camera and the storage device that is configured to: perform ball detection to detect a pixel location within an image plane of a stationary golf ball in the image; determine a camera-ball line including a straight line passing through the camera, an orientation of the golf ball in the 3D coordinate system based on the intrinsic and extrinsic calibration parameters, the camera-ball line including a straight line passing through the camera, the orientation of the golf ball in the 3D coordinate system based on the 3D model; determine an intersection point of the camera-ball line with the 3D model based on the 3D model; determine a 3D position of the stationary golf ball on the green; simulate a series of putt trajectories based on various launch speeds and launch directions using knowledge of the stimp and the 3D model of the green; and generate a putt break fan, which is a combined plot of the launch directions and launch speeds of the putts and corresponding outcomes of the putts, based on the series of simulated putt trajectories.

[0364] The putt break fan includes a portion of the plot where a combination of launch direction and launch speed results in a successful putt entering the hole, a first contour line where a combination of launch direction and launch speed results in a stationary ball position having a first distance from the hole, and a second contour line where a combination of launch direction and launch speed results in a stationary ball position having a second distance from the hole. A stimp of the green is determined from tracking a previous putt and determining a velocity decay of the previous putt. The previous putt is tracked within a predetermined time prior to the current putt. The processor is further configured to overlay the putt break fan onto the broadcast video. The processor is further configured to generate an output indicative of a probability of a successful putt in response to a given launch direction and a given launch speed.

[0365] The disclosure also relates to a method that includes: performing ball detection to detect a pixel location of a stationary golf ball in an image plane captured by the tracking camera, the tracking camera having a first field of view configured to capture images of a golf ball located on a green in a camera coordinate system, the tracking camera having intrinsic and extrinsic calibration parameters stored in a memory device associated therewith; determining a camera-ball line, including a straight line passing through the camera, an orientation of the golf ball in a 3D coordinate system based on the intrinsic and extrinsic calibration parameters; determining an intersection point of the camera-ball line with a 3D model based on a three-dimensional (3D) model stored in a memory device of at least a portion of a golf course overlapping with the first field of view; determining a 3D location of the stationary golf ball on the green; simulating a series of putt trajectories based on various launch speeds and launch directions using knowledge of the stimp and the 3D model of the green; and generating a putt break fan, which is a combined plot of the launch directions and launch speeds of the putts and corresponding outcomes of the putts, based on the series of simulated putt trajectories.

[0366] The present disclosure also relates to a processor coupled to a tracking camera and storage device configured to perform operations including: performing ball detection to detect a pixel location of a stationary golf ball in an image plane captured by the tracking camera, the tracking camera having a first field of view configured to capture images of a golf ball located on a green within a camera coordinate system, the tracking camera having intrinsic and extrinsic calibration parameters stored in storage device associated therewith; determining a camera-ball line including a straight line passing through the camera, an orientation of the golf ball in a 3D coordinate system based on the intrinsic and extrinsic calibration parameters; determining an intersection point of the camera-ball line with a 3D model based on a three-dimensional (3D) model stored in storage device of at least a portion of a golf course overlapping with the first field of view; determining a 3D location of the stationary golf ball on the green; simulating a series of putt trajectories based on various launch speeds and launch directions using knowledge of the stimp and the 3D model of the green; and generating a putt break fan, which is a combined plot of the launch directions and launch speeds of the putts and corresponding outcomes of the putts, based on the series of simulated putt trajectories.

[0367] It will be apparent to those skilled in the art that various modifications can be made in the present disclosure without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure is intended to cover the modifications and variations of the present disclosure provided they come within the scope of the appended claims and their equivalents.

Claims

1. 1. A system comprising: a robotic camera configured to capture data corresponding to a position of a sports ball, the robotic camera further configured to automatically adjust an orientation and zoom level of the robotic camera in response to the captured data or commands; and a processing device coupled to the robotic camera, pre-calibrating the robotic camera such that an initial orientation of the robotic camera is known in a world coordinate system, and associating each of a plurality of different zoom levels used by the robotic camera with a respective intrinsic parameter value; Detecting a first image location of the sports ball in an image coordinate system of a first image, the first image being captured by the robotic camera at a first orientation; reading from the robotic camera a first zoom level associated with the first image and an intrinsic parameter value associated with the first zoom level; determining the first orientation of the robotic camera based on a pan and tilt of the robotic camera relative to the initial orientation; determining a three-dimensional line passing through the robotic camera and the sports ball in the world coordinate system based on the intrinsic parameter values ​​of the detected first image position of the sports ball, the determined first orientation, and the first zoom level; and A system comprising: a processing device configured to determine a three-dimensional position of the sports ball in the world coordinate system that lies along the three-dimensional line based on extrinsic information to the robotic camera.

2. 2. The system of claim 1, wherein at a first location in the image, the sports ball is on a surface of a sports playing area, and extrinsic information for the robotic camera includes information received from a three-dimensional model of at least a portion of the sports playing area that encompasses the three-dimensional position of the sports ball, and the processing device determines the three-dimensional position of the sports ball by locating an intersection of the line with the three-dimensional model.

3. a pan and tilt sensor fixed to the robotic camera; The system of claim 1 , wherein the processing unit is further configured to read the pan and tilt of the robotic camera relative to the initial orientation from the pan and tilt sensors.

4. 2. The system of claim 1, wherein the pan and tilt of the robotic camera are determined from a control signal used to adjust the robotic camera to the first orientation, the control signal being generated in response to tracking data of the sports ball captured prior to capture of the first image.

5. The system of claim 1 , wherein the robotic camera is further configured to automatically adjust crop in response to the captured data or an external command.

6. The system of claim 1 , wherein the intrinsic parameters include focal length, principal point, and lens distortion.

7. The system of claim 1 , wherein the extrinsic information for the robotic camera includes information from a sensor regarding a distance of the sports ball from the robotic camera.

8. 1. A method comprising: pre-calibrating a robotic camera such that an initial orientation of the robotic camera is known within a world coordinate system, and associating each of a plurality of zoom levels used by the robotic camera with a respective intrinsic parameter value, the robotic camera being configured to capture data corresponding to a position of a sports ball, the robotic camera being further configured to automatically adjust orientation and zoom level in response to the captured data or commands; detecting a first image location of the sports ball in an image coordinate system of a first image, the first image being captured by the robotic camera at a first orientation; reading a first zoom level from the robotic camera and the associated intrinsic parameter value for the first zoom level of the robotic camera corresponding to the first image; determining the first orientation of the robotic camera based on a pan and tilt of the robotic camera relative to the initial orientation; determining a three-dimensional line passing through the robotic camera and the sports ball in the world coordinate system based on the intrinsic parameter values ​​of the detected first image position of the sports ball, the determined first orientation, and the first zoom level; and determining a three-dimensional position of the sports ball in the world coordinate system that lies along the three-dimensional line based on extrinsic information to the robotic camera.

9. a processor coupled to the robotic camera, pre-calibrating the robotic camera such that an initial orientation of the robotic camera is known within a world coordinate system and associating each different zoom level used by the robotic camera with a respective intrinsic parameter value, the robotic camera being configured to capture data corresponding to a position of a sports ball, and the robotic camera being further configured to automatically adjust its orientation and zoom level in response to the captured data or commands; detecting a first image location of the sports ball in an image coordinate system of a first image, the first image being captured by the robotic camera at a first orientation; reading a first zoom level from the robotic camera and the associated intrinsic parameter value for the first zoom level corresponding to the first image; determining the first orientation of the robotic camera based on a pan and tilt of the robotic camera relative to the initial orientation; determining a three-dimensional line passing through the robotic camera and the sports ball in the world coordinate system based on the intrinsic parameter values ​​of the detected first image position of the sports ball, the determined first orientation, and the first zoom level; and determining a three-dimensional position of the sports ball in the world coordinate system that lies along the three-dimensional line based on extrinsic information to the robotic camera.

Images