System and method for calibrating robotic cameras

Abstract

The processor and camera capture data corresponding to the position of the ball and automatically adjust the camera's orientation and zoom level. The processor pre-calibrates the camera so that an initial orientation is known in a world coordinate system (WCS) and different zoom levels are associated with respective parameter values. A first position of the ball in a first image is detected, and a first zoom level of the first image and intrinsic parameter values ​​for the first zoom level are retrieved. A first orientation of the camera is determined based on the pan and tilt relative to the initial orientation. A 3D line through the camera and the ball in the WCS is determined based on the first position, and a first orientation and parameter values ​​for the first zoom level and a 3D position are determined in the WCS along the line based on extrinsic information for the camera.

Description

[Technical Field]

[0001] Priority Claim This application claims priority to U.S. Provisional Patent Application No. 63 / 200,425, filed on 5 March 2021, and U.S. Provisional Patent Application No. 63 / 202,850, filed on 27 June 2021. The specifications of the above-mentioned applications are incorporated herein by reference.

[0002] This disclosure relates to a system and method for tracking objects, including golf balls and golf players, during a golf event and for determining statistics for use in broadcasting the golf event. [Background technology]

[0003] Detailed statistics on golf shots can be captured during amateur play or professional tournaments such as PGA Tour events. For example, radar-based tracking systems can be used to capture tee shot data during ball flight. However, many systems currently in use cannot determine more advanced shot measurement criteria, such as the ball's final resting position after bounce and roll. In professional tournaments, these decisions are currently made manually by human operators, for example, using laser range finders. This is an extremely laborious task, typically requiring hundreds of operators. In addition to generating specific shot data, many operations performed during the successful and visually engaging broadcast of a golf event require extensive manual input from human operators. [Overview of the project]

[0004] This disclosure relates to a system, method, and processing device for calibrating a robotic camera. The system comprises a robotic camera configured to capture data corresponding to the position of a sports ball, and further configured to automatically adjust the orientation and zoom level of the robotic camera in response to the captured data or commands; and a processing unit connected to the robotic camera configured as follows: pre-calibrate the robotic camera so that its initial orientation is known in the world coordinate system, and associate each of several different zoom levels used by the robotic camera with its respective intrinsic parameter value; detect a first image position of the sports ball in the image coordinate system of a first image, and the first image is captured by the robotic camera in a first orientation; read a first zoom level associated with the first image and the intrinsic parameter value associated with the first zoom level from the robotic camera; determine a first orientation of the robotic camera based on the pan and tilt of the robotic camera relative to its initial orientation; determine a three-dimensional line passing through the robotic camera and the sports ball in the world coordinate system based on the detected first image position of the sports ball, the determined first orientation, and the intrinsic parameter value of the first zoom level; and determine the three-dimensional position of the sports ball in the world coordinate system located along the three-dimensional line based on extrinsic information to the robotic camera. [Brief explanation of the drawing]

[0005] [Figure 1] Figure 1 shows exemplary layouts 100–115 for four exemplary hole tracking units.

[0006] [Figure 2] Figure 2 shows an exemplary diagram 200 for an exemplary golf hole 250 tracking system according to various exemplary embodiments.

[0007] [Figure 3]Figure 3 shows an exemplary chart 300 for a golf course tracking system according to various exemplary embodiments.

[0008] [Figure 4] Figure 4 shows an exemplary tracking unit 400 that includes a radar system 405 and / or a camera system 410 and / or a lidar system 415 according to various exemplary embodiments.

[0009] [Figure 5a] Figure 5a shows an example of the first principle of MFCW Doppler radar tracking.

[0010] [Figure 5b] Figure 5b shows an example of the second principle of MFCW Doppler radar tracking.

[0011] [Figure 5c] Figure 5c shows an example of the third principle of MFCW Doppler radar tracking.

[0012] [Figure 6] Figure 6 shows a chart 600 of exemplary data packets 605 - 630 that include specific data of a golf shot transmitted to a broadcast entity and / or other applications during the acquisition of real-time data of a golf shot according to various exemplary embodiments.

[0013] [Figure 7] Figure 7 shows an exemplary image 700 of a club head from a position behind the player.

[0014] [Figure 8a] Figure 8a shows a first image 800 from a video feed depicting a shot that includes an overlay of shot trajectories, i.e., a first tracer 805 of the shot, and some associated trajectory data.

[0015] [Figure 8b]Figure 8b shows a second image 850 from a video feed depicting the shot, including a superimposition of the shot trajectories, i.e., a second tracer 855 of the shot, and some associated trajectory data.

[0016] [Figure 9] Figure 9 shows exemplary images 900–915, which can be analyzed by a neural network to detect events related to the player's swing.

[0017] [Figure 10a] Figure 10a shows an exemplary image 1000 captured by the robot camera 1005 to determine the orientation of the robot camera 1005 in world coordinates 1020.

[0018] [Figure 10b] Figure 10b shows the process of determining the orientation of the robot camera 1005 based on the first image 1000 in Figure 10a and the second image 1030 captured by the robot camera 1005 using a different orientation.

[0019] [Figure 10c] Figure 10c shows the process for determining the orientation of the robot camera 1005 based on the first image 1000 in Figure 10a and the second image 1040 captured by the second camera 1035.

[0020] [Figure 10d] Figure 10d shows the process for determining the orientation of the robot camera 1005 based on the first image 1000 of Figure 10a and the second image 1030, which includes a zoomed-out version of the first image 1000.

[0021] [Figure 11]Figure 11 shows an exemplary tracking system 1100, which includes at least one camera 1105 that captures a series of images of a ball, including a stationary ball, according to various exemplary embodiments, and a processing device for tracking the ball and / or determining the stationary ball's position.

[0022] [Figure 12] Figure 12 shows an exemplary figure 1200, which includes a camera 1105, a golf course terrain 1205, and a corresponding 3D model 1135 superimposed on the golf course terrain 1205, according to various exemplary embodiments.

[0023] [Figure 13] Figure 13 shows a time series 1300 of ball detection in the (u,v) image plane from a series of images and a corresponding time series 1350 of the elevation angle relative to the camera-ball line determined by the non-projector 1115 described in Figure 11.

[0024] [Figure 14] Figure 14 shows plot 1400, which includes the starting position 1405, the hole position 1410, and lines 1415-1425 representing the putt trajectory that could result in a successful putt.

[0025] [Figure 15] Figure 15 shows a putt break fan 1500 as a chart 1510 of the combinations of launch direction and launch speed for a successful putt, and as contour lines 1515, 1520 showing the combinations of launch direction and launch speed that result in a ball position equal to the distance to the pin for the second putt.

[0026] [Figure 16a] Figure 16a shows an exemplary method 1600 for tracking the three-dimensional position of a ball launched during play on a golf course hole, the tracking being performed using data from both a first sensor and a second sensor, the operation of the second sensor being controlled based on 3D position tracking information determined from the data captured by the first sensor.

[0027] [Figure 16b] Figure 16b shows a method 1610 for transmitting tracking data to be included in a broadcast, according to various exemplary embodiments.

[0028] [Figure 16c] Figure 16c illustrates a method 1630 for identifying a player and associating a specific player profile with shot data associated with a unique player ID, according to various exemplary embodiments.

[0029] [Figure 16d] Figure 16d shows a method 1650 for automatic broadcast feed switching in various exemplary embodiments.

[0030] [Figure 16e] Figure 16e shows Method 1660 for the automatic generation of custom (e.g., personalized) broadcast feeds by various exemplary embodiments.

[0031] [Figure 16f] Figure 16f shows a method 1670 for the automatic insertion of tracers into a broadcast feed by various exemplary embodiments.

[0032] [Figure 16g] Figure 16g shows a method 1680 for calibrating a robotic camera to a world coordinate system according to various exemplary embodiments.

[0033] [Figure 16h] Figure 16h shows Method 1690 for a warning system for on-site spectators by various exemplary embodiments.

[0034] [Figure 17a] Figure 17a shows a method 1700 for determining the 3D position coordinates of a ball according to various exemplary embodiments.

[0035] [Figure 17b] Figure 17b shows a method 1720 for determining the bounce and roll of a moving ball based on image data, according to various exemplary embodiments.

[0036] [Figure 17c] Figure 17c shows a method 1740 for determining information about the next pad, according to various exemplary embodiments.

[0037] [Figure 17d] Figure 17d shows a method 1750 for determining the terrain parameters of a putting green based on tracked putts, according to various exemplary embodiments. [Modes for carrying out the invention]

[0038] Exemplary embodiments may be further understood by referring to the following description and associated accompanying drawings, where similar elements are given the same reference numerals. The exemplary embodiments relate to systems and methods for identifying and tracking objects on a golf course, including golf shots, golf clubs, and golf players. Multiple techniques relating to different aspects of object tracking are described, which, when used in combination, can provide complete coverage of relevant moving objects on a golf course. According to some embodiments, data generated according to the described techniques can be used to enrich and enhance coverage of live broadcasts of golf events, such as PGA Tour events. In particular, almost all of the described techniques are performed automatically and require little to no manual input from operators on the golf course.

[0039] In a typical round of golf, the statistics recorded may only include the number of shots a player took on each hole. This stroke counting is usually done manually. In some professional tournaments, including those hosted by the PGA Tour, more detailed statistics are recorded, such as the ball's position and lie for each shot. In recent years, ball trajectory data from radar-based tracking systems such as the TrackMan system has also been used on the PGA Tour to capture tee shot data. These systems generally measure the three-dimensional position of the ball in flight until it hits the ground. However, the final stationary position is generally determined manually by an operator using a surveyor-type laser rangefinder. Determining the final stationary position is highly labor-intensive, requiring hundreds of operators to track every shot throughout all rounds of a typical four-day PGA Tour tournament. In some scenarios, volunteers are also tasked with pressing buttons on handheld devices to capture the time of shots taken during the tournament.

[0040] The embodiments described herein enrich datasets, enhance captured data, and automate many tasks currently performed manually, compared to existing methods for capturing data during professional tournaments. These solutions may automate virtually all operations during professional tournaments, be applicable to any golf course, and make the same or similar data available to amateur golfers playing on local golf courses.

[0041] Embodiments of this disclosure perform fully automatic or semi-automatic tracking of all shots on a golf course and assign each golf shot to a unique player ID. As a result, all shots from the same player are tagged with the same player ID unique to that player. In addition to flight tracking, the tracking system collects various key tags for each shot, including the bounce and roll of each golf shot from launch to rest, the number of the golf hole played or other identifier, data on the lie of the ball, shot number (e.g., the number of shots taken by the player up to the current shot, including the current shot) on this hole, the time of impact when the shot was struck, the type of club used for the current shot, and environmental conditions such as wind and temperature.

[0042] The system described herein can automatically record all shots taken by all players during a round of golf. Those skilled in the art will understand that certain systems may also be subject to manual input when desired. For example, in tournament play, a camera may be moved to the fairway behind a player preparing for their current shot (e.g., the camera is further from the hole than the ball the player is preparing to hit). This camera may be manually or automatically calibrated and positioned, as will be described in more detail below, and tracking information may be inserted into or derived from images from the camera. Furthermore, such a manually positioned camera may also include a moving unit that provides further tracking information to the system. As will be understood by those skilled in the art, manual repositioning or aiming of the camera (and any attached radar or other tracking unit) can be seamlessly integrated with automatic tracking and image manipulation, as desired, to create a semi-automatic tracking system.

[0043] Recording all shots during a round of golf provides a complete and compelling dataset for players and / or third parties. However, it should be understood that this solution can also be used for only a portion of a round, such as a few complete holes or selected tee shots, as desired. Furthermore, the recording and / or distribution of this data may be performed only for a selected group of golfers or for a single golfer, and may be offered, for example, as an additional service (e.g., at an additional cost). As detailed below, a player identification system can be used to track individual players and associate data related to those players, such as shot data captured during play, with each individual player. Statistics of golf balls and / or golf clubs captured during play can be associated with a specific golfer who has a unique player ID. Captured statistics may include any information related to the golf match, such as: carry position, final resting position, trajectory including bounce and roll, ball launch data (velocity, angle, direction, spin rate and spin axis), number of shots on the hole, hole number, lie of the ball for each shot, impact time, club used, swing data, etc. Of course, data from such systems can be aggregated and used to evaluate different ball and / or club types under various conditions and / or using the anatomical structure or swing type of different players. Such aggregated data can also be used to evaluate course design (or redesign) and / or to select tee and pin placements.

[0044] In some embodiments, a full-course tracking system can be used during professional golf tournaments to enhance event broadcasts by providing real-time data that can be overlaid on a broadcast feed, such as shot tracers, advanced shot measurement criteria, and landing predictions. This information can also be made available to players for use in improving their game or to evaluate the difference in performance between practice and tournament conditions. However, the same system can also be used during any amateur golf round, whether during a golf tournament, practice round, or any other time. During amateur and practice rounds, golfer groupings and play schedules may or may not be available to the system, for example, through a tee reservation system or similar. In scenarios where the tracking system is available to golfers during unscheduled tee times, the system may include several mechanisms for obtaining consent from golfers before tracking their play. This consent may be given before the round, or possibly after the round.

[0045] The exemplary embodiments described below relate to concepts including: a tracking system for full-course tracking (tracking all or nearly all shots taken during an entire round or multi-round tournament of golf); tracking unit devices available for use in the tracking system, including specifications and operations for radar-based, camera-based, lidar-based, and / or other sensors capable of identifying and / or tracking golf shots and / or golf clubs; tracking unit devices and operations specifically configured to track the bounce and roll of golf shots; calibration schemes for tracking units and / or other sensors; broadcast systems for broadcasting video feeds of golf events, capable of working in conjunction with the tracking system, including calibration of broadcast cameras with the tracking system sensors; player identification systems; automated broadcast feeds that can be tailored to specific players; automated tracers; automated detection of player events, including when a player is about to hit a shot; calibration and control of robotic cameras based on real-time tracking data provided by the tracking system; safety / warning systems for on-site spectators; schemes for amateur round tracking; and generation of non-fungible tokens (NFTs). Throughout the description, a power-saving scheme will be described in which, for example, tracking units and / or sensors are triggered to power on based on tracking data provided by different tracking units, for example, suggesting that an object will come into the field of view of a tracking unit that is powered off. In some preferred embodiments, a world coordinate system and a common time reference used for time synchronization are used by the tracking system across the golf course to enable, for example, rapid transfer of data from one sensor to the local coordinate system of another sensor.

[0046] System Overview This disclosure relates to a tracking system comprising a plurality of tracking units distributed throughout a golf course. The placement of the tracking units may depend on several factors, including, for example, the layout of the hole, the capabilities of the tracking units, the desired tracking coverage of the hole, or other considerations described in detail below. Where used herein, the term “hole” should be understood to refer to the entire golf hole, including, for example, the tee box, fairway, rough, green, hazard, bunker, etc., according to general golf terminology. The actual hole located on the green, where the golf ball is intended to eventually sink and be marked with a flag, may be referred to herein as a “cup,” although general golf terminology also refers to this function as a “hole.” Those skilled in the art will understand the distinction between a broader “hole” encompassing the entire golf hole and a narrower “hole” encompassing only the cup located on the green and marked with a flag.

[0047] Figure 1 shows exemplary layouts 100–115 for tracking units in four exemplary holes. As shown in Figure 1, the configuration of each system component in a hole depends on the length and layout of the hole. Typically, each hole has a primary tracking unit located near the tee box of the hole, and additional tracking units, distributed at different locations depending on the length and layout of the hole, ensure strong tracking coverage at any location where the golf ball is expected to be located during play on the hole. It may also be advantageous to place tracking units in locations where a high degree of accuracy and / or high-quality tracking is expected to be obtained. Additional tracking units may be desirable to avoid / minimize obstructions by golfers, trees, or other objects that would obstruct the line of sight of the primary tracking unit. Layout 100 shows a par 3 hole where a single tracking unit, in this example, a radar unit (radar 1), is located near the tee box of the hole. The radar unit in this example can provide complete tracking coverage for the entire par 3 hole. Layout 105 shows a par 4 hole with two tracking units, in this example, a first radar unit (Radar 1) and a second radar unit (Radar 2), positioned near the hole's tee box and near the hole's green, respectively. The two radar units in this example can provide complete tracking coverage for the entire par 4 hole. Layout 110 shows a par 5 hole with two tracking units, in this example, a first radar unit (Radar 1) and a second radar unit (Radar 2), positioned near the hole's tee box and near the hole's green, respectively. The two radar units in this example can provide complete tracking coverage for the entire par 5 hole. Layout 115 shows a par 5 hole with three tracking units, in this example, a first radar unit (Radar 1), a second radar unit (Radar 2), and a third radar unit (Radar 3), positioned near the hole's tee box, along the hole's fairway, and near the hole's green, respectively.The three radar units in this example can provide complete tracking coverage for the entire par-5 hole.

[0048] Figure 2 shows an exemplary diagram 200 for a tracking system for an exemplary golf hole 250, according to various exemplary embodiments. The layout of each golf hole typically includes a tee area or tee box 255, a fairway 260, a green 265 that generally surrounds the fairway 260, and the green 265. The green 265 contains a cup, which is indicated by a flag 270, from which the golf ball is intended to end when a player completes the golf hole 250. In this example, the golf hole 250 also includes a water hazard 275 and a bunker 280. The tracking system in Diagram 200 includes a first tracking unit 205 located near the tee box 255 and a second tracking unit 210 located behind the green 265. As described above, for a particular hole, one or more additional tracking units may be distributed at different locations surrounding the hole. The tracking units 205, 210 provide position and / or motion data of the golf ball 215 and / or the golf club being used, which will be described in more detail below. All positional data may ultimately be determined in a global coordinate system 220, for example, used to generate tracks across the entire golf course, but some sensors may determine some positional data in coordinate systems specific to the hole and / or the sensor's local coordinate system, which are described in more detail below.

[0049] Figure 200 further illustrates an exemplary path of a ball 215 during play on hole 250. The ball 215 starts at a first resting position 215a on the tee box 255 and is struck along trajectory 215b to a second resting position 215c on the fairway 260. The ball 215 is then struck along trajectory 215d to a third resting position 215e on the green 265. From there, the ball 215 can be putted into the cup marked by the flag 270. The system according to the exemplary embodiments described herein can accurately recognize the ball in motion (215b, 215d) and / or at rest (215a, 215c, 215e). In some embodiments, the first tracking unit 205 may be a tee box (tee area) tracking unit having the ability to track tee shots (e.g., trajectory 215b launched from resting position 215a) and to derive specific information, e.g., launch parameters, from these tee shots. The second tracking unit 210 may be a green (greenside) tracking unit having a second capability selected for tracking approach shots (e.g., trajectory 215d launched from stationary position 215c and ending at stationary position 215e) and / or chips and putts, as well as for extracting specific information from these shots, such as putt parameters. A third type of tracking unit may comprise a fairway tracking unit having a third capability selected for tracking shots that land on and are launched from the fairway 260, while remaining portable for frequent changes in position. However, multiple different types of tracking units, including various combinations of sensors and capabilities, may be used, as will be described in more detail below.

[0050] Figure 3 shows an exemplary diagram 300 for a golf course tracking system according to various exemplary embodiments. In diagram 300, only three golf holes 350 are shown, for example, the first hole 350a (hole 1), the second hole 350b (hole 2), and the third hole 350c (hole 3). However, it should be understood that the principle described in exemplary diagram 300 can be extended to any number of holes and can encompass an entire golf course including any number of holes, e.g., 9, 18, 27, etc. In diagram 300, three players 355 are shown, for example, the first player 355a is shown on the first hole 350a, the second player 355b is shown on the second hole 350b, and the third player 355c is shown on the third hole 350c. As will be described in more detail below, in addition to tracking golf shots, the tracking system described herein can accurately identify players and associate these players with the golf shots tracked by the system, for example, using player IDs. The exemplary tracking system shown in Figure 3 is added to broadcast video feeds of a golf course, for example, when the golf course is used for professional events such as PGA Tour events. However, it should be understood that the broadcast mode of the tracking system is arbitrary, and the tracking system can be used for various purposes unrelated to the broadcasting of professional events. Specific technologies related to the broadcasting of events, including data transmission delivered to the broadcast (e.g., for player / shot information / tracking), broadcast camera calibration, and automatic feed switching, are described in more detail below.

[0051] Each hole in Figure 3 is equipped with a number of tracking units 305 to provide tracking coverage for the hole and is connected to a tracking server 320 via a wired or wireless connection 310. In this example, a number of broadcast cameras 315 are also positioned on each hole, and are also connected to the tracking server 320 via a wired or wireless connection 310. The first hole 350a includes a first tracking unit 305a and a first broadcast camera 315a adjacent to the tee box, as well as a second tracking unit 305b and a second broadcast camera 315b adjacent to the hole. The second hole 350b includes a third tracking unit 305c and a third broadcast camera 315c adjacent to the tee box, a fourth tracking unit 305d along the fairway, and a fifth tracking unit 305e adjacent to the hole. The third hole 350c includes a sixth tracking unit 305f and a fourth broadcast camera 315d adjacent to the tee box, as well as a seventh tracking unit 305g and a fifth broadcast camera 315e adjacent to the hole. The locations and number of tracking units 305 and broadcast cameras 315 shown in Figure 3 are provided for illustrative purposes only, and it should be understood that any number of tracking units 305 and broadcast cameras 315 may be used depending on the desired tracking coverage / accuracy, the layout of the holes on the course, and / or the desired broadcasting capabilities of the broadcasting entity, e.g., the PGA Tour.

[0052] Additional tracking units, similar to, less capable than, or having different capabilities than the primary tracking unit 305 shown in Figure 3, may be implemented for full-course tracking, subject to various considerations detailed below. For example, one or more mobile tracking units used for fairway tracking may have reduced capabilities compared to tracking units used for tee areas and / or greens. Furthermore, those skilled in the art will understand that some very short shots (e.g., mis-hits in areas obstructed by spectators, trees, etc.) may not be captured by the system. To address the possibility of mis-hits, the system may have a manual correction / additional input function. For example, the system may ask each player to confirm their shot count upon completion of a hole. In such a situation, there is no data on mis-hits, but the score remains accurate.

[0053] The tracking process may be carried out by a processing unit incorporated into each tracking unit 305, by the tracking server 320, or by any of the various steps performed in cooperation between the incorporated processing unit and the tracking server 320. The tracking server 320 may be, for example, an on-site server or a cloud-based processing service. The tracking server 320 includes a processing unit and a storage device and is connected to the tracking units 305, the broadcast camera 315, and optionally other sensors or devices that may be used in the whole-course tracking system and / or broadcast system, as described in more detail below. The processing unit of the tracking server 320 may include one central processor or multiple processors; in some embodiments, the tracking server 320 may process the tracking of tens or hundreds of objects simultaneously, in addition to various additional functions described in more detail below, which requires relatively high computing power. In some embodiments, the tracking server 320 may also function as, or collaborate with, a broadcast system providing video coverage of live events, such as PGA Tour events, for example, by outputting / switching video feeds, audio feeds, etc., to present viewers with a cohesive viewing experience. In other embodiments, the tracking server 320 can provide information to a broadcast entity responsible for the content displayed to viewers. In this scenario, the tracking server 320 and the broadcast entity may work together to enable the insertion of live data into the broadcast in substantially real time, as will be described in more detail below.

[0054] The tracking server 320 is responsible for associating each ball (in motion or stationary) with the individual player 355 playing that ball, and each player on the course is associated with a unique player ID. The tracking server 320 is also responsible for transmitting the associated ball tracking information, along with the player ID of the player who hit the tracked ball, to various consumers. When the tracking system is used for broadcast purposes, the tracking server 320 can transmit the tracking information, along with the player ID, to consumers such as broadcasters 325, web applications 330, and / or field applications 335, in conjunction with broadcast camera data. When the tracking system is used primarily for informational purposes, i.e., not associated with broadcasting, the information may be transmitted to a field database, for example, for later retrieval by individual players, or to individual players, for example, via a mobile application for a handheld device used by a player, or to other stakeholders. A player identification system used to identify players and associate those players with data captured by the tracking system is described in more detail below.

[0055] Tracking unit The exemplary tracking unit 305 shown in Figure 3 may include one or more sensors, e.g., one or more non-image-based sensors (e.g., microphones, radar and / or lidar systems) and one or more cameras in an imaging (e.g., camera-based) system. The tracking unit 305 may include only non-image-based sensors, only image-based sensors, or a combination of non-image-based and image-based sensors. For example, the camera system may include multiple camera sensors having different characteristics, such as field of view, resolution, orientation, frame rate, wavelength sensitivity, electronically controllable zoom (analog or digital), fixed or movable, fixed or electronically adjustable pan-tilt orientation. The camera sensors may include passive light receiving sensors, time-of-flight cameras, event-based cameras, etc., and may also include visual, infrared (IR) or near-infrared (NIR) illumination synchronized with the camera's frame exposure to increase the signal-to-noise ratio (SNR) of the illuminated scene. Similar to the camera system, the non-image-based sensors (e.g., radar and / or lidar) may include multiple sensors having different characteristics, e.g., different field of view, orientation, resolution, frame rate, wavelength, modulation, etc. For the sake of simplicity, the disclosed embodiments refer to the non-image-based components of the tracking unit 305 as the radar unit. However, those skilled in the art will understand that such components may be replaced by, or include, additional non-image-based sensors, as needed.

[0056] The tracking system may include any mixture of different tracking units having different sizes and configurations. Some tracking units may include a combination of radar and camera systems, while others may include only a radar system or only a camera system. For example, the angular accuracy of a radar unit is typically related to the separation between the various antennas, so greater separation may produce higher angular accuracy. In some cases, a larger radar unit may be required to achieve the desired accuracy, while other radar units may be smaller to provide portability or a smaller visual footprint. In one example, a fairway tracking unit may be relatively small and portable so that the operator can change the position and orientation of the tracking unit depending on the various positions of nearby shots, while tee box tracking units and green tracking units may be larger and substantially fixed in terms of the high probability or certainty that these tracking units provide good coverage of each specific shot hit on each hole, such as tee shots and chips / putts. Furthermore, some tracking units may include a robotic camera that can automatically change orientation based, for example, on commands from a tracking server (based on tracking data from different tracking units or sensors) or on tracking data from the same tracking unit. The robotic camera may be a tracking camera that performs optical tracking and changes orientation based on ball detection, or the robotic camera may be a broadcast camera that relies on tracking data acquired from other sensors.

[0057] In addition to orientation, zoom level and crop can be controlled. In some embodiments, additional sensors (e.g., radar or lidar) are included in the robot camera on a single tracking unit, and all sensors included in the tracking unit can change orientation together. As those skilled in the art will understand, a radar unit moving with the robot camera may continue to track a ball within a portion of the radar's field of view (e.g., near its center), for example, due to the fact that such radars often receive stronger reflected signals from objects moving near the center of their field of view, thus increasing the radar's sensitivity.

[0058] Each tracking unit 305, either by itself or in combination with other tracking units 305, provides position and / or motion data of the golf ball and / or golf club used by each player within its field of view. As will be described in more detail below, some tracking units 305 can also identify players within their field of view and associate ball / club data with the corresponding player. As described above, all position data can be converted to a world coordinate system to pinpoint the position of the ball and / or player on the golf course. Tracking of ball trajectory and associated parameters, such as the ball's spin rate and spin axis, may be performed based solely on radar data, or image-based tracking may be combined with radar-based tracking to determine the three-dimensional position of the ball trajectory. Furthermore, pure image-based tracking, or any other tracking technique such as a lidar, may be used in some scenarios. Preferably, these applications will employ non-invasive techniques that do not require changes to the equipment used by golfers (e.g., markings on the ball or club), as they are unlikely to be adopted in any professional tournament. This may not be so important in amateur scenarios where players readily accept the use of different equipment, as it may, as a result, lead to lower service costs, etc. Furthermore, club tracking, including club speed, attack angle, club path, face-to-path angle, and impact position on the clubface, can be performed using radar-based tracking, image-based tracking, lidar-based tracking, or any combination of these and other systems.

[0059] Figure 4 shows an exemplary tracking unit 400, including a radar system 405 and / or a camera system 410 and / or a lidar system 415, according to various exemplary embodiments. The tracking unit 400 may include any combination of the aforementioned sensors, as described above. The various sensors are operable to capture data according to their configured parameters. For some sensors, the captured data can be transmitted directly to a processing unit, such as the tracking server 320 described above with respect to Figure 3, or any other data processing unit that processes the data to determine the track of a coordinate system 420, which may be a local coordinate system or a world coordinate system depending on the application. Other sensors include an internal processing unit for performing some or all of the data processing on the captured data. The tracking unit 400 may also be positioned adjacent to the tee box and configured to capture the launch parameters of a tee shot 440 of a ball 425 by a player 435, which includes both the launch parameters of the ball 425 and the club path parameters of the club 430 used by the player 435. The tracking unit 400 may also be further positioned along the hole and configured primarily for 3D position tracking.

[0060] In a preferred embodiment, the exemplary tracking unit includes a 3D Doppler radar system and a camera system, typically including one or two integrated cameras. The 3D Doppler tracking radar can determine the 3D position of a moving object in XYZ coordinates even when prior information (e.g., launch position) is unavailable. This means, for example, that a ball suddenly appearing from behind a tree can be precisely positioned in 3D as soon as a line of sight is established by the radar. However, predetermined information regarding the launch position may be used if available.

[0061] In this embodiment, the camera is time-synchronized with respect to radar data and calibrated against the radar's coordinate system. This means that, at any given point in time, the 3D position of a moving ball determined by the radar can be accurately mapped to a frame captured by the camera that corresponds to the time and position of the image. Similarly, the 3D position determined for a moving ball, determined from image data, can be correlated with the corresponding radar data to more accurately determine the track, for example. In an alternative embodiment, instead of a 3D Doppler radar, one or more tracking units may include a 1D Doppler radar and a camera system. In this example, the 1D radar measures only the radial distance or range rate of the ball as it crosses the radar's field of view, while the camera system determines the angular position of the ball as it crosses the camera's field of view. When these respective fields of view overlap, the data may be combined to generate a three-dimensional track of the object, as described in U.S. Patent No. 10,989,791, which is incorporated herein by reference in its entirety.

[0062] Preferably, each camera is calibrated intrinsically and extrinsically, meaning that both internal and external parameters of the sensor are known or determined. Camera intrinsic parameters may generally include, for example, focal length, lens distortion parameters, and principal points, while radar intrinsic parameters may generally include, for example, phase offset between receivers. External parameters typically constitute the sensor's position and orientation. Intrinsic and extrinsic calibration of sensors within a tracking unit (or, in some embodiments, across different units within a tracking system) allows the angular orientation of each pixel in the image captured by the camera to be known in relation to the calibration of the radar system. Knowledge of distortion parameters for one or more cameras allows for, where desired, correction of image / video distortion to make it more pleasing to the human eye. Various applications of this distortion correction function are described in more detail below, particularly with regard to tracking the bounce and roll of golf shots, which are difficult to capture using radar alone.

[0063] When two or more cameras are embedded in a tracking unit, the cameras are typically configured with different fields of view (FOV) and operate at different frame rates. For example, the first camera may have 4K resolution, a 50-degree horizontal FOV, and be configured to operate at 30fps, providing a visually pleasing video stream. The first camera may be used, for example, as a broadcast camera or for generating shot clips. The second camera may be, for example, in a green-side tracking unit looking back towards the fairway, and may have a narrower field of view, providing higher pixel resolution at a distance, and be configured to track shots aimed at the green.

[0064] This precise alignment and calibration of the cameras and radar enables radar tracking and image tracking, complementing and enhancing each other. In some scenarios, a ball, club head, and / or a person are detected first by radar, while in other scenarios, these objects / bodies are detected first by one of the cameras. In many scenarios, both sensor types will always have reliable tracking of the object as long as the object is within their respective fields of view. This allows each sensor to be used in different scenarios where each sensor is most accurate, and, where available, data from multiple sensors can be combined and compared to measurements based on only one type of sensor for increased accuracy.

[0065] A preferred system is further provided with techniques enabling time synchronization between tracking units, as described in more detail below, as well as an advanced calibration scheme in which all units are calibrated to the same coordinates (e.g., the world coordinate system). Essentially, as described above, the combination of sensor data and cross-sensor enhancement possible within a single tracking unit is also available across multiple tracking units working in coordination, for example, with a first tracking unit tracking the initial part of the ball's flight and a second tracking unit taking over the latter part of the flight. Those skilled in the art will understand that for portions of the ball's flight that fall within both the fields of view of the first and second tracking units (overlapping fields of view), the data from both can be combined. Alternatively, the system may identify, for example, a first portion of the overlapping field of view where the data from the first tracking unit is more accurate, a second portion of the overlapping field of view where the combined data from the first and second tracking units provides the most accurate and / or most reliable data, and a third portion of the overlapping field of view where the data from the second tracking unit is more accurate. The depiction of these first, second, and third portions of the overlapping field of view can be done on a case-by-case basis, taking into account, for example, the distance from each of the first and second tracking units, occlusion elements, etc. Those skilled in the art will understand that the same general principle allows for similar modifications between different usage modes of data from various tracking units when the fields of view of three or more tracking units overlap.

[0066] In some embodiments, the operation of a second sensor can be controlled using real-time 3D tracking with data from a first sensor. For example, the system can detect events in the real-time 3D tracking information from the data of the first sensor, and these detected events can be used as a basis to trigger the second sensor to start tracking. These events include, for example,: the launch of a shot; the ball passing a certain distance from the tee; the ball moving beyond a predetermined distance from a baseline; and / or other events indicating that the ball has entered or is about to enter the field of view of the second sensor. Events that trigger a change in the operating state of the second sensor can be selected and configured based on the layout of a given hole. For example, a game theory can be used that takes into account the geography of the hole, including items that may obstruct the field of view / coverage of the first or second sensor, such as the length of the hole. Furthermore, the capabilities of each sensor, for example, whether a sensor provides good tracking coverage or not for a given hole, can be taken into consideration. The system may be programmed to begin tracking with the second sensor when the ball passes a certain distance from the tee, and / or moves laterally beyond a predetermined distance from the baseline, and / or moves closer to the second sensor than to the first sensor. In some embodiments, the sensors may have, but not necessarily, completely overlapping fields of view.

[0067] The detected events can also be used to more efficiently manage power usage for different sensors and subsystems of the tracking system. When an event requiring a particular system capability is not currently detected by data from the primary sensor or other system components, the power usage of other sensors, such as sensors located further down the hole, can be reduced, for example, by putting the components that draw various power from the item into sleep mode or low power mode until the system detects an event requiring the item to fully activate or an immediate pending event. This can have significant practical importance, as golf course systems are often powered either by batteries or generators.

[0068] In the relevant embodiments, a robotic camera, e.g., a tracking camera and / or broadcast camera, remotely controlled by a tracking server, is controlled in reliance on real-time 3D position tracking information. Using real-time position data relative to the ball, the tracking system can control the robotic camera to orient it towards the current ball position (e.g., to track the ball through its flight), and additionally control zoom and focus to ensure a clear, viewer-friendly image that is familiar to viewers (e.g., to provide a broadcast video feed) in the same way that is achieved by professional camera operators during golf tournaments. The 3D position of the ball relative to the robotic camera is always known by the tracking system based on real-time data acquired by other sensors within the tracking system. Thus, the optimal crop, orientation, and zoom levels relative to the robotic camera can be automatically controlled by the system based on real-time data. Optical tracking of the ball in flight may also be carried out using analysis of data from the robotic camera (or any other camera), which may be used to control the robotic camera to achieve the desired image using any known method. The robotic camera may also be used to replace a manual camera operator, making the video footage much more engaging for viewers. Furthermore, it is possible to create close-up images, for example, to depict the lie of the ball. To ensure stable and viewer-friendly movement of the robot camera, a special filtered version of real-time 3D data from the tracking system can be generated and used to control the robot camera. In addition, knowledge of robot camera control characteristics such as delay, maximum angular acceleration, maximum angular velocity, and focus delay can be considered and supplemented when controlling the camera. The robot camera can also be used as a tracking sensor for positioning the club and ball in world coordinates. However, this requires calibration of the robot camera because the field of view is not stationary. These embodiments are described in more detail below.

[0069] For example, in one embodiment, the second sensor may be a camera (e.g., a robotic camera), and the activation command may include parameters for controlling the optimal crop, orientation, and zoom level of the robotic camera. The zoom and focus of the robotic camera can be continuously updated based on the tracking of the ball in flight by other sensors or other tracking units, even if the ball is not visible in the image from the camera or is not easily found.

[0070] Figure 16a shows an exemplary method 1600 for tracking the three-dimensional position of a ball launched during play on a hole of a golf course, the tracking being performed using data from both a first sensor and a second sensor, the operation of the second sensor being controlled based on 3D position tracking information determined from data captured by the first sensor. In some embodiments, the first sensor is a radar (e.g., a radar in a tee box tracking unit). In other embodiments, the first sensor may be a radar in a different type of tracking unit, e.g., a fairway tracking unit, or a camera. In some embodiments, the second sensor is a radar, e.g., in a fairway tracking unit. In other embodiments, the second sensor is a camera. In some embodiments, the second sensor may be located in the same place as the first sensor, for example, the second sensor is a camera in a tee box tracking unit and the first sensor is a radar in a tee box tracking unit. In preferred embodiments, the first and second sensors are calibrated to a world coordinate system.

[0071] In the following, specific processing steps are described as being performed in the tracking system. As described above, tracking processing can be carried out by processing units incorporated into each tracking unit, by various steps performed on the tracking server, or in cooperation between the incorporated processing units and the tracking server. As described above, the tracking server can be a physical server on site at the golf course, a cloud-based service, or any combination of such elements.

[0072] In 1602, the tracking system uses a first sensor to capture first data, which corresponds to a first portion of the trajectory of the launched ball, and calculates the ball's 3D position in real time as the data is collected. The second sensor is in a reduced or different capability operating state while the first data is being captured by the first sensor. For example, the second sensor may be in a low-power state. In another example, the second sensor may be fully powered on but not yet performing data acquisition and / or tracking functions. In yet another embodiment, the second sensor may be a robotic camera that captures data according to initial camera parameters for crop, orientation, and zoom level, for example.

[0073] In 1604, the tracking system detects the event of a launched ball based on the first data. In one embodiment, the 3D position of the ball is used as a basis for determining whether an event has been detected to a second sensor. For example, an event can be detected if the 3D position of the ball satisfies one or more position criteria. As discussed above, these events include: a ball passing a certain distance from the tee; a ball moving beyond a predetermined distance from a reference line; and / or other events indicating that a ball has entered or is about to enter the field of view of the second sensor. In another example, an event could include the launch of a shot.

[0074] In 1606, the tracking system controls the operating state of a second sensor (e.g., activating the second sensor and adjusting the data acquisition parameters of the second sensor) so that the second sensor captures second data corresponding to a second portion of the trajectory of the launched ball, based on the detected event. In some embodiments, the activation command may include a command to fully power on and / or start data acquisition for 3D position tracking. In other embodiments, the activation command may include parameters for controlling the robotic camera. In yet another embodiment, the activation command may power a processing module for tracking the bounce and roll of the shot after the initial impact with the ground, as will be described in more detail below with respect to Figures 11-13; 17a-b. In another embodiment, the area of ​​interest may provide an area where the ball is likely to be found, so that the second sensor can detect the ball more quickly and with fewer processing resources.

[0075] The method 1600 described above is particularly related to activating sensors to capture data and / or track a ball in flight, based on three-dimensional tracking data of the ball in flight derived from data captured by another sensor. However, the general principle described above is applicable to many other exemplary embodiments, which will be described in more detail below.

[0076] For example, in one embodiment, the second sensor may be a camera (e.g., a robotic camera), and the activation command may include parameters for controlling the optimal crop, orientation, and zoom level of the robotic camera. The zoom and focus of the robotic camera can be continuously updated based on the tracking of the ball in flight by other sensors or other tracking units, even if the ball is not visible in the image from the camera or is not easily found.

[0077] In another example, event detection might relate to player detection, such as the identity of a player in a video feed, or specific movements taken by a player leading up to a shot, and the broadcast feed or some other processing module might be triggered based on event detection. For example, radar might be triggered based on the detection of a player ready to hit a shot from the tee box.

[0078] In yet another example, specific data packets are triggered for transmission to a broadcast (e.g., a broadcasting entity) based on real-time 3D position tracking of the ball. Metrics related to the ball's flight are processed and / or transmitted when certain events are detected, such as: launch parameters calculated at launch and transmitted afterward; peak height parameters transmitted immediately after the peak height is detected; smooth trajectory parameters calculated at the time of the first collision with the ground and transmitted afterward; and bounce and roll and / or final resting position parameters calculated when the ball is detected bouncing, rolling, and / or at rest, etc.

[0079] In yet another example, event detection may relate to the estimated landing location of a launched ball. If the landing location where a player and / or spectator are located can be estimated based on data acquired at the start of the shot, an automatic warning (an automatic "fore" warning) may be triggered so that these people can protect themselves from a misfire. These and additional embodiments are described in more detail below.

[0080] Radar tracking In radar technology, tracking a ball in flight is typically done using multi-frequency continuous-wave (MFCW) Doppler radar operating in the X-band. X-band Doppler radar is a powerful technology for tracking golf balls in flight. Under good conditions, the ball can be tracked for distances of over 300 meters, and the radar can function in all light and weather conditions.

[0081] Figure 5a shows an example of the first principle of MFCW Doppler radar tracking. The radar wave is transmitted at a specific frequency F TX It is transmitted by the unit. It is reflected by the ball, and returns at frequency F. RX The wave received by the unit is Doppler shifted V relative to the tracking unit, proportional to the radial velocity of the ball. R You will experience this. Frequency shifts can be detected by mixing the transmitted and received signals and performing frequency analysis. Various signal processing algorithms can be applied to detect and track the ball and ultimately estimate its trajectory and various data points.

[0082] Figure 5b illustrates an example of the second principle of MFCW Doppler radar tracking. Angle measurements can be performed by an MFCW Doppler radar including multiple receiving antennas. The antennas are positioned so that the wavefront reflected from the ball reaches two receiving antennas with a time difference that depends on parameters including the direction from the unit to the ball and the distance / direction between the antennas. A system including three or more receiving antennas across the main beam and a plane substantially perpendicular to the tracking direction can perform angle measurements in both horizontal and vertical directions, generating a three-dimensional track.

[0083] After mixing, the time shift may be observed as a phase shift coefficient of 2π between signals in the receiving channel. The 2π ambiguity can be resolved by having three or more receiving antennas cleverly spaced within a two-dimensional grid, as described in U.S. Patent No. 9,958,527, which is incorporated entirely herein by reference. Multiple receiving antennas can also increase the signal-to-noise ratio of the received signal when signals from multiple receivers are added in an orderly manner.

[0084] Figure 5c shows an example of the third principle of MFCW Doppler radar tracking. Range measurements can be performed using multiple transmission frequencies. Wavefronts reflected from the ball, club, or any other item being tracked for each of the two frequencies reach a common receiving antenna with different phases depending on the distance to the ball and frequency separation. After mixing, the phase difference can be observed as a phase shift coefficient 2π between signals in the receiving channel from two or more different transmitter frequencies. The 2π ambiguity can be resolved from predetermined knowledge about the two or more frequencies or the approximate distance to the item being measured. For example, for a ball trajectory, prior information about the distance to the ball at a particular part of the trajectory may be used, and for example, range measurements at launch may be assumed to be within a predetermined range interval of radars located at the tee box.

[0085] As an alternative to transmitting multiple frequencies to determine the range to the ball, frequency-modulated or phase-modulated continuous wave (CW) radar, such as FM-CW radar, can be used, as is well known to those skilled in the art.

[0086] The main advantages of radar tracking technology are its speed, robustness, and maturity; its ability to directly measure the three-dimensional position of a golf ball at any point in its entire flight, rather than making an educated guess; and the vast and increasing data record that allows for the construction of detailed aerodynamic models representing the flight of the golf ball.

[0087] Furthermore, the tracking unit can be equipped with the latest sensor fusion of radar and camera data and is inherently expandable. Where additional precision is important, multiple tracking units may be employed, for example, in combination with a tee area system having a greenside system (and, if desired, additional mid-hole tracking units) to provide extremely high precision in critical events (e.g., launch and / or landing) for every shot played on a hole.

[0088] At any moment when data is captured by the radar (for example, at intervals of approximately 20 ms), determinations can be made for all moving objects visible to the radar, including: radial velocity; range; horizontal and vertical angles; 3D position; and any additional information as described herein.

[0089] When a sufficient number of such detections of golf balls occur consecutively, typically over a time period of 100–500 ms, the launch of the ball is confirmed and an aerodynamic model is fitted to the data. The aerodynamic model may include, for example, a constrained parametric partial difference equation based on Newtonian physics that fits the data in a maximum likelihood sense under appropriate weighting of the data. If the radar does not directly detect the impact time in the raw radar data, the model may extrapolate the time backward to the expected impact position or height. Alternatively, the launch time may be determined using data from other sensors (e.g., a microphone). Launch parameters may then be extracted by evaluating the model at the estimated impact time. Launch parameters may include, for example, the timestamp of the impact; the stationary position of the ball prior to impact; the ball velocity; the vertical launch angle; and the horizontal launch angle. Thus, the aerodynamic model helps to smooth the raw radar data and provide a suitable representation of the ball's flight to date.

[0090] The ball's remaining flight is tracked, for example, using a nonlinear Kalman filter, increasing the likelihood of tracking the ball's entire flight until impact with the ground. Furthermore, if the line of sight to the ball is obstructed for any period of time, the signal from the ball can be reacquired once the line of sight is re-established. Once landed, the bounce can be continuously tracked, if visible on radar, or by an image-based tracking unit or its image-based components. A more detailed description of the methodology for tracking the ball's bounce and roll and determining its final resting position is provided below.

[0091] The live data captured by the system, including data captured by the radar unit, may be continuously transmitted during the ball's movement to the tracking server 320 and / or directly to a third party for broadcast after processing by the tracking unit. Similarly, data received by the tracking server 320 may be continuously processed and transmitted to a third party or for broadcast. Alternatively, data may be transmitted only at critical points in the shot, such as at launch, highest point, carry landing, and when the ball is stationary.

[0092] Figure 6 shows a diagram 600 of exemplary data packets 605-630 containing specific data of a golf shot to be transmitted to a broadcast entity and / or other application during the acquisition of real-time data of the golf shot according to various exemplary embodiments. Data packets 605-630 are shown against an exemplary plot of the shot trajectory over time, providing the approximate transmission time of the data packets relative to the shot launch time. Furthermore, the approximate transmission times of data packets 605-630 are shown and distinguished for different types of radar sensors. For example, the approximate transmission time may vary between radar sensors having different parameters and / or processing capabilities, requiring different amounts of processing time for data packets and / or tracking different types of shots that typically have shorter or longer time from launch to pause. For the purposes of this discussion, latency required for data transmission can be ignored, for example, from tracking unit 305 to tracking server 320 or broadcast / application, and / or from tracking server 320 to broadcast / application. In this embodiment, for clarity, it is assumed that the tracking server 320 receives data packets from the tracking units and transmits the data packets to the receiving broadcasting entity broadcasting live video of the shots from any number of viewpoints. The tracking server 320 can receive data / tracks from a single tracking unit or multiple tracking units. When processing of the raw radar signals occurs at the tracking server 320, the data packets 605-630 shown in Figure 6 are generated by the tracking server 320 and transmitted for broadcast.

[0093] Live video may include a shot tracer, which is determined substantially in real time using the techniques described in more detail below. During the broadcast of a shot, the broadcast can be enhanced with real-time information about the shot in progress by displaying the tracer and simultaneously displaying information contained in the received data packets. Those skilled in the art will understand that a tracer generally refers to a video feed graphic that shows a visual representation of the shot's trajectory, inserted into the image at positions corresponding to the positions in the image occupied by the ball, from launch to the ball's position in the current image. The tracer graphic may include numerical data indicating various points in the trajectory, such as ball velocity, highest point height, carry, etc.

[0094] The first data packet 605 transmitted from the tracking server to the broadcast includes launch data, such as impact time, stationary ball position, launch velocity, and both horizontal and vertical launch angles. To process and transmit these parameters to the broadcast subject via some communication medium, a typical tracking unit (combined with the tracking server) may transmit with a delay of, for example, less than approximately 600 ms (for a tee box tracking unit), approximately 1000 ms (for a green tracking unit), or approximately 650 ms (for a fairway tracking unit). That is, the data is processed, incorporated into the image, and ready for broadcast in a manner that requires minimal delay from pure live timing (very close to "real time"), making the efficiency of data collection and processing extremely important. Launch data can be displayed in the broadcast in various other ways, including in the rendering of the hole / shot, in addition to being stored as numerical or graphical data overlaid on the video feed or for later operation and analysis.

[0095] A second data packet 610 transmitted from the tracking unit to the broadcast includes real-time trajectory data, e.g., the three-dimensional position of the ball, which is captured and determined. At any point during the ball's flight, the tracking unit 305 may initiate transmission of this real-time trajectory information in the data packet 610, and may proceed with the transmission of these data packets 610 at predetermined intervals, which may be limited by the capabilities of the tracking unit and / or transmission medium through which the data is communicated. Transmission may end after a predetermined time when the shot reaches a predetermined position, e.g., its highest point reached, or when a further data packet containing additional information is transmitted. In the embodiment of Figure 6, the transmission of the real-time trajectory data packet 610 is triggered to begin when the first data packet 605 containing launch data is transmitted. Transmission of the real-time trajectory data packet 610 ends when the third data packet 615 is transmitted, as described below. However, those skilled in the art will understand that the real-time trajectory data packet 610 may be configured for transmission for any time during the shot's flight.

[0096] A nonlinear Kalman filter can be smoothed at every opportunity when data is sent to the broadcast for live rendering of the trajectory. This provides the most accurate post-hoc estimate of the ball's trajectory according to all the data recorded so far. The smoothed trajectory can be further filtered by an automated regression process before display to improve the viewer experience. The trajectory data can be displayed in various ways, including, for example, as a trajectory plot on a rendering of the hall. Numerical data can also be used directly, for example, as counters for distance, curve, height, etc.

[0097] An optional third data packet 615 transmitted from the tracking unit 305 to the broadcast includes a landing prediction. The landing prediction may include an estimated three-dimensional landing position of the ball based on information collected about the ball's flight so far. The landing prediction may also provide uncertainty information regarding the estimate. The tracking unit can specify that the third data packet 615 be transmitted approximately 1500 ms (for the tee box tracking unit) after the launch time, for example, at the highest point reached by the shot (for the green tracking unit), or approximately 1500 ms (for the fairway tracking unit). The landing prediction can be displayed on the broadcast in various ways, including, for example, as the position on the rendering of the hole. A fourth data packet 620 transmitted from the tracking unit 305 to the broadcast includes the spin rate. The tracking unit can specify that the fourth data packet 620 be transmitted approximately 2500 ms (for the tee box tracking unit) after the launch time. In green tracking units and / or fairway tracking units, spin rate may not be transmitted to the broadcast; determining spin rate is a relatively complex calculation, and this information may be omitted for short shots generally captured by green and fairway units. The spin rate determination can be displayed on the broadcast, for example, as a numerical value or in any desired graphic format.

[0098] A fifth data packet 625 transmitted from the tracking unit 305 to the broadcast includes, for example, the live highest point, including the height, range, and lateral aspect of the highest point. The highest point lateral aspect refers to the lateral distance of the ball from the shot's known or assumed target line, for example, the direction to the pin on a short hole, or the direction to the center of the fairway at a specific distance from the tee box on a long hole. Alternatively, the highest point lateral aspect can be determined relative to the ball's initial launch direction. The tracking unit can specify that, regardless of the type of sensor included in the tracking unit 305, the tracking unit 305 transmits the fifth data packet 625 when it determines that the shot has reached its highest point. The live highest point determination can be displayed on the broadcast, for example, as a number or graphic overlaid on the hole rendering.

[0099] The sixth data packet 630 transmitted from the tracking unit 305 to the broadcast contains the final measurements of the shot after the ball has landed. Once the ball hits the ground, a more complex aerodynamic model than the one used at launch may be fitted to all the collected data to provide a determination of the most likely trajectory of the ball from impact through its highest point to landing. The aerodynamic model can capture the ball's spin decay, in addition to quantifying the way in which the ball's spin and velocity affect instantaneous drag and lift on the ball. The model may further take into account the effects of weather and wind. In one embodiment, an aerodynamically smoothed trajectory can be used to estimate the effective wind speed and direction that affect the ball's flight.

[0100] The model is appropriately evaluated to extract values ​​for all final data points, including: impact timestamp; stationary ball position before impact (if measured); ball velocity; vertical launch angle; horizontal launch angle; height, lateral, and range of highest point; estimated spin axis (if not measured); spin rate (if not measured); carry and side (flat); landing angle and velocity (flat); and flight time (flat). The "flat" values ​​for carry and side, landing angle, velocity, and flight time refer to values ​​determined at the same height / level from which the ball was launched. Flat values ​​are useful for comparing shots that were not launched / landed in the same location. The aerodynamic model is further used to extract actual values, rather than flat values, for flight time; landing angle; landing velocity; and range and lateral by intersecting the trajectory with a known 3D representation of the golf hole layout. "Actual" values ​​refer to values ​​determined by the actual collision between the ball and the ground, taking elevation differences into account. If desired, the aerodynamic model can be summarized in a set of polynomials, which allows for easy movement, visualization, and manipulation by third parties.

[0101] The sixth data packet 630 may contain any desired primary metrics, including smoothed trajectory data and more precise versions of those metrics determined / predicted and transmitted in the previous data packets 605-625. To process and transmit these parameters to the broadcasting entity via some communication medium, a typical tracking unit 305 may request, or otherwise specify, to transmit the sixth data packet 630 for the time of impact with the ground, for example, about 300 ms later (for a tee box tracking unit), about 350 ms later (for a green tracking unit), or about 350 ms later (for a fairway tracking unit). The final shot measurement can be displayed on the broadcast in various ways, including any of the methods described above with respect to data packets 605-625.

[0102] The specific configurations and transmission times of the data packets 605-630 described above are provided for illustrative purposes only, and it should be understood that any number of similar or different data packets may be transmitted to the broadcast for any reason, such as a specific request from the broadcaster or the capabilities of the tracking unit. For example, packet transmission architectures specific to different sensors may be configured so that sensors transmit different data at different times, depending on factors such as the processing time for shot parameter determination or the type of shot typically being tracked. Additional data related to the bounce and roll of the shot after landing may be provided, for example, in a sixth data packet 630 along with the final trajectory measurement data, or in separate packets at different points in time. Specific techniques for analyzing the bounce and roll of the shot are described in more detail below with respect to Figure 13.

[0103] Figure 16b shows a method 1610 for transmitting tracking data to be included in a broadcast, according to various exemplary embodiments. As described above, the tracking system may include a radar unit for tracking launch and / or flight and / or impact of the ball with the ground (and in some embodiments, the radar unit may track bounce and roll after the initial impact with the ground). The tracking system may also include a camera for tracking bounce and roll. Data may be acquired about the entire shot trajectory, and various tracking units may work in conjunction with a tracking server to calculate the ball's 3D trajectory and detect events related to the ball's flight.

[0104] In 1612, the launch event is detected by the tracking system, and the launch data of the launched ball is captured using the sensors of the first tracking unit. The tracking unit (and / or tracking server) begins processing the launch data immediately after launch and derives the launch parameters.

[0105] In 1614, at least a first data packet containing the determined launch parameters is transmitted to the broadcasting entity. The first data packet may be transmitted immediately after the launch parameters are determined by the tracking server (or received by the tracking server after determination by the tracking unit), or at a predetermined point in time after launch.

[0106] In 1616, multiple second data packets are transmitted to a broadcasting entity containing real-time 3D trajectory data. In one embodiment, the transmission of the second data packets may be triggered at a predetermined time after launch and terminated at a predetermined time after start. In another embodiment, the transmission of the second data packets may be triggered at the time of transmission of the first data packets. In yet another embodiment, the transmission of the second data packets may be triggered when a 3D position event of the ball is detected, for example, when the ball reaches a specific point in its trajectory, for example, a predetermined height or distance from the tee. In yet another embodiment, the termination of the transmission of the second data packets may be triggered when a 3D position event of the ball is detected, for example, when the ball reaches a specific point in its trajectory, for example, a height determined as the highest point reached or a percentage of the predicted highest point reached.

[0107] In 1618, additional data packets may be transmitted to the broadcasting entity while the ball is in flight. For example, a third data packet may include the predicted landing position (e.g., transmitted after the transmission of real-time trajectory data in 1616 has finished), a fourth data packet may include the spin rate (e.g., transmitted after a predetermined time has elapsed after the flight, or immediately after sufficient data has been acquired and processed to accurately determine the spin rate), and a fifth data packet may include the live highest point reached (transmitted, e.g., at detection).

[0108] In the 1620, collision events are detected by the tracking system (for example, the system detects when the ball first hits the ground in flight), and ball trajectory data is accumulated across sensors and / or fitted to an aerodynamic model to determine the final measurement of the ball's trajectory (which may be more accurate than the measurement provided in previous data transmissions).

[0109] In 1622, at least one sixth data packet is transmitted to the broadcasting entity containing the final measurement of the ball's trajectory. The sixth data packet may be transmitted immediately after the final measurement is determined, or at a predetermined point in time after impact with the ground.

[0110] In 1624, the ball's landing data, including bounce and roll and / or final resting position, may be tracked by a tracking unit and / or further tracking units, such as a camera located near the point of impact with the ground. During this part of the ball's motion, the tracking unit (and / or tracking server) may process the bounce and roll, determine 3D position data, and further determine the resting position, as will be described in more detail below with respect to Figures 11-13.

[0111] In 1626, at least one seventh data packet is transmitted to the broadcasting entity containing landing information. The seventh data packet may be transmitted immediately after the landing information is determined, or at a predetermined time after the ball has come to rest.

[0112] In 1628, the broadcasting entity receives each packet transmitted during the ball's flight and after collision / come to rest, and presents the data contained in the data packets in the broadcast video feed. The information can be processed into visually pleasing graphics, etc. The bounce and roll data can be combined with the initial trajectory data to display the ball's complete path from launch to come to rest.

[0113] Although method 1610 described above involves the transmission of seven types of data packets, those skilled in the art will understand that different numbers and / or types of data packets can be used.

[0114] Certain phenomena can interfere with the quality of radar tracking of a ball, for example: shots with particularly low ball speeds; shots traveling nearly perpendicular to the line of sight to the radar unit; reflections of strong multipath signals, mainly from the ground; objects obstructing the direct line of sight to the radar; and external devices emitting electromagnetic interference. Most phenomena affecting the quality of radar tracking can, however, be assisted by augmenting raw radar data with visual tracking of the ball.

[0115] In preferred embodiments, one or more tracking units may include a tracking camera capable of image plane tracking or three-dimensional position tracking, in addition to the tracking capabilities of the radar itself. The radar may also be used to orient and focus the camera, often to provide the camera with a region of interest, reduce the calculations required to detect the precise pixel position of the ball, and further improve the quality of the image-based tracking data. Data from the two sensors are then integrated, along with accurate horizontal and vertical angle measurements from the camera, to extract accurate estimates of range and range rate from the radar. This fusion of the two technologies yields the advantages of both: speed, robustness, weather resistance, and range measurement capabilities of radar tracking combined with the angular accuracy of camera-based tracking.

[0116] The tracking system is inherently scalable, allowing for the addition of any number of tracking units for collaborative tracking and sensor fusion. This may be decided on a case-by-case basis and applied when accuracy and / or reliability are paramount. Apart from the obvious benefit of improved accuracy, additional sensors can equally address many of the aforementioned problems that hinder radar tracking to ensure that every shot is reliably tracked.

[0117] calibration For the tracking units to output meaningful data, it is crucial that each tracking unit 305 can map its measurements to a world coordinate system different from its own local coordinate system. This allows the trajectory to be overlaid, particularly on images or videos of the course, and also enables the insertion of graphics such as tracers into broadcast video feeds. The accuracy of specific data points depends on both the measurement accuracy of the tracking unit 305 and the accuracy of the calibration of these measurements to world coordinates. The types of measurements thus affected include: stationary ball position (before impact); horizontal launch angle; highest point side view; and carry side view (flat).

[0118] The method for calibrating the tracking units to the world coordinate system may be the same across all tracking units 305, regardless of whether each individual tracking unit 305 is placed on the tee overlooking the fairway or on the green looking back at the fairway. Calibration can be performed in many different ways. In a preferred embodiment, the GPS positions of key reference points on each hole are determined. In addition, the GPS position of each tracking unit 305 is determined. For each tracking unit 305 including a camera system, one or more reference positions are determined in the camera image, allowing the orientation of the tracking unit 305 to be determined in world coordinates. If the tracking unit 305 is moved after calibration has been performed (for example, if the tracking unit 305 is hit by someone on the course or if a structure to which it is attached moves it), the above measurements may be affected. Changes in the orientation of the tracking unit 305 can be detected by feature matching in images from the built-in camera. Alternatively, changes in orientation can be determined by motion sensors incorporated in one or more tracking units 305 or by other means.

[0119] The fairway tracking unit 305 can be calibrated to the world coordinate system in a similar manner, although for portable tracking units, it may be preferable to omit calibration to the world coordinate system for speed reasons. The fairway tracking unit 305 may also be calibrated to a broadcast camera, for example, to provide tracer and / or other information to a common coordinate system with the camera, as described in more detail below.

[0120] Add another camera for broadcasting / filming. As described above, some or all of the tracking unit 305 may include a camera system. However, one or more additional cameras may be placed on each or some of the holes for the purpose of broadcasting / filming golf shots. Hereafter, a camera dedicated to filming will be referred to as a broadcast camera, insofar as its primary purpose is to provide a good viewing experience in real time, near real time, or after the fact, and does not primarily provide data for tracking, regardless of whether the camera is actually used for broadcasting.

[0121] The position and orientation of the broadcast camera can be determined in a manner similar to that used for the tracking unit 305. Thus, any ball, whether in motion or stationary, can be positioned, at least substantially, within the image generated by the broadcast camera. In this way, ball tracking information, such as a tracer or other image illustration, can be inserted into the image from the broadcast camera at positions in the image corresponding to positions determined based on tracking information from the tracking system.

[0122] In some cases, the camera system located within the tracking unit 305 may be used as a broadcast camera. In this case, the same images used for tracking can be used for broadcast.

[0123] Adding additional cameras to supplement the tracking unit 305 is a straightforward process. The calibration scheme facilitates the inclusion of other cameras in the tracking and imaging tasks, including time synchronization between the tracking units 305. For example, a pan, tilt, and zoom (PTZ) camera, such as a robotic camera, may be added. With a PTZ camera, pan, tilt, and zoom can be directly controlled based on the real-time ball trajectory captured by the tracking system, as will be described in more detail below.

[0124] Player identification As described above, a unique player ID may be associated with each player on the golf course and can be used to accurately associate each golf shot tracked by the tracking system with the player who hit the shot. This player identification function can be implemented in many different ways, either alone or in combination with alternative options, as described below.

[0125] In the first option, the system analyzes the swing characteristics of each golfer in the tournament and generates a unique identifier for each golfer based on the unique aspects of their swing characteristics. For example, unique swing characteristics may include a combination of biometric data points (e.g., ratios of forearm and upper arm length and height) and biomechanical data of each golfer's swing (e.g., the position of the forearm and upper arm relative to the club shaft at different points during the swing, the degree of body twist relative to the legs).

[0126] In the second option, the system recognizes the golfer's clothing and / or face. For example, each golfer may be manually identified by the system at the start of each round, and the characteristics of the clothing worn by the golfer may be matched to the golfer in the system for recognition throughout the round. This information can be used by the system alone or in combination with any other biometric and / or biomechanical characteristics to recognize each golfer for each shot. This option can be used alone or in combination with any or all of the other options, for example. In the third option, each golfer carries an electronic identifier that allows the system to locate them. For example, this could be as simple as a smartphone app that transmits a unique identifier recognized by the system as well as GPS coordinates, which can then be mapped to world coordinates. This option can be used alone or in combination with the first or second option, for example.

[0127] In the fourth option, the player identification function can additionally use theories about how players generally play a round of golf. For example, players generally play in groups of 2-4 (e.g., usually 3 golfers during a tour), and during a round, players in such a group play each hole together, taking turns so that the member of the group furthest from the hole takes the next shot. In another example, after a player has hit a shot, that player's next shot will occur from a position close to where their tee shot landed. In yet another example, the system may employ knowledge about the course layout. For example, after a group finishes a hole (e.g., Hole 1), the system predicts that the group will move to the tee box for the next hole (in this case, Hole 2). These competition theory methods by the fourth option can be used in combination with the aforementioned options 1-3 to improve player identification and / or reduce the processing load required for player identification. Using this type of competition theory, and by tracking the progress of each group as they advance through the course, player ID tracking may generally only need to distinguish between members of a given group (for example, the system only needs to track each group and then, by knowing where that group is on the course, determine which of the three players in a group known to be playing at a particular point on the course is about to hit a particular shot). This additional assistance should be available for all shots during a round, except for the first shot of a group that starts the round (e.g., a group's tee shot on the first hole). Even for these first tee shots, the system may be able to obtain information about the identity of the golfer in the group currently at the first tee. This option can be used in combination with any of the three options mentioned above to narrow down the list of options considered by the player identification system, for example, when determining the identity of a player detected to be preparing to hit.

[0128] Regarding the image-based options discussed above (the first and second options), a visual profile can be generated when a player is first detected and associated with a unique player ID. When a player is detected in a video stream, the algorithm can match the detected visual profile with previous visual profiles detected in the past (e.g., during the same round, or in previous events for non-clothing-based characteristics). For example, a player would rarely change their shoes, trousers, or hat during a round, nor would they likely change the appearance of their upper body clothing. If it starts raining, a player might put on a rain jacket, or if it gets hot, they might take off a sweater. By utilizing these characteristics, the algorithm can re-identify a player in any camera (given sufficient image quality). This functionality can be implemented, for example, by a convolutional neural network (CNN). Feature extraction and classification of features associated with unique player IDs can be used to distinguish players. Feature extraction can be performed by detecting clothing color, height, limb length, golfer's swing type, or by training an automated encoder. In the latter case, the feature vector is provided by an automated encoder, but it is not necessarily an explicit feature for human interpretation. When a golfer is first identified, the feature vector v_0 may be associated with the detection, and subsequent detections of players that yield a feature vector v_ni similar to the original detection v_0 are interpreted as being the same golfer. Clusters of detections corresponding to a particular golfer can be updated with new feature vectors if the detections are made with high confidence.

[0129] Once the batting order is determined at the first tee, each player's name and / or player ID can be associated with a unique visual profile, and any subsequent matches to this visual profile throughout the round can be tagged with the player ID associated with that player. This identification function can be further improved by utilizing competition theory, for example, knowledge about which players are playing in the same group during the round, and knowledge that these players will play hole 2 after playing hole 1. For non-tee shots, for example, where image quality may not be sufficient to identify a player, the player identification system can use knowledge from the tracking system, including ball flight tracking and the final resting position of the previous shot as determined by the tracking system. Using this data, the system can determine with high reliability where a given player's next shot will be taken.

[0130] A player database can be created so that even at the first tee, the system can recognize players found in the database and associate them with their corresponding player IDs. For example, this type of database can be created for all players on the PGA Tour. For players participating in a PGA Tour event for the first time, the system can create a new visual player ID when the player hits their first shot from the first tee, or before this first shot (e.g., during a practice round, warm-up activity). In short, automated player ID tracking would be provided throughout the round. The system can operate on logic similar to that employed by trained humans who know all of the players' visual appearances.

[0131] Figure 16c illustrates Method 1630 for identifying a player and associating shot data with a specific player profile associated with a unique player ID, in various exemplary embodiments. A system for carrying out Method 1630 includes at least a camera and one or more tracking units for capturing shot data of a player identified in images from the camera.

[0132] In 1632, the player identification system receives player information to create a database of visual profiles containing information for recognizing players associated with visual profiles. For example, the database may include visual information of players, such as swing characteristics (e.g., a combination of biometric data points including biomechanics data of each golfer's swing), clothing characteristics, facial characteristics, or a combination thereof. Each visual profile is further associated with a player ID that can be used by the player identification system to associate shot data with the visual profile. In some embodiments, the database may include PGA Tour players to track professional events, for example, and in other embodiments, the database may include amateur players.

[0133] In 1634, the player identification system can receive / generate player information for new players who have not yet been identified in the database or associated with a unique player ID / visual profile. For example, the system can analyze the new player's swing, clothing, or facial features and associate the determined information with a new visual profile. Furthermore, clothing features can be updated for any player already associated with a visual profile, for example, at the start of the current round. In some embodiments, for example, during amateur player tracking, an application connected to the player identification system and / or tracking system can be installed, for example, on a smartphone or smartwatch. Each golfer can provide a positive indication within the application, which can be used to associate the tracking system player ID with the golfer, for example, at the start of a round. Further considerations regarding amateur round tracking are described in more detail below. The database can be entered using one of 1632 and 1634 or a combination thereof.

[0134] In any 1636, the player identification system implements algorithms that utilize golf competition theory to improve its player identification capabilities (for example, all players playing together in a group can be associated with each other for a round of golf so that the system can assume any golfer playing with a recognized member of the group is another member of that group). This may include theories about how players typically play golf; course layouts; etc., so that when players / groups are tracked around the course in relation to shots hit from a particular location, the player identification system only needs to distinguish between members of a given group. In some embodiments, a trained convolutional neural network (CNN) is used to implement the competition theory and / or player detection / matching functions described below.

[0135] In 1638, players are detected in a video stream from a first camera during a round of golf, and a visual profile is detected based on, for example, clothing, face, swing mechanism, etc. The detected visual profile is matched with a visual profile in a database. In some embodiments, the player's position is determined, for example, in world coordinates. As described above, players may be detected based solely on visual features observed in the video stream. Electronic identifiers (e.g., transmitted from the player's phone) can be used to improve the match (and in some embodiments, can be used instead of image analysis). A CNN may also be used to improve the match between the detected visual profile and the visual profile in the database.

[0136] In any 1640, any CNN in step 1636 can be updated based on player detection, such as in step 1638. In some embodiments, the CNN is updated after each player detection in the camera feed, while in other embodiments, the CNN is not retrained much (e.g., after many detections or after a predetermined time since previous detections).

[0137] In 1642, a golf shot is detected by a player detected in the video stream. In some embodiments, a first camera used to identify the player is also used to detect the shot, while in other embodiments, the shot is detected by other sensors in the tracking system. In some embodiments, the golf club or type of golf club used by the player is also identified and associated with the shot, as described in more detail below.

[0138] In some embodiments described in more detail below, particularly with respect to amateur round tracking, it is not necessary for a player to be detected in a video stream for a shot to match. For example, a player may have a wearable device associated with the player that is capable of detecting swing motion. The wearable device can detect the swing and associate the swing with a timestamp and / or location (e.g., GPS location) and send this information to a tracking server. When the tracking system next detects and tracks a shot, the tracking server compares the swing detection (and associated timestamp and / or location) based on the timestamp and / or location associated with the swing and shot, either in real time or later (e.g., at the end of the round), to match the swing with the detected shot and associate the shot with the player from whom the swing information originated. In an alternative embodiment, instead of using a wearable device that detects swing motion, a device that provides timestamped location information, such as a mobile phone, can be used to match shot tracking recorded in world coordinates with the corresponding location of the device at the time the shot was hit.

[0139] In 1644, a detected player and shot are matched and associated with the player's unique player ID. The match is based on the shot's firing position in world coordinates, determined by the tracking system, and matches the position determined for the detected player in the video stream from the first camera.

[0140] As will be described in more detail below, associating shots with unique player IDs can be used for a variety of purposes, such as sending shot data to the player immediately after processing, accumulating shot data for an entire round for a player, or accumulating clips of each shot taken by a given player during a round. In further embodiments described in detail below, the method 1630 described above may be optimized for either professional event tracking or amateur round tracking.

[0141] In any 1646, shot and associated player data are output during play. For example, in professional round tracking, player shot information can be transmitted to the broadcast. In another example, in amateur round tracking, shot information may be transmitted to an application on the player's personal device. Ball tracking data can be transferred to the personal application immediately after recording so that the player can review the data while playing a round of golf. One particularly interesting application of this function could be to help golfers find their balls. For example, when a golfer hits a shot that is difficult to find, the personal application may narrow down the most likely locations where the ball could be found, for example, on a golf hole map provided in the application. In another embodiment, if the ball lands in a water hazard or goes outside the playing area, for example, the system can determine precisely the area where a new ball is allowed to fall and show this area to the golfer via the application. Further examples of applications of the player ID function to round tracking for both professional and amateur golfers are described in more detail below.

[0142] In 1648, data on all shots taken by a particular player during a round of golf is stored in a database, for example, to be presented to the player at the end of the round or for other purposes. The data can be used for various purposes, such as providing statistics to the player for analysis or generating graphics.

[0143] Automatic club type tagging For cameras positioned behind or to the side of the player (e.g., on a tee box overlooking the fairway), algorithms can be trained to recognize the type of club each player is using for all or some of their shots. In this field of view geometry, the images from the cameras would provide a view of the clubhead from behind. Figure 7 shows an exemplary image of the clubhead from a position behind the player. The cameras used for tracking club data during tee shots can operate, for example, at 200fps with a pixel resolution of 2.5mm / pixel. These camera specifications may meet the minimum camera capabilities required for club trajectory tracking.

[0144] One option for determining club types based on camera data is to train a neural network to recognize different categories of clubs. For example, a neural network could be trained to distinguish between seven different categories of golf clubs, including drivers, woods, hybrids, long irons, short irons, wedges, and putters. More sophisticated training of the neural network would allow for more detailed analysis, enabling the determination of more specific club characteristics, such as specific club types and the distinction between features of different clubs within the same club type.

[0145] An alternative option is to use electronic tags that emit a unique identifier when in use (e.g., during a swing) to identify club types. A receiver can receive a signal when a club is in use and match the club's identifier with the player profile associated with the identifier. Receivers can be placed, for example, at each tee box. With this option, the club identification system can additionally use data from a tracking system to, for example, verify that the club transmitting the signal is likely to have been used in a shot.

[0146] On the PGA Tour, the club category for a specific player's shot is currently entered manually, so a way to automate this would be highly desirable. Club information should be associated with the shot, and further associated with the player who hit that shot.

[0147] Broadcast camera calibration To accurately render the tracer on the broadcast camera feed, the broadcast camera is calibrated to world coordinates. This can be achieved in the same way as calibrating the tracking unit to world coordinates, as described above.

[0148] For portable units, such as fairway tracking units, it is not necessary to map the shot trajectory to a world coordinate system. In some scenarios, a broadcast camera can be mounted together with the fairway tracking unit. All that is required to draw the tracer within the feed from the broadcast camera mounted on a tripod together with the fairway tracking unit is calibration of the broadcast camera to the tracking unit coordinates. This baseline calibration can be performed, for example, the morning before each round, by establishing the relationship between the intrinsic parameters of the tracking unit's built-in camera and the broadcast camera at different zoom levels. For each stroke, the zoom level is determined using feature matching between images from two cameras, including other information that may include tracking data such as the position of the ball in the image from the same time (e.g., the ball's stationary position before the shot), and the tee position and the ball's range at launch. This allows the broadcast camera to zoom independently for each stroke and accurately render (or place any other graphic) the tracer at the desired position relative to the ball's position in the image from the broadcast camera, without the tracker having to read the zoom level from the broadcast camera.

[0149] Automatic generation of broadcast feeds As described above, the whole-course tracking system enables automatic tagging of shots, i.e., automatic identification of the golfer who hits each shot. A coordinated tracking and broadcasting system, including the ability to film, track, and insert into video feed graphics such as tracers (visual representations of the shot trajectory inserted along the ball's path) for virtually every shot hit on the golf course, may also make available to each golfer the collection of video and data of all shots hit by that golfer. This allows for the assembly (automatic or semi-automatic) of broadcasts customized for individual consumers or different markets. For example, if a golfer is very popular in a particular region or country, the broadcast feed for that region or country could isolate all shots (or any desirable subset) hit by this golfer and automatically insert them into a custom feed. The same may be done in a more personalized form, where consumers can indicate, for example, their desire for lists of preferred golfers and / or preferred golfer-tournament leader pairings, etc., to be included in a custom feed that can be viewed, for example, through internet-based distribution.

[0150] To make the production of these individual feeds more efficient, images and / or video provided by standard broadcast cameras may be supplemented by images and / or video from cameras included in some or all of the tracking units installed around the course. In one embodiment, a first tracking unit installed adjacent to the tee box of each hole includes, for example, a radar tracking unit, plus at least a first camera and potentially additional cameras. For certain holes where the green is not clearly visible from the first camera (e.g., holes with a large elevation difference or holes where trees or other obstacles curve so that they obstruct the view of the first camera before the ball reaches the green), a second tracking unit including a second camera may be positioned behind the green (i.e., on the side of the green opposite the direction of the approach from the fairway to the green). If necessary or desired, additional tracking units including additional cameras may be positioned along the length of the fairway between the tee box and the green to ensure full coverage of all shots. For example, the initial part of the tee shot may be shown by a standard broadcast camera (or a first camera) located behind the tee box, and after the ball is launched, the system may automatically switch to another camera in one of several different ways.

[0151] Firstly, based on a theory of play that includes, for example, an analysis of the geography of a given hole (e.g., knowing the length of the hole and the distance to any items that may obstruct the view of the broadcast camera behind the tee box or the first camera), the system may be programmed to switch to a second camera (e.g., located behind the green) when the ball has passed a certain distance from the tee and / or has moved laterally beyond a predetermined distance from the baseline. Alternatively, the system may match the trajectory of a shot measured by the first tracking unit with that measured by the second tracking unit (i.e., the system can determine that the trajectories being tracked simultaneously by the first and second tracking units correspond to the same ball in flight), and when a predetermined criterion is met (e.g., the ball is closer to the second tracking unit than to the first), the system can automatically switch the broadcast feed to the second camera. This may be useful when one of the tracking units is a moving unit, as the use of a predetermined cutoff point is impractical when the positions of the first and second tracking units change relative to each other with each shot.

[0152] Furthermore, the system can manage individualized broadcasts so that, if two desired golfers hit their shots simultaneously or nearly simultaneously in a given feed, the system can identify the shot from one of the golfers after the feed for the other desired golfer's shot is complete and buffer it for broadcast. This parameter may be set in the system (for example, delaying the feed for the second shot until a certain amount of time has elapsed after the previous shot has come to a standstill). This automation enables the production of numerous different, high-quality individualized feeds without increasing the need for human operators to manage a significantly increased number of camera angle switches required, etc. Using the same approach as described above, it is also possible to automatically generate highlights for broadcast / webcast or on-demand services (for example, generating highlight clips for a specific golfer or group of golfers).

[0153] Furthermore, to ensure a smooth transition between announcements / commentary between feeds, computer-generated commentators can be used to automatically provide viewers with relevant information, similar to what human commentators would provide, based on information known to the tracking system. For example, historical data could be used to provide information about the situation of the upcoming shot. Historical data could include, for example, phrases commonly used in each of multiple repeating scenarios, shots performed by the same player in similar shot scenarios, shots performed by other players in similar shot scenarios (from the same day of the same event, from the day before the same event, from previous tournaments, etc.), or general performance metrics accumulated for a player. In another example, current data could be used to provide information about the situation of the upcoming shot, such as the current hole number, current shot number, distance to the cup, and lie. Automated commentary may be provided in text format (e.g., as subtitles), audible (e.g., computer-generated speech), or a combination of audio and text. If the broadcast feed does not include an audio feed, or if it is desirable to enhance the audio feed, artificial sounds such as shot sounds (inserted into the feed when firing is detected, or when natural sounds are attenuated, or unavailable for any reason), background sounds, etc., may be inserted into the broadcast feed. Automated commentary can be provided in virtually real time (during the live broadcast) or in highlight clips generated after the live event.

[0154] The principles described herein for tracking and imaging golf balls and for the automatic generation of broadcasts should be understood to be applicable in other settings, for example, for other sports, particularly in scenarios where coverage of multiple different events occurring simultaneously is desirable. For example, these principles can be applied to live broadcasts of tennis tournaments. In another example, these principles can be applied to live broadcasts of American football where many different American football games are taking place within the same time period. Regarding video feeds generated after events (e.g., highlight reels), these principles can be applied to any sport, such as baseball or football (soccer).

[0155] Figure 16d shows Method 1650 for automatic broadcast feed switching in various exemplary embodiments. A system for implementing Method 1650 may include a tracking system that includes multiple cameras providing a broadcast video feed to a broadcast entity, a tracking unit for 3D position tracking of a launched ball, and a processing unit for detecting events in the tracking data that can trigger a switch from the current broadcast video feed to a new broadcast video feed.

[0156] In 1652, a broadcast video feed is generated based on video feeds from one or more broadcast cameras according to predefined rules. For example, a third-party broadcast, i.e., a video feed controlled by a different entity (broadcasting entity) or selected based on criteria outside the scope of this embodiment may be used. For the purposes of this embodiment, the current broadcast feed can be assumed to be a broadcast feed of the tee box showing the player in the moment before hitting the tee shot.

[0157] In 1654, the tracking system detects the event of a launched ball based on the ball's trajectory data. In one embodiment, the ball's 3D position is used as a basis for determining whether an event has been detected with respect to the current broadcast camera compared to other potential broadcast cameras. For example, an event can be detected if the ball's 3D position meets one or more position criteria. As discussed above, these events include: a ball passing a certain distance from the tee; a ball moving beyond a predetermined distance from a baseline; and / or other events indicating that the ball has entered or is about to enter the field of view of a second camera. In another example, an event could include the launch of a shot.

[0158] In 1656, the broadcast feed switches to the video feed of the second camera based on the detected event, so that the second camera captures the second portion of the ball's trajectory. Delays may be implemented after event detection and before the supply switch. For example, a shot may be detected from the tee box unit, and a predetermined delay (e.g., 3 seconds) may be applied before switching to the fairway unit.

[0159] If the second camera is a robotic camera, its parameters can be controlled before switching feeds. For example, optimal cropping, orientation, and zoom for the robotic camera can be adjusted to provide a visually pleasing video feed for the broadcast audience.

[0160] Figure 16e shows Method 1660 for the automatic generation of custom (e.g., personalized) broadcast feeds in various exemplary embodiments. A system for carrying out Method 1660 may be similar to a system for carrying out Method 1650 described above. Furthermore, the system may include a camera (e.g., the same as a broadcast camera) that detects the player of the image data, identifies the player and / or its features, and triggers a switch from the current broadcast video feed to a new broadcast video feed.

[0161] In 1662, criteria for custom broadcast feeds are received. Criteria can, for example, specify the priority of players displayed in the broadcast feed. For example, a custom feed may be generated based on criteria including: all shots for one or more specific golfers being displayed in the custom feed; all shots for all golfers in a specific country / region being displayed in the custom feed; identifying the hierarchy of specific players and setting broadcast priorities accordingly; prioritizing specific holes; prioritizing specific types of shots (e.g., very difficult shots), etc. Criteria can further identify features overlaid on the broadcast, such as specific data points and / or graphics displayed in the broadcast feed. Criteria can encompass prioritizing players, shots, holes, or any way of a scenario, including criteria based on metrics from the broadcast or social media indicating, for example, that a particular player, combination of players, or shot or hole is subject to social media interest trends or increased betting activity. Highlights can be generated for particularly important or well-executed shots. Shots can be identified as shots based on a measure of how good a shot is, for example, the well-known "stroke count" metric. Therefore, shots with particularly high or low stroke values ​​can be included in the highlight reel. Furthermore, all shots from the current leader (e.g., 5 or 10 players) can be included.

[0162] In 1664, custom broadcast feeds are generated according to predefined rules, including custom feed criteria. For example, a third-party broadcast can be used in a custom feed broadcast when the custom feed criteria allow it (e.g., when the player of interest is not currently taking a shot). In some embodiments, multiple players of interest are identified by the custom feed criteria, and the video feed is switched between players until an event is detected that triggers, for example, a switch to broadcasting to a specific camera.

[0163] In 1666, events are detected that trigger a switch to another camera in the broadcast feed. These events include: players detected and identified by a particular camera unit; identified players located near the launch area (e.g., the tee box); identified players in their hitting stance; and others; events of these types related to pending shots that are about to be hit. In some embodiments, a neural network can be used to detect these events related to the upcoming shot, as described in more detail below with respect to Figure 9. Other types of events may include determining the current 3D position of the ball in flight (after launch), which is used to determine whether to switch to another camera, such as a camera with an FOV that covers the fairway and / or green, which can provide a better field of view than the current broadcast camera, as described above with respect to method 1650 in Figure 16d.

[0164] In 1668, the broadcast feed switches from the current camera to another camera based on event detection. The camera and / or radar unit detecting the event may be different from or located separately from the camera being switched to.

[0165] Automatic tracer for all shots Figure 8a shows a first image 800 from a video feed depicting a shot, including a shot trajectory superimposition, i.e., a first tracer 805 of the shot and some associated trajectory data. In this example, the first image 800 is captured by a tee box tracking unit, which includes radar and at least one camera. The first tracer 805 can be automatically generated within the image provided by the camera in the tracking unit, since, as described above, all pixel orientations in the image are known to the radar's coordinate system. A superimposition like the one shown in Figure 8a can be generated even if the tracking unit is not calibrated to the world coordinate system. Figure 8b shows a second image 850 from a video feed depicting a shot, including a shot trajectory superimposition, i.e., a second tracer 855 of the shot and some associated trajectory data. In this example, the second image 850 is captured by a green-side tracking unit. For example, a similar graphic superimposition including the first tracer 805 or the second tracer 855 described above can be inserted into an image from any camera, as long as the camera is calibrated to the radar tracking the shot. The tracer can be inserted into any image, such as a video feed of an external broadcast, even if the radar unit tracking the ball is not located in the same tracking unit as the camera used for the external broadcast, as long as the radar coordinate system of the radar tracking the ball trajectory is calibrated relative to the camera. Additional information such as ball speed, carry, and curve can be further tracked and displayed as numerical values, as shown in Figures 8a-8b.

[0166] The ball's position, determined solely based on radar tracking, may contain some uncertainty until more advanced aerodynamic models are fitted to the data. To provide a visually pleasing tracer, it is crucial that the tracer starts in the image at a position precisely correlated with the ball's launch location. Therefore, it is preferable to detect the ball's position as accurately as possible in one or more images before the ball is launched.

[0167] A ball identification algorithm can be used to determine the ball's position. Once a player is identified by the camera and it is determined that they are ready to hit a shot (for example, based on their identified posture), the ball identification algorithm searches for the ball's exact position in the image. This position can then be used to determine the tracer's starting position. If this data is unavailable (for example, if the ball is obscured by deep rough), the system may need to extrapolate from the ball's initial detected position in flight to a launch position that can still be corrected using the player's stance as a general marker of the original ball position. Thus, once the tracking system detects a shot and outputs trajectory data, this data can be automatically displayed in the broadcast as a tracer with little to no human input.

[0168] Finding the ball's position can also involve analyzing player movement to detect the player taking their stance over the ball (as is always done before hitting a shot) with the ball positioned near the clubhead before the swing begins. Alternatively, a neural network could be trained to search for and recognize the ball by looking for its position near the clubhead while the player is taking their stance before hitting the ball.

[0169] Figure 16f shows a method 1670 for the automatic insertion of a tracer into a broadcast feed by various exemplary embodiments. A system for carrying out the method 1670 comprises a radar tracking unit that provides a 3D position track of a launched ball, and a broadcast camera that provides a video feed of the ball launch, wherein the broadcast camera (or a broadcast entity upstream of the actual camera) is configured to insert 3D graphics into the broadcast feed, and the 3D graphics include the broadcast camera which includes at least a tracer. However, in other embodiments, a tracking camera (separate from or the same as the broadcast camera) may be used to provide the 3D position track.

[0170] In 1672, the radar coordinate system of the radar tracking the ball's trajectory is calibrated against the broadcast camera. The broadcast camera can be calibrated against the radar in any of the previously discussed methods. For example, for a mobile and / or robot tracking unit (including either or both of the tracking unit and the broadcast camera (same mounted)) to pull a tracer into the feed from the broadcast camera, only calibration of the broadcast camera to the tracking unit coordinates is required. For camera tracking units, as described above, the relationship between the intrinsic parameters of the tracking unit's camera and the broadcast camera can be established for different zoom levels.

[0171] In any 1674, the ball's position is precisely detected in one or more broadcast images before the ball is launched. This optional step can improve the rendering of the tracer starting point in the image by determining the position in the image that precisely correlates with the position from which the ball is launched. A ball identification algorithm can be used to search for the ball's precise position in the image and then determine the position of the tracer starting point. In another embodiment, the player's stance may be used to locate the ball, and / or a neural network can be trained to search for and recognize the ball by searching for its position near the club head when the player takes their stance before hitting the ball.

[0172] In 1676, a tracer is generated for the broadcast video feed based on trajectory data captured by the tracking unit. If information about the exact hole position before launch is unavailable (e.g., the ball is obscured by deep rough), the system may need to extrapolate from the ball's initial detected position in flight to a launch position that can still be corrected using the player's stance as a general marker of the original ball position. The broadcast camera can zoom to any zoom level independently for each stroke without requiring the tracking unit to read the zoom level from the broadcast camera (in scenarios where baseline calibration is performed and intrinsic parameters are mapped to different zoom levels, the current parameters can be determined from similar detections from the tracking and broadcast cameras). In some embodiments, a first tracer is generated for a first video feed from a first broadcast camera (e.g., located at the tee box) and can follow the ball during the first part of its trajectory (e.g., its initial trajectory), while a second tracer is generated for a second video feed from a second broadcast camera (e.g., located on the fairway or green) and can follow the ball during the second part of its trajectory (e.g., its descent).

[0173] Live video can be automatically zoomed in with graphics such as tracer and lower third graphics (e.g., ball speed, highest point, carry, curve, spin rate, etc.). Furthermore, other graphics can be automatically added to display, for example, the player names for this hole, shot number, hole number, current score, and other lower third graphics seen in live sports broadcasts.

[0174] In this way, fully automated broadcasts can be generated. The video of each shot can be clipped and trimmed to precisely include the relevant period based on ball tracking. Furthermore, each clip can be tagged with the player's name (or player ID), the shot number for this hole, the hole number, etc. Clips can be saved with and / or without tracer information and / or other graphics. This database of clips will, in principle, contain all shots from all players in any tournament.

[0175] Automatic event detection A neural network can be trained using live video streams from tracking unit cameras for specific applications in particular golf scenarios (e.g., tournament scenarios). The network can be applied to any camera stream and is particularly useful for tracking unit cameras located behind the tee boxes. In this case, the network outputs detections of events such as when a player and / or other person steps onto the tee, when a player prepares to hit the ball, and when a shot is struck. These detections can be used to automate event generation useful for on-site staff, broadcasters, and betting applications. Detected events may include: a player on the tee; a player preparing to hit a shot (addressing the ball); a player swinging; a shot being struck; and a tee box being cleared.

[0176] Figure 9 shows exemplary images 900–915, which can be analyzed by a neural network to detect events related to a player's swing. In 900, the player is detected in the image before hitting the shot. The neural network has not yet detected any shot-related events. In 905, the player is detected as approaching the ball. In 910, the player is detected as swinging. In 915, the player is detected as hitting the ball.

[0177] When developing an algorithm or network for recognizing golfers, the neural network can consider various factors when detecting events. One factor to consider is that, generally, only golfers hold a golf club and simultaneously position themselves in the address position to hit a golf shot. The specific angle of the golf shaft at address is also unique and can be used. Furthermore, the relative positions of the ball and clubhead are good indicators that can be used to identify a player who is addressing the ball and about to begin their swing. Using techniques similar to those used to identify individual golfers, people moving with the golfer may be excluded from such identification. For example, a caddy and their clothing may be identified and stored in the system so that such individuals are not identified as golfers, even when they are holding a club and / or standing over the ball.

[0178] Event detection can also be used to more efficiently manage power usage across different sensors and subsystems of a tracking system. When an event requiring a particular system capability is not currently detected in the tracking unit's field of view or other system components, the item's power usage can be reduced, for example, by putting the various power-drawing components of the item into sleep or low-power mode until the system detects an event requiring the item to fully power up or an immediate pending event. This can have significant practical importance, as golf course systems are often powered either by batteries or generators.

[0179] In a further embodiment, the detection of events related to the upcoming shot, such as the player's approach to the ball or the formation of a posture on the ball, can be used to predict when the upcoming shot will occur. This can be of great practical importance in live betting applications, such as betting on the distance of the upcoming driver shot or the distance from the pin. For example, if the system estimates that the upcoming shot will be executed within, say, 5 seconds, the live betting window for that upcoming shot can be closed depending on the estimated shot time. Existing methods typically rely on manual triggers to close betting windows.

[0180] In the case of live betting applications, the tracking system may be designed to prevent tampering or to detect tampering, for example, by delaying the live feed of shots so that bets are placed after the actual shots have been taken, thereby giving the sensors an advantage in betting.

[0181] Controlling a robot camera from real-time 3D tracking data The tracking system tracks a moving ball with very low latency, constantly updates the predicted future trajectory of the ball, and allows the system to control other cameras in real time. Using real-time positional data of the ball, the tracking system can control a robotic camera to orient this camera towards the current ball position (tracking the ball throughout its flight), and additionally control zoom and focus to ensure a clear, viewer-friendly image that viewers are accustomed to, in the same way that professional camera operators achieve during golf tournaments. The robotic camera described herein may correspond, for example, to the broadcast camera 315 shown in Figure 3. However, the robotic camera may also be used primarily for tracking purposes, rather than as a broadcast camera.

[0182] The 3D position of the ball relative to the robot camera is always known by the tracking system based on real-time data acquired by other sensors in the tracking system. Therefore, the optimal crop, orientation, and zoom level for the robot camera can be automatically controlled by the system based on real-time data. Optical tracking of the ball in flight may also be performed using analysis of data from the robot camera (or any other camera), and may be used to control the robot camera to achieve the desired image using any known method. The advantage of the proposed system is that the zoom and focus of the robot camera can be continuously updated based on the tracking of the ball in flight by other sensors or other tracking units, even if the ball is not visible or easily found in the image from a particular camera (e.g., when a white ball disappears into the background of white clouds). That is, controlling the aiming and focus of the robot camera based on three-dimensional tracking data may also be used to enhance optical tracking using images from the robot camera, ensuring that the sharpest possible image of the ball is captured and increasing the likelihood of detecting the ball in the image. Without knowledge of the ball's 3D position relative to the robotic camera, detection is often impossible or delayed as the optical tracking system attempts to reacquire the ball whose position it cannot locate (for example, when the camera is focused at a distance that does not correspond to the ball's current position, the ball may be lost against a background of similar color, or its line of sight may be obstructed).

[0183] In some scenarios, the robot camera may be positioned within a tracking unit that includes additional sensors, such as radar or lidar, and the entire tracking unit changes its orientation based on the camera's tracking capabilities. This change in orientation can improve tracking of the additional sensors in addition to tracking of the robot camera. For example, some shots may have unexpected trajectories (e.g., poor shots) that are not adequately covered by the sensor's initial field of view. When these shots are detected by the tracking camera, the orientation of the entire tracking unit, including the additional sensors, may be changed to provide, for example, improved sensitivity to radar.

[0184] Robotic cameras can be used to replace manual camera operators and can make video footage far more engaging for viewers. They can also create close-up shots, for example, to depict the lie of a ball. To ensure stable and viewer-friendly movement of the robot camera, a special filtered version of real-time 3D data from the tracking system can be generated and used to control the robot camera. Furthermore, knowledge of robot camera control characteristics such as delay, maximum angular acceleration, maximum angular velocity, and focus delay can be considered and supplemented when controlling the camera. The robot camera can also be used as a tracking sensor for positioning clubs and balls in world coordinates. However, this requires calibration of the robot camera because the field of view is not stationary.

[0185] Calibration of robotic cameras To use a robot camera as a tracking sensor, it needs to be calibrated. Calibration involves two parts: extrinsic calibration and intrinsic calibration. As mentioned above, extrinsic parameters essentially involve two parts: determining the camera's position in world coordinates and its orientation in world coordinates. The determination of the robot camera's position is performed in a manner similar to that employed in the tracking unit and / or broadcast camera described above. What remains is the determination of the orientation in world coordinates and the intrinsic parameters.

[0186] Figure 10a shows an exemplary image 1000 captured by the robot camera 1005 to determine the orientation of the robot camera 1005 in world coordinates 1020. Image 1000 is shown in a two-dimensional (u,v) coordinate system local to the camera 1005. Ball 1010 is the pixel coordinate (u,v) of the image. b Pixel (u,v) at world coordinate 1015 is located within image 1000. b There are various methods for determining the position of 1015.

[0187] According to the first option, pixel (u,v) bThe position of 1015 can be determined in world coordinates 1020 by pre-calibrating the intrinsic parameters (e.g., focal length, principal point, lens distortion, etc.) for each different zoom level used by the robot camera 1005. The orientation of the robot camera 1005 in world coordinates 1020 is determined, for example, by using the pan and tilt offsets of the camera 1005 orientation when viewing the ball 1010, for a given orientation of the robot camera 1005 that has been calibrated to world coordinates in a manner similar to that described above for the tracking unit camera and / or broadcast camera, and is determined from the camera's pan / tilt sensor or from the tracking-based control signals applied to the camera orientation. That is, the system is determined based on pan / tilt information. A three-dimensional line and a point p in world coordinates are described by vector R, where p is the position of the camera. This vector R extends in the direction from the camera (e.g., from the camera's focal point, point p) toward the ball (or any other object whose position is determined). By knowing this line from the camera in three dimensions, the three-dimensional position of a ball (or other object) on the line described by vector R and point p can be obtained by combining the data of this line with the distance information of the ball (radar or other information in world coordinates or any other coordinates that can be translated into a coordinate system relative to the camera), or by determining where the line described by vector R and point p intersects with the surface of the golf course as displayed in a three-dimensional model of the golf course. This makes it possible to determine the three-dimensional position of the ball along the line described by vector R and point p in three dimensions in world coordinates. Those skilled in the art will understand that by identifying the intersection of the line described by vector R and point p and the three-dimensional model of the golf course surface, it is possible to determine only the position of a ball stationary on the surface of the golf course. Further options described later describe different ways of identifying this vector R so that this information can be used together with distance information or three-dimensional surface model information to determine the position of the ball in three dimensions in world coordinates based on the pixel position of the image from the camera.

[0188] According to the second option, the position (u, v) of the pixel b 1015, in world coordinates 1020, first detects the position (u, v) of the pixel in the image 1000 in local (u, v) coordinates b 1015, and then, without changing the zoom level, can be determined by changing the orientation of the robot camera 1005 so that the robot camera 1005 faces a predetermined reference position 1025, where the reference position 1025 is the same image (u, v) where the ball 1010 is located within the image 1000 b Ensure that it is located at the position within.

[0189] Figure 10b shows a process for determining the orientation of the robot camera 1005 based on the first image 1000 of Figure 10a and a second image 1030 captured by the camera 1005 with a different orientation. Similar to Figure 10a, the ball 1010 is located within the first image 1000 at pixel coordinates (u, v) b 1015. The camera 1005 changes its orientation so that the reference position 1025 (whose position is known in world coordinates) is located at the same coordinates (u, v) b 1015 in the second image 1030 as the coordinates (u, v) b 1015 in the first image 1000. Thereby, the system can identify the line described by the three-dimensional vector R from the camera to the ball in the first image 1000

[0190] Next, the pan and tilt offsets between the two scenarios are used to calculate the vector R (the three-dimensional orientation of the camera in world coordinates 1020 when looking at the ball 1010 compared to the reference position 1025). An obvious advantage of this method is that the intrinsic parameters of the camera 1005 do not affect the accuracy of the determination

[0191] According to the third option, the position of the pixel (u, v) b 1015 is first the pixel (u, v) of the first image 1000 in local (u, v) coordinates bThe position of 1015 can then be detected and then determined in world coordinates 1020 by detecting the same position in a second image 1040 from the tracking unit camera 1035. In a preferred embodiment, the robot camera 1005 and the tracking unit camera 1035 are located in the same place, but this is not mandatory.

[0192] Figure 10c shows the process for determining the orientation of the robot camera 1005 (three-dimensional vector R) based on the first image 1000 from Figure 10a and the second image 1040 captured by the second camera 1035. The second camera 1035 may be a tracking unit camera calibrated to the world coordinate system 1020. The first image 1000 from the robot camera 1005 may be placed within the second image 1040 from the tracking unit camera 1035. One option for finding the first image 1000 within the second image 1040 is to perform feature matching between the two images 1000, 1040 using an identified feature 1045 whose location is known in world coordinates (e.g., trees, bunkers, ponds, golf flags, green contours, and other unique features that can be identified in the image). Since the position of the second image 1040 is known in world coordinates 1020, it can be transferred to the first image 1000, thereby enabling the determination of the world coordinate position of the ball 1010 in the robot image 1000 using the distance and / or a three-dimensional model of the golf course, as described above, through a combination of vector R, the camera position p in world coordinates, and distance information.

[0193] According to the fourth option, pixel (u,v) b The position of 1015 is, first, the pixel (u,v) in the first image 1000 in local (u,v) coordinates. b By detecting the position of 1015 and then zooming out the first image 1000 (for example, generating a second image 1050 that includes a zoomed-out version of the first image 1000), at least two reference points 1060, e.g., 1060a and 1060b, can be determined in world coordinates 1020 so that they are visible in image 1050.

[0194] Figure 10d shows the process for determining the orientation of the robot camera 1005 based on the first image 1000 of Figure 10a and the second image 1030, which includes a zoomed-out version of the first image 1000. Similar to Figure 10a, the ball 1010 has pixel coordinates (u,v) b Point 1015 is located within the first image 1000. Camera 1005 zooms out so that reference point 1060 is visible in the second image 1030, while the coordinates (u,v) of the second image 1030 are... b The pixel position at 1015 is the coordinate (u,v) of the first image 1000. b This is the same as 1015. The reference point 1060 is detected at several pixel positions 1055, e.g., 1055a and 1055b, in the second image 1050, while tracking the position 1015 of the ball's position in image 1000. Assuming slight lens distortion, only the determination of the focal length is necessary. The focal length is determined by correlating the reference point position 1055 with a predetermined angle in world coordinates 1020 of the two corresponding reference positions 1060. Thereafter, the position of the ball 1015 in the zoomed-out image 1050 can be determined in world coordinates 1020.

[0195] Figure 16g shows Method 1680 for calibrating a robotic camera to a world coordinate system in various exemplary embodiments. A system for carrying out Method 1680 may include only a robotic camera and a processing unit for controlling its operation. In some embodiments, the system may include additional cameras.

[0196] In 1682, the robot camera is provided with uncalibrated intrinsic parameters. For example, the robot camera may have recently been powered on or recently had its orientation changed from its previous orientation.

[0197] In 1684, the first image (e.g., a calibration image) is captured by the robotic camera, and the ball is detected in the image. The (u,v) coordinates of the ball are determined by the camera coordinate system.

[0198] In 1686, the second image is received from either a robotic camera (using different orientations and / or different zoom levels) or a second camera that simultaneously captures a second image of the same ball.

[0199] In 1688, the orientation of the robot camera is determined based on the first and second images.

[0200] Safety / warning system for on-site spectators Real-time ball flight tracking, as mentioned above, can update and predict the landing point in real time while the ball is in the air. This landing point prediction can be used, for example, in an automated "forehand" to warn spectators that a misfire is approaching. Spectators in the affected area can then be given appropriate instructions, such as asking for cover or protecting their heads.

[0201] The alarm system may include, for example, wired or wireless georeferenced speakers that can be triggered by a tracking system. Preferably, the golf course is divided into different spectator zones. Fore-warnings may be triggered in one or more spectator zones whenever there is a sufficient probability that a mis-shot will approach these zones. The alarm system may also be a personal application on a smartphone that tracks the location of each spectator (or any spectator who has opted in to the service). Special warnings, such as vibration / sound or similar, may be triggered by the smartphone application when there is a high probability that a mis-shot will land in a zone where a spectator is located.

[0202] Figure 16h shows Method 1690 for a warning system for on-site spectators by various exemplary embodiments.

[0203] In 1692, orbital data is acquired using one or more primary sensors.

[0204] In 1694, the landing point is predicted based on the trajectory data captured so far. The landing point prediction is initially performed as early as possible during the ball's flight and can then be refined as more trajectory data is captured. The landing point prediction may include a specific point, and the warning area may have a predetermined uncertain radius relative to the point, or include an area based on the actual estimated uncertainty.

[0205] In 1696, an alarm is triggered and delivered, for example, to any interested party within the alarm radius. In one embodiment, an auditory alarm can be provided by activating a speaker located near the predicted landing site. In another embodiment, the alarm may be transmitted to devices of spectators located at the landing site.

[0206] Amateur round tracking All the features of the system described above can be implemented on any golf course. Ball trajectory tracking is entirely independent of shot quality and can be applied to amateurs as well as professionals. Furthermore, amateur rounds may be conducted in a manner similar to the customs and rules governing professional golf tournaments, and therefore the competition theory previously discussed for professional events can be similar to that for amateur rounds. For example, typically a group of 1 to 4 players might play hole 1 first, then hole 2, and so on. Thus, the detailed player ID tracking described above can be implemented in a substantially similar manner.

[0207] One aspect of professional event tracking that might not be available in amateur rounds is the pre-stored information about the identities of players in a given group. Several additional features can be used to associate golf round tracking with specific golfers.

[0208] In one embodiment, the application connected to the tracking system can be installed, for example, on a smartphone or smartwatch. Each golfer can provide a positive indicator within the application, which can be used to associate the tracking system player ID with the golfer, for example, at the start of a round. There may be a special area around the first tee, for example, a small, marked area on the ground that has a space for just one person, where a player can synchronize their smartphone with the tracking system. For example, a user can associate their name and history with a unique player ID in the tracking system by pressing a "sync" button in the application while in the special area. In another embodiment, tracking and association of player IDs can also be done via some kind of transmitter, for example, a smartphone or smartwatch with location-based capabilities.

[0209] In another embodiment, a device worn by the player, such as a smartwatch, can be used to detect the player's swing motion. The device may be associated with the player and may include an accelerometer (or other sensor capable of detecting such motion) and processing logic for detecting changes in acceleration corresponding to a golf swing. The device may also have GPS capabilities (or be linked to another nearby device with GPS capabilities, such as a smartphone) so that it can also detect the player's approximate location at the time of the swing. The device may also be operable to associate the swing motion with a timestamp. Thus, when the tracking system detects a shot (and the time of the associated shot), the shot can be matched with a detected swing that matches the timing and location of the shot. Particularly in amateur round tracking, there is no requirement that the matching between swings and shots be performed in real time. Therefore, a swing detected by the device (and associated timestamp) can be matched in time with a shot detected later by the tracking system, for example, at the end of the round. This is particularly useful in scenarios where the device worn by the player has limited capabilities, for example, with respect to wireless reception range, processing power, etc. The device can store the swing detection along with the associated timestamp until a later time when the swing data can be associated with the tracking data. In one embodiment, a player associated with the device can review shot information related to the detected swing at the end of a round and verify the accuracy of the match. In an alternative embodiment, the player carries a location-determining device, such as a mobile phone. The device records the trajectory of the device's world coordinates with associated timestamps and can then match this to the determined launch location of the shot, which is tracked by the tracking system. The location-determining device does not need to be worn by the player but may be in the player's bag, always near where the player launches the ball.

[0210] Player image profiles may be stored in the tracking system for future golf rounds, or they may be created before each golf round. Each tracking system across multiple courses may potentially share these image profiles depending on the circumstances. After each round is played, the captured data may be transferred to a personal application, and each round can be reviewed with all tracking data for post-round analysis across golf rounds.

[0211] In one embodiment, ball tracking data may be transferred to a personal application immediately after it is recorded, allowing the player to review the data during a round of golf. One particularly interesting application of this feature could be to help a golfer find their ball. For example, when a golfer hits a shot that is difficult to find, the personal application can narrow down the likely locations where the ball might be found on a golf hole map provided in the application or other map applications (e.g., Apple Maps or Google Maps). This feature can be particularly useful when trying to find a ball in deep rough, as narrowing the search area significantly increases the chances of finding the ball and also reduces the time required to find it.

[0212] When a golfer hits a shot that lands in a hazard, depending on the golf rules for that particular hazard, the golfer may be given the option to drop a new ball according to where the previous shot crossed the hazard line. Because the system has a track of the ball's trajectory, thereby accurately determining where the ball crossed the hazard line, the system can precisely determine the area where a new ball can be dropped. This can resolve the sometimes controversial decision of where a ball can be dropped according to the rules, saving golfers time and constraints while achieving a fairer outcome. Identification of the permissible drop area may be shown in a software application that displays the layout of a hole with a permissible drop area.

[0213] Generation of Nonfungible Tokens (NFTs) The proposed system can automatically film and track any special events that occur during a round of golf, which could be a hole-in-one or any other specific, unique golf shot or event. The system can be configured to automatically detect these moments and create video clips that may include ball tracer, club, and / or ball trajectory information. These clips can be created in a way that only one digital copy can exist, thereby allowing them to be used as non-fungible tokens (NFTs).

[0214] Introduction to non-camera projection applications For example, for tracking a shot from a tee box or on the green, the tracking system may include radar, a combination of radar and cameras, or a combination of several additional sensors. These different sensors can track different and / or overlapping portions of the golf ball's flight. Each sensor acquires object data consisting of ball measurements one or more times in a device-specific coordinate system, and when calibrated with each other, the data can be projected into a highly consistent world coordinate system for further processing and data fusion. Doppler radar capable of angular and absolute range measurement can accurately track the ball in three dimensions; tracking units equipped with radar and cameras can use image tracking to enhance angular measurements for more accurate tracking; and systems including multiple cameras with overlapping fields of view can use stereoscopic images to determine the ball's position in three dimensions.

[0215] Accurate measurement requires sensor calibration. Sensor calibration involves determining various parameters, both internal and external. Internal parameters may include focal length, lens distortion parameters, and principal point for cameras and phase offset for radar. External parameters typically constitute the sensor's position and orientation.

[0216] Known systems based on Doppler radar tracking have drawbacks, for example, when: the ball is moving slowly or rolling along the ground; and the radial velocity relative to the radar is v r When the ball moves at a specific angle relative to the direction from the radar to the ball, the angle becomes low; when the ball is stationary and Doppler radar tracking completely fails. Known systems that rely on stereoscopic images to track a ball can determine the three-dimensional trajectory of the ball even at low speeds, but without a detailed model of the golf course, they generally cannot provide metrics relevant to understanding a short game of golf.

[0217] Sensor intrinsic parameters may generally be defined as parameters related to the sensor's internal operation. Camera intrinsic parameters are those necessary to link the pixel coordinates of image points to the corresponding coordinates of the camera reference frame, and include parameters such as focal length, principal point, and lens distortion. Camera intrinsic parameters may be measured in the production setup and stored in the system, and methods for determining intrinsic parameters in a laboratory setting are considered to be known to those skilled in the art.

[0218] Sensor external or extrinsic parameters can generally be defined as parameters related to the positioning of the sensor (e.g., position and orientation) relative to world coordinates. Knowledge of sensor external parameters allows for mapping measured data from the sensor's coordinate system to the world coordinate system so that it can be compared with data from other sensors or the position of a three-dimensional model of a part of a golf course.

[0219] Extrinsic parameters can be determined in several ways, as will be understood by those skilled in the art. A preferred method of calibration is to measure the GPS position of the camera and several reference points on the golf course within the camera's field of view. The calibration image is captured by the camera, and the positions of the measured reference points in the image plane are determined by human annotation or automatic detection means, allowing the extrinsic parameters to be calculated. Easily recognizable features that may be used in calibration include, for example, the position of the pin (hole), the edge of a bunker, or an object temporarily or permanently placed on the course for calibration. The GPS position of the reference points can be supplemented by measuring the distance and / or height difference between the reference points and the camera, for example, using a laser range finder or similar instrument.

[0220] The tracking camera according to this embodiment for tracking the bounce, roll, and / or stationary position is typically installed behind the green and mounted on a TV tower or similar structure, with the camera's field of view from that position including the green and surrounding area. The camera can also be positioned along the fairway on a long par-5 hole to capture the landing position of the ball on the fairway. The tracking function according to this embodiment can be implemented using a single camera and does not require multiple cameras utilizing, for example, stereoscopic imaging technology.

[0221] In one embodiment, the lie of the ball is determined based on images from a single camera of the ball when it is stationary and from a 3D model of the terrain at the ball's stationary position.

[0222] In some embodiments, a system and method are provided that uses a single camera and an accurate 3D model of a portion of a golf course to track and provide tracking metrics for bounce and chip shots, and the single camera is calibrated against the portion of the golf course covered by the 3D model by capturing a series of images with the camera, detecting the ball in the images, deprojecting the pixel positions of the ball detection to generate lines in 3D space from the camera to the ball positions, determining start and end lines corresponding to the start and end points of the trajectory, intersecting the 3D model with the start and end lines, and applying a physical model to determine the ball's track.

[0223] In one embodiment, it is determined from a series of images whether the ball is bouncing or rolling.

[0224] In one embodiment, 3D tracking of the ball's roll is provided by detecting the ball in a series of images and deprojecting the pixel positions of the ball detection, thereby generating lines in 3D space from the camera to the ball's position, intersecting a 3D model with these lines, and determining the ball's track.

[0225] In one embodiment, the stimpmeter on the green is determined from the putt track and terrain slope in a 3D model.

[0226] In one embodiment, the putt break fan is determined for a given initial ball position and compared with a measured 3D putt trajectory to determine the sensitivity of the ball launch parameters to a successful putt. The putt break fan is described in more detail below.

[0227] In one embodiment, radar tracking of a golf ball is used to guide a ball detector to a suitable image and a suitable search area within that image in a timely manner.

[0228] In one embodiment, an image is captured by a camera, and it is determined whether the image contains a person and / or whether the person is a golfer in an address position. Based on this detection, the ball detector is guided in time to the appropriate image and the appropriate search area within the image.

[0229] In one embodiment, the camera's external calibration parameters for the 3D model are updated based on jump tracking.

[0230] 3D surface model of a golf course or part of a golf course According to various exemplary embodiments described herein, a three-dimensional model of an entire golf course, or a specific portion of a golf course (e.g., one or more greens and the area immediately surrounding these greens), can be used to improve and enhance shot tracking within the area covered by the 3D model.

[0231] A 3D model is a representation of a golf course (or a portion of a golf course) stored in computer memory and can be used to render 3D graphics of the golf course. A 3D model of a particular golf hole preferably covers a portion of the area within the field of view of a single camera used to track the golf ball according to this embodiment. Thus, the 3D model covers at least the terrain around the green where a shot from the tee or fairway is likely to land. The model may also cover part or all of the fairway and the surrounding rough and / or semi-rough, or it may cover the entire hole. Multiple 3D models may be used to cover the entire golf course where multiple cameras are used, each covering a portion of the course, or a single 3D model covering the entire course may be used. Those skilled in the art will understand that any reference to world coordinates can be applied to a coordinate system common to all holes on a golf course, or to the area of ​​a single hole on a golf course. The use of a world coordinate system allows for the fusion of measurements from different sensors and enables the mapping of ball position measurements onto a 3D model. World coordinates do not need to be common to all sensors covering a round of golf. In other words, the desired result is obtained as long as all sensors tracking the flight of any given shot can translate their data into a common coordinate system. Therefore, if a group of sensors covers only hole number 1 and hole number 2, these sensors do not need to be able to translate data from sensors on hole number 2 into a common coordinate system. Thus, the use of the term "world coordinate system" only suggests that this system correlates with the physical environment of the sensors, and does not imply that this coordinate system needs to be common across all sensors operating on the golf course.

[0232] The 3D model includes at least a surface model representing the elevation of the terrain at a given location on the course. This surface may be represented as a triangular mesh, as a spline surface, or as another representation well known to those skilled in the art. The surface may also be represented by a coarse model that provides a coarse representation of the course elevation, in combination with one or more fine models that provide a more detailed surface map of some or all of the features represented by the 3D model, with the fine models representing offsets to the coarse model. The coarse model may be provided for a given geographical area, while the fine model may be provided only for parts of the area covered by the coarse model, such as where play most frequently occurs or where additional surface precision is desired. These areas may include the fairway and surrounding semi-rough, bunkers, greens, and the areas surrounding the greens. This scheme, including a coarse model and one or more fine models, may be advantageous by avoiding the creation and processing of a detailed model of the entire course.

[0233] In a preferred embodiment, the 3D representation of the golf course is given in world coordinates so that GPS locations can be easily mapped onto the 3D representation. The 3D representation may also be provided in a local coordinate system and associated with a well-known mapping to world coordinates. The ability to map GPS, and / or a world coordinate system, to the 3D model allows for the association of camera extrinsic parameters with both the golf course and the 3D model, thus enabling the determination of the position of the ball on the ground by capturing an image with the camera, detecting the pixel position of the ball in the image, and applying intrinsic and extrinsic camera parameters to deproject the detection to determine the camera-ball line in a coordinate system such as a world coordinate system, and to determine the intersection of the camera-ball line with the 3D model.

[0234] In one embodiment, the 3D model includes representations of terrain types present on a portion of the golf course represented by the model. For example, each triangle in the mesh may have an associated terrain type, or the terrain type may be represented in a fine-grained height model. The terrain types may include at least the features of fairway, rough, semi-rough, bunker, green, and fringe, but may include more features for a more detailed representation of the various terrains found on the golf course. These features can be used to improve tracking of the ball's trajectory, which is affected by these terrains, as described below. The model may also include representations of non-terrain features such as trees, bushes, and buildings, which can provide realism to any graphic rendering based on the 3D model, although these non-terrain features are generally not used for any tracking purposes.

[0235] 3D models can be obtained, for example, by drone scanning using LiDAR, optimetry, or a combination thereof, or by other methods known to those skilled in the art. The scan can yield a point cloud from which a 3D surface model can be created using techniques well known to those skilled in the art. The relationship between the course's local coordinate system and world coordinates can be established by mapping several fixed points identifiable in the 3D model using accurate GPS measurements. For example, the PGA Tour has detailed maps of the golf courses used for play on the tour.

[0236] A system that determines the ball's position using a non-projector. Figure 11 shows an exemplary tracking system 1100, which includes at least one camera 1105 that captures a series of images of a ball, including a stationary ball, according to various exemplary embodiments, and a processing unit for tracking the ball and / or determining the stationary ball's position. The processing unit includes a ball detector 1110 that determines the ball's position in an image coordinate system, a non-projection module (non-projector) 1115 with intrinsic and extrinsic camera parameters 1130 for determining the camera-ball line in 3D space from camera 1105 to the ball, and an intersection module (intersector) 1120 with a 3D model 1135 of a portion of a golf course for determining the intersection of the camera-ball line and the 3D model 1135. The processing unit further includes an output generator 1125 for outputting information determined for use, for example, during broadcast. Figure 11 is described below in conjunction with Figure 12.

[0237] Figure 12 shows an exemplary figure 1200, which includes a camera 1105, a golf course terrain 1205, and a corresponding 3D model 1135 superimposed on the golf course terrain 1205, according to various exemplary embodiments. Figure 1200 includes the trajectory of a golf ball 1210, which includes the ball's collision point 1215 and final resting position 1220, and an exemplary camera-ball line 1225 that intersects the 3D model 1135 at the ball's final resting position 1220.

[0238] Camera 1105 has sensors and lenses selected to provide a field of view (FOV) of a portion of the golf course, such as the green and its surroundings, including the fairway, in the direction from which a shot is most likely to be struck, and which is tracked using the tracking system 1000. Alternatively, if camera 1105 is positioned to capture shots landing on the fairway, the FOV of camera 1105 can cover the portion of the fairway from which a shot from the tee or an approach shot to the green is most likely to land, such as the terrain 1205 shown in Figure 12. The resolution of camera 1105 is selected to provide enough ball pixels for the ball detector 1110 to operate efficiently. For example, the minimum cross-section of a ball seen in the captured image should be in the range of 3 to 10 pixels. The camera captures a series of images 1190(...,i (n-1) ,i (n), i (n+1),... ) to a specific frame rate f s The capture is preferably performed at a rate of 30 to 100 frames / second. In a preferred embodiment, the captured image i (n) This is at least the time t when the image was captured. n This includes metadata, which may also include exposure time for the image, the cropped area of ​​the sensor captured in the image, and potentially other parameters.

[0239] Once captured, each image from the series of images 1190 captured by the camera is passed to the ball detector 1110, which includes one or more algorithms for determining whether a ball is present in a particular image 1195 and the pixel location (u,v) of the ball. The algorithms may include a convolutional deep neural network (CNN) trained to detect golf balls in images. If a ball is detected in the image, the ball detector 1110 transmits the (u,v) coordinates, and optionally the image itself and some metadata, to the non-projector 1115.

[0240] The non-projector 1115 receives information from the ball detector 1110 that includes at least the (u,v) coordinates of the ball and reads the stored intrinsic and extrinsic calibration parameters 1130 from memory or an external storage device. Using this information, as shown in Figure 12, the non-projector 1115 determines a camera-ball line 1225 in 3D space that includes a straight line passing through the focal point of the camera 1105 in the direction of the ball, based on the (u,v) coordinates of the image plane and the intrinsic and extrinsic parameters 1130 of the camera. If the captured image does not cover the entire field of view of the camera 1105, the non-projector 1115 also uses information about the cropped area of ​​the image to determine the camera-ball line 1225. The cropped area may be part of the image metadata passed from the camera 1105 or the ball detector 1110, or, if constant for the entire series of images, it may be stored in the system 1100 during calibration or system initialization and made accessible to the non-projector 1115 in a similar manner to the stored calibration parameters 1130. The non-projector 1115 transmits the camera-ball line representation to the intersection module 1125 1120.

[0241] The intersection module 1120 executes an intersection algorithm based on the camera-ball line 1225 and 3D model 1135 of at least a portion of the golf course that overlaps with the camera's FOV. The intersection module 1120 receives the camera-ball line 1225 from the non-projector 1115, reads the 3D model 1135 from memory or external storage, and determines the intersection 1220 between the camera-ball line 1225 and the 3D model 1135. The 3D point representation of the intersection 1220 is used to determine the ball's position on the course. When the ball is stationary, this intersection 1220, which can be represented as a 3D point in space, is the ball's position on the course and the elevation of the terrain at the ball's position. The method for determining the intersection may depend on the representation of the 3D model, e.g., whether the 3D model is a triangular mesh, a spline surface, or any other surface representation. In some examples, the method may be iterative, or the intersection may be formulated as a solution to an optimization problem that can be solved using a numerical solver.

[0242] In scenarios where the ground is uneven or has small hills, or when the ball is located at the edge of a bunker, there may be multiple intersections between the 3D model 1135 and the camera-ball line 1225. For example, additional intersections 1230 and 1235 between the 3D model 1135 and the camera-ball line 1225 are shown in Figure 12. In this case, the intersection module 1120 may determine the ball's position to be at the intersection 1220 between the camera-ball line 1225 and the 3D model closest to camera 1105.

[0243] The intersection module 1120 passes the 3D coordinates of the intersection 1220 to the output module 1125. The output module 1125 connects to any output means necessary to use the ball position information. For example, a database 1140 that stores information on shots hit during play; a 3D graphics rendering engine 1145 that can depict the ball position for television or online viewers, for example, a top view of the course; calculation of the distance from the stationary ball position to the tee position and / or input into a graphical diagram (for actual carry, including bounce and roll); or calculation of the distance from the ball position to a flag, or other interesting features of the golf course such as a bunker, water hazard, or green, and / or input into a graphical diagram.

[0244] System 1100 may include a radar 1150 and a radar tracker 1155 that can measure the ball position as a function of time, the radar 1150 being calibrated to the same coordinate system as the tracking camera 1105, preferably the world coordinate system 1240, and the radar 1150 being time-synchronized with the tracking camera 1105 so that each image can be correlated with the ball position measured by the radar. In this scenario, the ball position detected by the radar 1150 can be mapped to the image plane of the camera 1105 using a projector 1160 to narrow the search window of the ball detector 1110. This is particularly advantageous when the ball is in the field of view of the camera 1105, including the green and the area surrounding the green for the green-side camera 1105, because in some situations the ball detector 1110 only needs to operate when the ball has been previously detected by the radar 1150.

[0245] Radar 1150 may be a continuous-wave Doppler radar emitting microwaves at an X-band frequency (10 GHz) with a power of up to 500 milliwatts EIRP (Equivalent Isotropic Radiated Power), and thus may comply with FCC and CE regulations for near-field international radiators. However, in other jurisdictions, other power levels and frequencies may be used in accordance with local regulations. In exemplary embodiments, microwaves are emitted at higher frequencies, for example, between 5 and 125 GHz. Frequencies above 20 GHz may be used for more accurate measurements at lower object velocities. Any type of continuous-wave (CW) Doppler radar may be used, including phase-modulated or frequency-modulated CW radars, multi-frequency CW radars, or single-frequency CW radars. It will be understood that other tracking devices, such as lidar, may be used in conjunction with radiation in either the visible or invisible frequency domain. Current-pulsed radar systems have limited ability to track objects close to the radar device. However, the distance at which an object must be away from these pulse radar systems is decreasing over time and is expected to continue decreasing. Therefore, these types of radars may soon be effective for these operations, and their use in the systems of the invention described below is intended.

[0246] Any type of radar capable of tracking an object in three dimensions may be used. An MFCW Doppler radar operating anywhere in the X-band or permitted band may have multiple receiving antennas spaced apart in a two-dimensional array to measure the phase difference between receivers, thereby deriving the direction to a ball in a local radar coordinate system using knowledge of the wavelength of the transmitted wave, and to transmit multiple frequencies to measure the phase difference, thereby deriving the range to the ball from the knowledge of the wavelength of the transmitted wave.

[0247] The radar tracker 1155 is responsible for detecting a moving object of interest within the raw sensor data and, where possible, splicing together consecutive detections of the same moving object into a "track." Thus, a track is one or more detections of a moving object in continuous time. The track may be smoothed to remove the effects of noise, and the track may be represented in a way that allows positions along the track to be interpolated between measured positions or extrapolated both forward and backward in time beyond the measured positions.

[0248] To calibrate radar 1150 to world coordinates, or to any common coordinate system used for camera 1105 and 3D model 1135, the radar's position and orientation must be determined. For this purpose, the radar unit may include a camera having known internal parameters as described above, and a known relative position and orientation to the local radar coordinate system. The radar external parameters can then be determined in the same way as the camera external parameters.

[0249] In a preferred embodiment, the radar 1150 and camera 1105 are incorporated into a single physical hardware unit having a known camera-versus-radar calibration, which can be determined when the unit is manufactured. Thus, once the unit is calibrated to a golf course using the technique described above or a similar method, both the radar 1150 and camera 1105 are calibrated to world coordinates, and the measurements can be referenced to a 3D model 1135 of a portion of the golf course in the same coordinate system.

[0250] In a preferred embodiment, the track of a golf ball in flight is determined using the position determined by the radar 1150, in combination with the detection of a ball detector 1110 based on an image captured by camera 1105. Thus, a more accurate ball track can be obtained. If a Kalman filter is used to smooth the radar position over time, the ball detection from camera 1105 can be used as input to the Kalman filter to improve the accuracy of position determination. Using the enhanced radar trajectory, a more accurate landing position of the ball can be determined than the position determined using radar measurements alone. The landing position of the ball can be determined as the intersection 1220 between the ball track and a 3D model 1135 of the golf course portion, in a manner similar to how the intersection between the camera-ball line and the 3D model is determined, as described above.

[0251] In a preferred embodiment, the ball detector 1110 may be guided to reduce the search area to the vicinity of the expected ball position in the image, thereby reducing the computation required to detect the ball. The ball position from the radar may be projected into the image through knowledge of the internal and external parameters of each sensor, or the ball detector may include a memory of the ball's position detected in previous images, so that for each new image received from the camera, the search area for the ball can be determined using prior information about where the ball was detected in one or more previous images. This information may be stored in the ball detector 1110 as an array of (u,v) coordinates of the previous frame. Alternatively, the ball detector 1110 may include an array of previous images so that, when processing a new image, it can simultaneously supply image information of the search area from several consecutive images to the neural network to improve the detection speed and accuracy of the neural network.

[0252] Furthermore, the ball detector 1110 may be equipped to handle several balls within the search area. In a golf round or tournament, under normal conditions, only one ball moves on a particular hole at a time, while the others remain stationary. By using a neural network to detect multiple balls within the search area and using detection information stored from previous images, it is possible to determine which ball detections represent stationary balls and which represent moving balls. Alternatively, a neural network trained to use several consecutive image regions for ball detection can determine, as part of its output, which detections represent stationary or moving balls. This allows the search window to be updated to follow the moving ball, even when multiple balls are present in the image, so as not to lose track of the ball. The method described herein allows for the use of a limited search window after the initial ball detection when radar is not present in the system or when the radar tracker 1155 is unable to track the ball.

[0253] The ball detector 1110 determines when a moving ball comes to rest by comparing the (u,v) coordinates of the ball detection in two or more consecutive images. After this determination is made, the ball detector 1110 sends signals to the non-projector 1115, the crossbody 1120, and the output module 1125, which perform the appropriate calculations as described above and generate the output. If a previous radar track for a golf shot has been determined, the resting position of the stroke thus determined is connected to the radar measurement and preferably labeled with an identifier that is also used to label the radar track.

[0254] Figure 17a shows a method 1700 for determining the 3D position coordinates of a ball by various exemplary embodiments. A system for carrying out method 1700 includes at least a camera and processing device for detecting the presence of a ball in an image, projecting the image coordinates onto 3D coordinates, determining the camera-line ball in 3D coordinates, and determining the intersection of the camera-line ball. Various of the steps described above can be enhanced if additional radar data is available, as described above. The intersection of the 3D model and the camera-ball line determines the position of the ball in 3D coordinates. In some embodiments, the system determines whether the ball is stationary and the position in 3D coordinates is the stationary position.

[0255] In 1702, the camera captures a series of images. The camera has sensors and lenses selected to provide a field of view (FOV) of a portion of the golf course where tracking is taking place, and a resolution selected to provide enough ball pixels for the ball detector to operate effectively. The camera operates at a specific frame rate f s a series of images (...,i (n-1) ,i (n) ,i (n+1), ...) captures the time the image was captured, t n Metadata, including exposure time, cropping area, and potentially other parameters, can be included in the image.

[0256] In 1704, a processing device, e.g., its ball detection module, sequentially receives images from a camera and executes a ball detection algorithm on the images to determine whether a ball is present in the image and the pixel positions of the (u, v) coordinates of the detected ball. In some embodiments, the ball detector can use radar data to, for example, narrow the image regions to be searched or trigger the execution of the ball detection algorithm within a time. The radar and the camera may be incorporated into one physical hardware unit, or may be positioned separately as long as, for example, they are commonly calibrated to world coordinates. In some embodiments, the ball detector may provide ball detection to the radar / processing device to improve radar tracking and subsequently improve the accuracy of ball detection. In one embodiment, the ball detector determines when a moving ball stops by comparing the (u, v) coordinates of ball detection in two or more consecutive images.

[0257] The ball detector can narrow the search area based on previous information, e.g., an array of (u, v) coordinates for ball detection in the previous frame. The ball detection algorithm may include a convolutional deep neural network (CNN) trained to detect golf balls in an image, which can receive image information of the search area from an array of previous images to increase the detection speed and accuracy of the CNN. Further, the ball detection algorithm can be equipped to handle some of the balls within the search area using logic to determine which ball detections represent stationary balls and which represent moving balls. In another embodiment, the neural network is trained to determine which detections represent stationary or moving balls, enabling the search window to be updated without losing track of the ball.

[0258] In 1706, a processing device, e.g., a non-projection module, reads the intrinsic and extrinsic calibration parameters of the camera and uses the (u, v) coordinates received from the ball detector to execute an algorithm to determine a camera-ball line that includes a straight line passing through the focus of the camera in the direction of the ball in 3D space, e.g., world coordinates. The processing device can further use metadata associated with an image, e.g., a crop region, to determine the camera-ball line. The crop region may also be stored in the system if it is kept constant across a series of images.

[0259] In 1708, a processing device, e.g., an intersection module, executes an intersection algorithm based on the camera-ball line and at least a partial 3D model of the golf course that overlaps with the FOV of the camera, and determines the intersection points of the 3D model of the camera-ball line. This intersection point is determined based on the representation of the 3D model. In some embodiments, the process of determining the intersection point may be iterative or may solve an optimization problem using a numerical solver. The camera and / or radar and / or 3D model are calibrated with respect to a common coordinate system, which may be, for example, world coordinates. The 3D model is a representation of a golf course (or a part of the golf course) stored in computer memory and includes a surface model (e.g., a triangular mesh, a spline surface, etc.) that represents the height and / or type of the terrain at a given position on the course. The 3D model may be provided in world coordinates for ease of mapping, such as GPS position to the 3D model. The 3D model may also be provided in a local coordinate system and have a mapping to world coordinates.

[0260] In 1710, a processing device, e.g., an output module, outputs the 3D position of the ball to several output means, such as a database, a 3D graphics rendering engine, a graphical diagram of the position of the ball relative to other course features.

[0261] When the ball is determined to be stationary, additional decisions may be made, such as the type of terrain the ball is currently located on, as described in more detail below.

[0262] Tracking bounce and roll In another aspect of this disclosure, camera-based tracking may be used to track and determine the bounce and roll of the ball. The determination of the bounce is preferably performed by the non-projector 1115 described above, but may also be performed by the ball detector 1110 based solely on (u,v) coordinates. In one exemplary method, a series of images are processed to determine the presence of bounce, as described below.

[0263] Figure 17b shows a method 1720 for determining the bounce and roll of a moving ball based on image data. A system for carrying out method 1720 may be similar to the system described above in Figures 11 and 17a, comprising at least a processing unit including a camera, a ball detector, and a non-projector. Method 1720 can be used, for example, to track a chip shot, an approach shot, or any other shot that exhibits a bouncing motion.

[0264] In 1722, as in 1702 above, the camera captures a series of images. In 1724, similar to 1704, a processing unit, such as its ball detection module, receives images from the camera and determines whether a ball is present in the images and the pixel position of the detected ball at (u,v) coordinates.

[0265] In 1726, after a series of images have been processed, the ball detector (or, in some embodiments, the non-projector) generates a time series of ball detections in the (u,v) image plane from the series of images. In 1728, the non-projector transforms the (u,v) coordinates into a series of camera-ball lines using the intrinsic and extrinsic camera calibrations available to the non-projector. The elevation angle of each line, i.e., the angle between the camera-ball line and the ground plane, is calculated to obtain the perpendicular projection angles of the time series from the camera.

[0266] Figure 13 shows a time series 1300 of ball detection in the (u,v) image plane from a series of images and a corresponding time series 1350 of the elevation angle relative to the camera-ball line determined by the non-projector 1115 described in Figure 11. Time series 1350 is analyzed with respect to the minimum value representing detection, indicated at points 1352 and 1354 of time series 1350, where the projection angle from the camera to the ball is steepest and may correspond to a collision between the ball and the ground.

[0267] At 1730, these minimum camera-ball lines in time series 1350 correspond to the bounce, but are passed to the cross module to determine the 3D position of the bounce, as described above with respect to Figures 11 and 17a. Since the ball may be assumed to be on the ground or very close to the ground for these detections, the 3D ball position may be precisely determined by the method used to determine the ball's stationary position. The accuracy of the bounce detection may be improved to subframe accuracy by extrapolating either the (u,v) coordinates or the vertical projection angle both before and after the minimum value in the time series to determine the most likely subframe time, vertical projection angle, and camera-ball line of the bounce.

[0268] In systems involving radar, it is sometimes possible to track bounces within the radar as a separate track, tracking signals from a rolling ball. In these cases, this information can be used to enhance bounce and roll tracking performed using only cameras. By measuring the ball's radial velocity and distance to the ball, and because the ball's radial velocity typically exhibits discontinuities when the ball bounces, the timing of the bounce can be determined with great precision by detecting the start and end points of the radar track representing the bounce.

[0269] In 1732, bounce is distinguished from rolling. Here, as described below, the time series 1300 of ball detection in the (u,v) plane is used in a different manner in this specification.

[0270] A physical model of bounce can be used to better distinguish between the bounce and roll of a ball. In an exemplary method, given two consecutive minimums in the (u,v) time series 1300 of pixel position detection, the position of the ball at each minimum may be determined from unprojected data as previously described, under the assumption that the ball collides with the ground at each minimum, and the time of each assumed ground collision may be determined as the time of the image closest to each minimum, or alternatively, it may be determined using extrapolation of the (u,v) detection as previously described. From the determined 3D position and the start and end times of the bounce, an approximate 3D track of the ball can be determined based on the physical model 1165. A simple model can assume that the ball is affected only by gravity, and therefore, traction and lift (Magnus force) are irrelevant at low speeds and short distances associated with the bounce. The track determined from the physical model 1165 is projected onto the image plane of camera 1105 and compared to the ball detection between the two (u,v) time series minimums. A good response suggests that the ball likely bounced during the minimum range.

[0271] Detection of (u,v) time-series images that are not determined to be bounces is classified as part of the ball's roll. Since the ball can be presumed to be on the ground while rolling, a track corresponding to the roll can be constructed by deprojecting the (u,v) position and then determining the intersection between the camera-ball line and the 3D model of the course.

[0272] In 1734, this track is passed to output module 1125 for output to the user, to be overlaid on the broadcast, or to be associated with the main ball track by a common identifier and stored in a database. Thus, the ball track determined using the method described above provides important information about the bounce that is passed to output module 1125.

[0273] Determining the terrain relative to the lie of a stationary ball. In another embodiment, the lie or terrain features of the ball, in which it will be positioned for the next stroke, can be determined. Two exemplary methods are described, or a combination of the two methods may be used. In one method, when the ball is determined to be stationary, the position of the ball determined by the cross module 1120 can be compared with information from the 3D model 1135 of the golf course to determine which terrain the ball is on. This could be, for example, the fairway, semi-rough, rough, fringe, bunker, or green. In another method, ball detection in the image can be classified into one of the following: fairway, semi-rough, rough, fringe, bunker, or green, based on how much of the ball is visible to the camera to characterize the length of the grass, or based on the classification of the background of a small crop of the image centered on the ball. The latter determination can be made using a neural network trained to detect the terrain type of the lie based on a small crop of the image containing the ball and terrain.

[0274] If the ball is not visible in the image, the lie may be determined when the next stroke is captured by capturing the image, the golfer is detected in the image, the club is detected in the image when the golfer addresses the ball, the direction to the club head is estimated by non-projection, the approximate position of the golf club head is determined, and the terrain type is identified as the terrain associated with this position in a 3D model of the course.

[0275] Chip shots and putts In another embodiment, chip shots and putts may be tracked by the system by employing techniques similar to those described above for tracking bounce and roll. Tracking chips and putts is generally not possible with radar-only systems because shots are hit in any direction, either on the green or from the area immediately surrounding the green toward the green, and shots are often slow, resulting in low radial velocities that can be measured by radar. Chip shots generally follow the arc of the initial shot and involve one or more bounces and rolls. The arc of the initial shot can be tracked by tracking the bounce using the non-projection and crossing techniques described above. The bounce and roll of subsequent shots can similarly be tracked using the techniques already described.

[0276] A putt trajectory can be subdivided into up to three segments characterized by the physics of the moving ball: the bounce segment; the slide segment; and the roll segment. During the bounce segment, the ball exhibits small bounces, and its velocity decreases each time it bounces off the ground. During the slide segment, the ball does not bounce, but its rotation does not match the speed at which it travels on the ground, causing it to slide. In this segment, the ball loses velocity, but the spin resulting from friction between the ball and the ground increases. During the roll segment, the ball loses velocity due to friction. Generally, the velocity that decreases as a function of time is greatest during the bounce segment and least during the roll segment. Not all putts exhibit bounce and slide segments. By measuring these aspects of putts and using logic related to the principles described above, putting segments can be identified in tracking data. Further details regarding putting segments and corresponding velocity decreases are provided in U.S. Patent No. 10,444,339, which is incorporated herein by reference in its entirety.

[0277] Putts can be tracked in a similar manner to chip shots. When tracking putt trajectories using currently disclosed camera and 3D modeling systems, it is often effective to assume that the ball is on the ground throughout its entire trajectory, thereby reducing putt tracking to roll tracking. Even without bounce tracking, the ball's velocity during the putt trajectory can be matched with the 3D model to determine the bounce, slide, and roll segments of the trajectory.

[0278] To limit the power consumption and / or computational load on the system, system 1100 can capture images at a low frequency (e.g., 1 fps) and perform person detection 1170. If person detection 1170 is successful (a person is detected in the image), address detection 1175 can be performed to determine whether a shot is about to be hit. If an address is detected, system 1100 can switch camera 1105 to a higher frame rate as identified above and initiate ball detection 1110 in the area of ​​the image around the detected address. Using prior knowledge of the ball's position in the image, e.g., determined by an exemplary method for determining the ball's stationary position, it can be determined whether person detection 1170 is likely to result in address detection 1175. For example, if a person is detected near the ball, an address is likely to occur. If no ball is detected in the image near the person detection, an address is unlikely to occur. It is advantageous to implement the person detector 1170 and address detector 1175 as convolutional neural networks trained to detect people and golfers in an addressing posture.

[0279] In another aspect, the putting break fan can be determined from the ball's rest position in front of the putt and the 3D model of the golf course. The 3D put break fan should display different putt trajectories from the ball's initial position and, due to changes in launch speed and launch direction, result in a successful putt attempt. The predicted successful putt attempt is determined as the ball trajectory that intersects the position of the hole at a speed that drops the ball into the cup. Generally, this depends on whether the ball drops more than half its diameter due to gravity when it is above the hole. Thus, a putt trajectory that crosses the center of the hole can result in a successful putt at a higher speed than a putt trajectory that has less impact on the hole. Using knowledge of the green's stimp and the 3D model of the course, the simulated putt trajectory of the ball can be determined by assuming specific launch speeds and launch directions for the putt. The method of determining the put break fan is to simulate many putts with different launch speeds and launch directions and observe which launch conditions result in a successful putt.

[0280] FIG. 14 shows a plot 1400 including a start position 1405, a hole position 1410, and lines 1415 - 1425 representing putt trajectories that may result in a successful putt. Line 1415 represents the highest launch speed that results in a successful putt, line 1425 represents the lowest launch speed that results in a successful putt, and line 1420 represents the launch speed between the highest and lowest launch speeds that result in a successful putt. Lines 1415 and 1425 correspond to the boundaries of the so-called put break fan. In many cases, line 1420 represents the putt trajectory where the ball passes the hole position and comes to rest at 2 feet if the cup did not exist to interrupt the ball in its path. Putts between lines 1415 and 1420 are likely to result in a short follow-up putt of less than 2 feet if the putt is not successful, while putt trajectories between lines 1420 and 1425 result in a follow-up putt of more than 2 feet.

[0281] Figure 15 shows a putt break fan 1500 as a chart 1510 of the combinations of launch direction and launch speed for a successful putt, and as contour lines 1515, 1520 showing the combinations of launch direction and launch speed that result in a ball position equal to the distance to the pin for the second putt.

[0282] A method is provided for determining the tolerance for a successful putt. Once successful putts are hit, tracked by the system, and the results are determined, the tolerance for success can be determined by simulating small changes in launch velocity and launch direction and the resulting simulation results. Similarly, for unsuccessful putts, it is possible to determine how much the launch conditions need to be changed for the putt to be successful. This information can be output as a two-dimensional diagram showing the combinations of launch velocity and launch direction for successful putts from the initial position of the putt.

[0283] In some situations, a golfer will not try to sink the putt, but instead attempt to place the ball in a more favorable position, perhaps closer to the pin, in order to increase the probability of a successful next putt. The results of such layup putts can be analyzed in a similar manner using simulated putting. An isocurve on a plot of launch velocity versus launch direction can show the launch conditions that result in a similar distance from the pin for the next shot.

[0284] In another embodiment, inverse calculations are performed to determine the stimpmeters on the green and a 3D model of the green from a 3D ball track. As will be understood by those skilled in the art, stimpmeters are a measure of the green that describes the resistance applied to the movement of the ball by the surface (i.e., how much the surface quality slows the ball down as it rolls, such as the effect of gravity on a hill). Stimpmeters can be determined by calculating velocity decay, taking into account any change in height along the ball track, as they measure the degree to which the ball is slowed down by the resistance of the green. Due to factors such as sunlight, irrigation, and drainage, stimpmeters vary across the green and are often directional. However, based on tracking several putts, a 2D map of stimpmeters across the green can be determined for further analysis of putt trajectories and resistance to successful putts.

[0285] In yet another embodiment, a camera-based track representing known bounces can be used to update the calibration between the camera and the 3D model of the course. The accuracy of this calibration is crucial for accurately determining the camera-ball line when not projected, and therefore for determining the precise ball position in the intersecting module.

[0286] As described above, bounce can be characterized by a physical model based on the 3D positions of the start and end of the bounce, as well as the duration of the bounce. By projecting the bounce ball position onto the image plane, the projected ball pixel position can be compared to the pixel position of the ball detection. If the calibration from the camera to the 3D model is slightly changed, a new physical model of the bounce is resulting, which can be projected onto the image plane and compared to the ball detection. Therefore, the calibration offset can be determined as a change in calibration that best matches the projection of the physical model ball position to the detected ball position.

[0287] Similarly, if the distance to the ball can be determined from the radar at the start or end of the bounce, the range determined by the radar may be compared to the distance to the ball determined in the cross module, and the camera calibration to the 3D model may be updated to obtain the range from the camera to the ball in the cross module that best matches the range to the ball measured by the radar. Such updates may be applied after several detections have been made to more accurately determine which calibration update to apply, or adaptive filters may be used to remove noise related to the detection of calibration offsets. Conversely, if the camera calibration to the 3D model is more reliable, the radar range measurements may be updated to reflect the bounce location determined from the camera and ball detection in the 3D model.

[0288] Figure 17c shows a method 1740 for determining information about the next putt, according to various exemplary embodiments. A system for carrying out method 1740 may be similar to the system described above in Figures 11 and 17a-b, and may include at least a processing unit comprising a camera and a ball detector, a non-projector and a crossing module. Method 1740 can be used, for example, to generate an analysis of the next putt.

[0289] In 1742, as described above, the camera captures a series of images, the processing unit determines whether a ball is in the images and the pixel position of the detected ball at (u,v) coordinates, the processing unit determines the camera-ball line, and the processing unit determines the intersection of the camera-ball line with the 3D model.

[0290] The ball's 3D position is its resting position on the green. Based on this resting position, information regarding the next putt can be determined.

[0291] In 1744, knowledge of the green's stimpmeter and the course's 3D model can be used to determine the simulated putt trajectory of the ball by assuming a specific launch velocity and direction for the putt. Many putts are simulated with different launch velocities and directions, and it is observed which launch conditions result in a successful putt.

[0292] In 1746, various simulated launch conditions are mapped onto the Pat Break fan diagram.

[0293] Figure 17d shows a method 1750 for determining the terrain parameters of a putting green based on tracked putts, according to various exemplary embodiments. A system for carrying out method 1750 may be similar to the system described above in Figures 11 and 17a-c, and may include at least a processing unit including a camera and a ball detector, a non-projector and a crossing module. Method 1750 can be used, for example, to analyze the conditions of the green based on putts performed.

[0294] In 1752, as described above, the camera captures a series of images, the processing unit determines whether a ball is in the images and the pixel position of the detected ball at (u,v) coordinates, the processing unit determines the camera-ball line, and the processing unit determines the intersection of the camera-ball line with the 3D model. However, in this step, the camera performs this positioning function to track the current putt.

[0295] In 1754, parameters of the putt are extracted, including the velocity decay of the tracked putt. Velocity decay can be characterized for the entire putt or for multiple segments of the putt.

[0296] In 1756, the parameters of a putt are refined based on the elevation difference encountered by the putted ball along its path. This step is preferably carried out based on a 3D model of the green. However, the elevation difference may be tracked directly by a tracking device.

[0297] In 1758, the stimpmeter on the green is determined. As mentioned above, due to factors such as sunlight and irrigation, the stimpmeter changes across the green and is often directional. However, based on tracking several putts, a 2D map of the stimpmeter across the green can be determined for further analysis of putt trajectory and resistance to successful putts.

[0298] This disclosure relates to a system including: a database configured to store profiles, each visual profile including player identification information, and each profile including an associated player ID; a camera configured to capture a video stream containing images of athletes; a tracking device configured to capture data corresponding to the trajectory of a sports ball launched by an athlete; and a processing device connected to the database, camera, and tracking device configured as follows: detecting a first athlete in images from a video stream, determining the visual characteristics of the first detected athlete; matching the determined visual characteristics with a first visual profile and associated first player ID in the first profile stored in the database; and associating a first trajectory associated with a first sports shot with the first player ID.

[0299] The visual profile includes information about the characteristics of an athlete's ball-hitting swing. These characteristics include biometric data points combined with the biomechanics of the swing. The athlete's biometric data points include height, limb length, and related parameters. The biomechanics of the swing include the position of the limbs relative to the ball-hitting equipment, or the degree of body twist relative to the legs. The visual profile also includes information about the athlete's facial features or clothing characteristics.

[0300] The clothing characteristics of athletes are visually detected by a camera or additional cameras at the start of a sporting event. A new profile and associated visual profile are generated for new athletes not yet identified in the database by analyzing the characteristics of the new athlete's ball-hitting swing, facial features, or clothing characteristics. The profile is generated in the database when the processing unit receives a display for generating a new profile from a new athlete, and the camera or additional cameras capture a video stream of the new athlete and analyze the characteristics of the new athlete's ball-hitting swing, facial features, or clothing characteristics. An electronic identifier carried by and associated with the first athlete is placed by the processing unit to improve the matching of determined visual characteristics to the first visual profile. The electronic identifier includes a device that transmits GPS coordinates to the processing unit. If the athlete is a golfer, the processing unit is further configured to run an algorithm that utilizes golf theory to improve the matching of the determined visual characteristics of the first detected athlete with the first visual profile and associated first player ID by narrowing the list of profiles in the database that are considered when matching the determined visual characteristics of the first detected athlete with the first visual profile.

[0301] The golf theory includes theories related to golfer grouping, golf hole layout, or golf course layout. The golf theory includes theories related to the first estimated current lie of a sports ball associated with a first detected athlete, determined based on data from a tracking device. The algorithm includes a neural network trained to detect golfers and match them to visual profiles. The neural network is updated throughout the play based on changes in the golfer's visual characteristics. The neural network is updated after each athlete's detection and matching to a visual profile, after many athletes' detections and matching to visual profiles, or after a predetermined time since a previous player detection. The processing unit is further configured to output the parameters of the first trajectory and associated player ID to the broadcast. The processing unit is further configured to output the parameters of the first trajectory to the first athlete's personal device associated with the player ID before the first athlete's next shot. The parameters of the first trajectory include a second measured or estimated lie position of the second sports ball after the second sports ball has come to rest, so that the personal device application can provide information to indicate the lie position and help find the second sports ball.

[0302] The athlete is a golfer, and the parameters of the first trajectory include an indication that the ball hit by the first golfer stopped at the location of a first area on the golf course where the athlete is playing, the first area having special rules associated with it, and the parameters further include an indication of a second area from which the first golfer can hit the next shot based on the location and special rules. The processing unit is further configured to store a summary containing parameters for all trajectories of all shots hit by the first one of the athletes during the sporting event. The athlete is a golfer, and the processing unit is further configured to: detect the type of golf club used by the first athlete for the first shot hit by the first athlete; and associate the detected type of golf club with the first trajectory. The type of golf club is detected in the video stream using a neural network trained to recognize several different types of golf clubs. The type of golf club is detected based on signals emitted from an electronic tag attached to the golf club when it is being used. The processing unit is further configured to output the parameters of the first trajectory to a database so that the database can associate the parameters of the first trajectory with a first visual profile.

[0303] The disclosure also relates to a method comprising: detecting a first athlete in an image from a video stream captured by a camera; determining the visual characteristics of the first detected athlete; matching the determined visual characteristics with a first visual profile and associated first player ID of a first profile stored in a database, the database storing profiles including visual profiles of multiple athletes, each visual profile including player identification information, and each profile including an associated player ID; and associating a first trajectory associated with a first sports ball with a first player ID, the first trajectory being determined from data corresponding to the trajectory of a sports ball launched by an athlete captured by a tracking device.

[0304] The disclosure also relates to a database, a camera, and a processor coupled to a tracking device configured to perform the following operations: detecting a first athlete in an image from a video stream captured by the camera; determining the visual characteristics of the first detected athlete; matching the determined visual characteristics with a first visual profile and associated first player ID in a first profile stored in a database, the database storing profiles including visual profiles of multiple athletes, each visual profile including player identification information, each profile including an associated player ID; and associating a first trajectory associated with a first sports ball with the first player ID, the first trajectory being determined from data corresponding to the trajectory of a sports ball launched by an athlete captured by the tracking device.

[0305] The disclosure also relates to a system including: a database configured to store the respective profiles of several athletes, each profile including the identification information of one of the players and a player ID associated with one of the athletes; a tracking device configured to capture shot data corresponding to the trajectory of a sports ball launched by an athlete; a motion sensor device configured to capture motion data corresponding to the swing motion of a player or an object used to hit the ball; and a processing unit connected to the database, tracking device and motion sensor device configured as follows: detects a first swing of a first athlete from motion data captured by the motion sensor device, and the first athlete is associated with a first player ID; associates the first swing of the first athlete with a first timestamp and a first location detected for the first swing; and, based on the first timestamp and first location, associates a first trajectory of a sports ball from shot data captured by the tracking device corresponding to a first shot corresponding to the first swing.

[0306] The system stores swing data corresponding to the subsequent detection of a further swing, which has a first swing, a first timestamp, a first location, and a further timestamp and further location corresponding to the further swing, and the first swing and further swings are matched to the trajectories of the first and further swings at the end of the first athlete's play, based on the corresponding first timestamp and first location and further timestamp and further location, respectively. A motion sensor device or further device associated with the motion sensor device has GPS functionality for determining the location of the first swing. An electronic identifier carried by the first athlete and associated with the first athlete is detected, and the location of the first swing is determined. The athlete is a golfer, and the processing unit is further configured to run an algorithm utilizing golf theory to improve the match between the first swing and the first trajectory by narrowing the list of profiles in the database that are considered when making the match.

[0307] Golf theory includes theories relating to the grouping of golfers, golf hole layouts, or golf course layouts. Golf theory includes theories relating to the estimated current lie of a first athlete's ball, determined based on data from a tracking device. The processing unit is further configured to output first trajectory parameters to the first athlete's personal device associated with the player ID before the first athlete's next shot. The first trajectory parameters include the estimated lie position of one of the corresponding sports balls after the corresponding sports ball has come to rest, so that an application on the personal device can provide data to facilitate the discovery of the corresponding sports ball. The athlete is a golfer playing on a golf course, and the first trajectory parameters include an indication that the ball hit by the first golfer has come to rest in a position within a first area of ​​the golf course, the first area having special rules associated with it, and the parameters further include an indication of a second area from which the first golfer can hit their next shot according to the position and special rules.

[0308] The processing unit is further configured to store a summary containing parameters for all trajectories of all shots hit by the first athlete during a sporting event. The athlete is a golfer, and the processing unit is further configured to: detect the type of golf club used by the first athlete for the first golf shot; and associate the detected type of golf club with the first trajectory. The type of golf club is detected based on a signal emitted from an electronic tag attached to the golf club when the club is in use.

[0309] The Disclosure also relates to a method including: detecting a first swing of a first athlete associated with a first player ID from motion data captured by a motion sensor device, wherein the motion sensor device captures motion data corresponding to the swing motion of a player or a ball-hitting tool; a database stores profiles for each of a plurality of athletes, each profile including identification information of one of the players and a player ID associated with one of the athletes; associating the first swing of the first athlete with a timestamp and location; and associating a first trajectory of a sports ball with the first swing based on a timestamp and location, wherein the first trajectory is determined from data corresponding to the trajectory of a sports ball launched by an athlete captured by a tracking device.

[0310] The disclosure also relates to a database, a motion sensor device, and a processor coupled to a tracking device configured to perform the following operations: detecting a first swing of a first athlete associated with a first player ID from motion data captured by the motion sensor device, wherein the motion sensor device captures motion data corresponding to the swing motion of a player or a ball-hitting device; detecting a database that stores a profile for each of a plurality of athletes, each profile including identification information of one of the players and a player ID associated with one of the athletes; associating the first swing of the first athlete with a timestamp and position; and associating a first trajectory of a sports ball with the first swing based on a timestamp and position, wherein the first trajectory is determined from data corresponding to the trajectory of a sports ball launched by an athlete captured by the tracking device.

[0311] The disclosure also relates to a system including: a database configured to store the respective profiles of multiple athletes, each profile including the identification information of one of the players and a player ID associated with one of the athletes; a tracking device configured to capture shot data corresponding to the trajectory of a sports ball launched by an athlete; a positioning device associated with a first athlete among multiple athletes associated with a first player ID, wherein the positioning device is configured to capture position data approximating the position of the positioning device at a given time; and a processing unit connected to the database, tracking device and positioning device configured as follows: detecting a first trajectory of a first sports ball within the shot data; determining a first location and a first time when the first sports ball was launched; receiving position data including the position of the positioning device over a duration including the first time, each position associated with a timestamp; determining that the first time associated with the first location and first trajectory matches a first location and first timestamp in the position data; and associating the first trajectory of the first sports ball with a first player ID.

[0312] Matching a first position and first time with a first location and first timestamp includes: determining the first position and first location in a world coordinate system; and determining the correspondence between the first position and first location within a given distance. The system stores position data for the first round of golf of a first athlete, and the trajectories detected in the shot data are matched with position data at the end of the first round of golf. The positioning device has GPS capabilities for determining the position data. The athlete is a golfer, and the processing unit is further configured to improve the matching of the first position and first time with the first location and first timestamp by executing an algorithm using golf theory to narrow the list of profiles in the database that are considered when making the match.

[0313] Golf theory includes theories relating to the grouping of golfers, golf hole layouts, or golf course layouts. Golf theory includes theories relating to the estimated current lie of a first athlete's ball, determined based on data from a tracking device. The processing unit is further configured to output parameters of a first trajectory to the first athlete's personal device, associated with the player ID, before the first athlete's next shot. The parameters of the first trajectory include the estimated lie position relative to the sports ball corresponding to the first trajectory after the sports ball corresponding to the first trajectory has come to rest, so that an application on the personal device can provide data to facilitate the discovery of the sports ball corresponding to the first trajectory. The athlete is a golfer playing on a golf course, and the parameters of the first trajectory include an indication that the ball hit by the first golfer has come to rest in a position within a first area of ​​the golf course, the first area having special rules associated with it, and the parameters further include an indication of a second area from which the first golfer can hit their next shot according to the position and special rules.

[0314] The processing unit is further configured to store a summary containing parameters for all trajectories of all shots hit by a first athlete during a sporting event. The athlete is a golfer, and the processing unit is further configured to: detect the type of golf club used by the first athlete for the first golf shot corresponding to the first trajectory; and associate the detected type of golf club with the first trajectory. The type of golf club is detected based on a signal emitted from an electronic tag attached to the golf club when the club is in use.

[0315] The Disclosure also relates to a method including: detecting a first trajectory of a first sports ball in shot data captured by a tracking device corresponding to the trajectory of a sports ball launched by an athlete; determining a first location and a first time when the first sports ball was launched; receiving location data from a location device associated with a first athlete associated with a first player ID, wherein the database stores a profile for each of a plurality of athletes, each profile including identification information of one of the players and a player ID associated with one of the athletes, the location data approximating the location of the location device at a given time, the location data including the location of the location device over a duration including a first time, each location associated with a timestamp; determining that a first time associated with the first location and first trajectory matches a first location and first timestamp in the location data; and associating the first trajectory of the first sports ball with a first player ID.

[0316] The Disclosure also relates to a processor coupled to a database, a tracking device and a positioning device configured to perform the following operations: detecting a first trajectory of a first sports ball in shot data captured by a tracking device corresponding to the trajectory of a sports ball launched by an athlete; determining a first location and a first time when the first sports ball was launched; receiving position data from a positioning device associated with a first athlete associated with a first player ID, wherein the database stores profiles for each of a plurality of athletes, each profile including identification information of one of the players and a player ID associated with one of the athletes, the position data approximating the position of the positioning device at a given time, the position data including the position of the positioning device over a duration including a first time, each position associated with a timestamp; determining that a first time associated with a first position and a first trajectory matches a first location and a first timestamp in the position data; and associating the first trajectory of a first sports ball with a first player ID.

[0317] The disclosure also relates to a system including: a robotic camera configured to capture data corresponding to the position of a sports ball, and further configured to automatically adjust the orientation and zoom level of the robotic camera in response to the captured data or commands; and a processing unit coupled to the robotic camera configured as follows: pre-calibrate the robotic camera so that the initial orientation of the robotic camera is known in a world coordinate system, and associate each of several different zoom levels used by the robotic camera with its respective intrinsic parameter value; detect a first image position of the sports ball in the image coordinate system of a first image, and the first image is captured by the robotic camera in a first orientation; read from the robotic camera a first zoom level associated with the first image and the intrinsic parameter value associated with the first zoom level; determine a first orientation of the robotic camera based on the pan and tilt of the robotic camera relative to the initial orientation; determine a three-dimensional line passing through the robotic camera and the sports ball in a world coordinate system based on the detected first image position of the sports ball, the determined first orientation and the intrinsic parameter value of the first zoom level; and determine the three-dimensional position of the sports ball in a world coordinate system located along the three-dimensional line based on extrinsic information to the robotic camera.

[0318] In the first position of the image, the sports ball is on the surface of the sports play area, and the extrinsic information to the robot camera includes information received from a three-dimensional model of at least a portion of the sports play area encompassing the three-dimensional position of the sports ball, and the processing unit determines the three-dimensional position of the sports ball by positioning the intersection of the lines and the three-dimensional model. The system further includes pan and tilt sensors fixed to the robot camera, and the processing unit is further configured to read the pan and tilt of the robot camera relative to the initial orientation from the pan and tilt sensors.

[0319] The pan and tilt of the robot camera are determined from control signals used to adjust the robot camera to a first orientation, which are generated in response to tracking data of the sports ball captured before the acquisition of the first image. The robot camera is further configured to automatically adjust the crop in response to acquired data or external commands. Intrinsic parameters include focal length, principal point, and lens distortion. Extrinsic information for the robot camera includes sensor information regarding the distance of the sports ball from the robot camera.

[0320] The disclosure also includes a method comprising: pre-calibrating a robotic camera such that its initial orientation is known in a world coordinate system, and associating each of several different zoom levels used by the robotic camera with its respective intrinsic parameter value, wherein the robotic camera is configured to capture data corresponding to the position of a sports ball, and the robotic camera is further configured to automatically adjust its orientation and zoom level in response to the captured data or a command; detecting a first image position of a sports ball in an image coordinate system of a first image, wherein the first image is captured by the robotic camera in a first orientation. , detection; reading a first zoom level from the robot camera and associated intrinsic parameter values ​​for the first zoom level of the robot camera corresponding to the first image; determining a first orientation of the robot camera based on the pan and tilt of the robot camera relative to the initial orientation; determining a three-dimensional line passing through the robot camera and the sports ball in the world coordinate system based on the detected first image position of the sports ball, the determined first orientation and the intrinsic parameter values ​​of the first zoom level; and determining the three-dimensional position of the sports ball in the world coordinate system located along the three-dimensional line based on extrinsic information to the robot camera.

[0321] This disclosure also relates to a processor coupled to a robotic camera configured to perform the following operations: the initial orientation of the robotic camera is known in a world coordinate system, and the robotic camera is pre-calibrated such that each of the different zoom levels used by the robotic camera is associated with its respective intrinsic parameter value, wherein the robotic camera is configured to capture data corresponding to the position of a sports ball, and the robotic camera is further configured to automatically adjust its orientation and zoom level in response to the captured data or a command; and the first image position of the sports ball is detected in the image coordinate system of the first image, wherein the first image is first Detecting the sports ball as it is captured by the robot camera; reading the associated intrinsic parameter values ​​for the first zoom level and the first image from the robot camera; determining the first orientation of the robot camera based on the pan and tilt of the robot camera relative to the initial orientation; determining a three-dimensional line passing through the robot camera and the sports ball in the world coordinate system based on the detected first image position of the sports ball, the determined first orientation and the intrinsic parameter values ​​of the first zoom level; and determining the three-dimensional position of the sports ball in the world coordinate system located along the three-dimensional line based on extrinsic information to the robot camera.

[0322] The disclosure also relates to a system including: a robotic camera having a predetermined position in a world coordinate system, configured to capture data corresponding to the position of a sports ball, and further configured to automatically adjust orientation and zoom level in response to captured data or commands; and a processing unit coupled to the robotic camera configured as follows: detecting a first image position of the sports ball in the image coordinate system of a first image, the first image being captured by the robotic camera in a first orientation; detecting a reference point in the image coordinate system of the first image or in a further image captured after initial adjustment of the orientation of the robotic camera, in order to bring a reference point into its field of view, the three-dimensional position of the reference point being known in the world coordinate system; adjusting the orientation of the robotic camera to a second orientation so that the reference point is located at the first image position in a second image; determining the difference in orientation between the first and second orientations; determining a three-dimensional line passing through the robotic camera and the sports ball in the world coordinate system based on the three-dimensional position of the reference point relative to the robotic camera and the difference in orientation between the first and second orientations; and determining the three-dimensional position of the sports ball in the world coordinate system located along the three-dimensional line.

[0323] In the first image position, the sports ball is on the surface of the sports play area, and the three-dimensional position of the sports ball is determined based on the identification of intersections between lines and a three-dimensional model of at least a portion of the sports play area encompassing the three-dimensional position of the sports ball.

[0324] The second image is captured without adjusting the zoom level relative to the first image. The system further includes pan and tilt sensors fixed to the robot camera, and the processing unit is further configured to read the orientation difference between the first and second orientations from the pan and tilt sensors.

[0325] The Disclosure also relates to methods including: detecting a first image position of a sports ball in an image coordinate system in a first image, wherein the first image is captured by a robotic camera in a first orientation, the robotic camera has a predetermined position in a world coordinate system, the robotic camera is configured to capture data corresponding to the position of the sports ball, and the robotic camera is further configured to automatically adjust orientation and zoom level in response to captured data or commands; detecting a reference point in the image coordinate system in the first image or in a further image captured after initial adjustment of the orientation of the robotic camera, wherein the three-dimensional position of the reference point is known in the world coordinate system; adjusting the orientation of the robotic camera to a second orientation so that the reference point is located at the first image position in a second image; determining the difference in orientation between the first and second orientations; determining a three-dimensional line passing through the robotic camera and the sports ball in the world coordinate system based on the three-dimensional position of the reference point relative to the robotic camera and the difference in orientation between the first and second orientations; and determining the three-dimensional position of the sports ball in the world coordinate system located along the line.

[0326] This disclosure also relates to a robotic camera configured to perform the following actions: detecting the first image position of a sports ball in an image coordinate system in a first image, wherein the first image is captured by the robotic camera in a first orientation, the robotic camera has a predetermined position in a world coordinate system, the robotic camera is configured to capture data corresponding to the position of the sports ball, and the robotic camera is further configured to automatically adjust orientation and zoom level in response to captured data or commands; and capturing a reference point in the image coordinate system in the first image or the robotic camera in its field of view. The method involves detecting a reference point in a further image captured after initial adjustment of the orientation of the robot camera, the three-dimensional position of the reference point being known in the world coordinate system; adjusting the orientation of the robot camera to a second orientation so that the reference point is located at the position of the first image in the second image; determining the difference in orientation between the first and second orientations; determining a three-dimensional line that penetrates the robot camera and the sports ball in the world coordinate system based on the three-dimensional position of the reference point relative to the robot camera and the difference in orientation between the first and second orientations; and determining the three-dimensional position of the sports ball in the world coordinate system located along the three-dimensional line.

[0327] This disclosure also relates to a system including: a first camera which is a robotic camera configured to capture first data corresponding to the position of a sports ball, the robotic camera further configured to automatically adjust orientation and zoom level in response to captured data or commands; a second camera calibrated to a world coordinate system which is configured to capture second data corresponding to the position of a sports ball; and a processing unit coupled to the robotic camera and the second camera which is configured as follows: detects a first image position of a sports ball in an image coordinate system of the first image, the first image is captured in a first orientation of the robotic camera; detects at least a first feature in the first image; detects a first feature in a second image captured by the second camera; determines the three-dimensional position of the first feature in a world coordinate system based on the calibration of the second camera; performs feature matching between the first and second images based on the first feature to identify the first image position in the second image; determines a three-dimensional line passing through the robotic camera and the sports ball in a world coordinate system based on the three-dimensional position of the first feature in a world coordinate system and the first image position of the sports ball; and determines the three-dimensional position of the sports ball in a world coordinate system located along the three-dimensional line.

[0328] In the first image position, the sports ball is on the surface of the sports playing area, and the three-dimensional position of the sports ball is determined based on the intersection of a line and a three-dimensional model of at least a portion of the sports playing area encompassing the three-dimensional position of the sports ball. The second camera is positioned in the same location as the first camera. The sports playing area is a golf course, and the features of the first camera include trees, bunkers, ponds, a golf flag, and a green.

[0329] The Disclosure also relates to methods including: detecting a first image position of a sports ball in an image coordinate system in a first image, wherein the first image is captured by a first camera in a first orientation, the first camera being a robotic camera configured to capture first data corresponding to the position of the sports ball, the robotic camera being further configured to automatically adjust orientation and zoom level in response to captured data or commands; detecting at least a first feature in the first image; detecting a first feature in a second image captured by a second camera, the second camera being calibrated in a world coordinate system and configured to capture second data corresponding to the position of the sports ball; determining the three-dimensional position of the first feature in the world coordinate system based on the second camera calibration; performing feature matching between the first and second images based on the first feature to pinpoint the first image position in the second image; determining a three-dimensional line passing through the robotic camera and the sports ball in the world coordinate system based on the three-dimensional position of the first feature in the world coordinate system and the first image position of the sports ball; and determining the three-dimensional position of the sports ball in the world coordinate system located along the three-dimensional line.

[0330] This disclosure also relates to a processing unit coupled to a first camera and a second camera configured to perform the following operations: detecting a first image position of a sports ball in an image coordinate system in a first image, wherein the first image is captured in a first orientation of the first camera, and the first camera is a robotic camera configured to capture first data corresponding to the position of the sports ball, the robotic camera further configured to automatically adjust orientation and zoom level in response to captured data or commands; detecting at least a first feature in the first image; detecting a first feature in a second image captured by the second camera. The system includes detecting a second camera, which is calibrated to a world coordinate system and configured to capture second data corresponding to the position of a sports ball; determining the three-dimensional position of the first feature in the world coordinate system based on the second camera calibration; performing feature matching between the first and second images based on the first feature to identify the position of the first image within the second image; determining a three-dimensional line passing through the robot camera and the sports ball in the world coordinate system based on the three-dimensional position of the first feature in the world coordinate system and the first image position of the sports ball; and determining the three-dimensional position of the sports ball in the world coordinate system located along the three-dimensional line.

[0331] This disclosure also relates to a system including: a robotic camera configured to capture data corresponding to the position of a sports ball, the robotic camera further configured to automatically adjust orientation and zoom level in response to the captured data or commands; and a processing unit connected to the robotic camera configured as follows: detecting the image position of a sports ball in an image coordinate system in multiple images, the first image position of the sports ball in the first image being captured in a first orientation of the robotic camera; adjusting the zoom level of the robotic camera to a second zoom level so that a first and second reference point are located within the field of view of the robotic camera, the image position of the sports ball being captured in the adjusted zoom level While being tracked; the second image position of the first reference point and the third image position of the second reference point are detected in the image coordinate system within the second image; the fourth image position of the sports ball in the second image is determined; the angular position of the fourth image position relative to the second and third image positions is determined by correlating the second and third image positions with the first and second reference points relative to the robot camera at a predetermined angle; a three-dimensional line passing through the robot camera and the sports ball in the world coordinate system is determined based on the first image position, the second zoom level, and the angular position of the first image position relative to the second and third image positions; and the three-dimensional position of the sports ball in the world coordinate system located along the three-dimensional line is determined.

[0332] In the first image position, the sports ball is on the surface of the sports play area, and the three-dimensional position of the sports ball is determined by identifying the intersection between a line and a three-dimensional model of at least a portion of the sports play area encompassing the three-dimensional position of the sports ball. The second image is captured without adjusting the orientation relative to the first image. If the sports ball is not visible in the second image, the fourth image position of the sports ball in the second image is estimated based on tracking the image position of the sports ball while the zoom level is adjusted.

[0333] This disclosure also relates to methods including: detecting the image position of a sports ball in a plurality of images in an image coordinate system, wherein the first image position of the sports ball in the first image is captured in a first orientation of a robotic camera, the robotic camera is configured to capture data corresponding to the position of the sports ball, and the robotic camera is further configured to automatically adjust its orientation and zoom level in response to the captured data or a command; adjusting the zoom level of a robotic camera to a second zoom level so that a first reference point and a second reference point are located within the field of view of the robotic camera, the image position of the sports ball is tracked while the zoom level is being adjusted; and adjusting the second image Detecting the second image position of a first reference point and the third image position of the second reference point in the internal image coordinate system; determining the fourth image position of the sports ball in the second image; determining the angular position of the fourth image position relative to the second and third image positions by correlating the second and third image positions with the first and second reference points relative to the robot camera at a predetermined angle; determining a three-dimensional line in the world coordinate system that penetrates the robot camera and the sports ball based on the first image position, the second zoom level, and the angular position of the first image position relative to the second and third image positions; and determining the three-dimensional position of the sports ball in the world coordinate system located along the three-dimensional line.

[0334] This disclosure also relates to a processor coupled to a robotic camera configured to perform the following operations: detecting the image position of a sports ball in an image coordinate system in multiple images, wherein the first image position of the sports ball in the first image is captured in a first orientation of the robotic camera, the robotic camera is configured to capture data corresponding to the position of the sports ball, and the robotic camera is further configured to automatically adjust its orientation and zoom level in response to the captured data or a command; adjusting the zoom level of the robotic camera to a second zoom level so that a first reference point and a second reference point are located within the field of view of the robotic camera, the image position of the sports ball is tracked while the zoom level is being adjusted. Adjusting; detecting the second image position of the first reference point and the third image position of the second reference point in the image coordinate system within the second image; determining the fourth image position of the sports ball in the second image; determining the angular position of the fourth image position relative to the second and third image positions by correlating the second and third image positions with the first and second reference points relative to the robot camera at a predetermined angle; determining a three-dimensional line in the world coordinate system that penetrates the robot camera and the sports ball based on the first image position, the second zoom level, and the angular position of the first image position relative to the second and third image positions; and determining the three-dimensional position of the sports ball in the world coordinate system located along the three-dimensional line.

[0335] The Disclosure also relates to a system comprising: a first sensor having a sensor field of view configured to capture data corresponding to the trajectory of a sports ball; a first broadcast camera having a first field of view configured to capture a first video stream containing a first image of a first portion of the trajectory of a sports ball; and a processing unit coupled to the first sensor and the first broadcast camera, configured as follows: calibrating the first sensor against the first broadcast camera such that the pixel orientation in the first image is known to the coordinate system of the first sensor, the calibration further comprising the relationship between the parameters of the first sensor and the first broadcast camera at different zoom levels; generating a broadcast video feed using the first video stream containing the first image of a first portion of the trajectory of a sports ball as the sports ball crosses the first field of view; determining trajectory parameters based on the data of the sports ball trajectory as the sports ball crosses the sensor field of view; and inserting a tracer showing the path of the sports ball into the broadcast video feed, the tracer being generated based on the calibration between the first sensor and the first broadcast camera.

[0336] The first sensor is a tracking camera. The processing unit is further configured to determine the current zoom level of the first broadcast camera by feature matching between images from the tracking camera and the first broadcast camera. The tracer is inserted into the broadcast video feed regardless of the current zoom level of the first broadcast camera, based on calibration at different zoom levels. The position of the sports ball is detected in one or more first images before launch to improve the rendering of the tracer's starting point. A ball identification algorithm searches for the position of the sports ball in the first images. The player's stance is detected before launch to find the sports ball. The system according to claim 28 further includes a second broadcast camera having a second field of view configured to capture a second video stream containing a second image of a second portion of the trajectory of a sports ball, the processing unit further comprises: calibrating the first sensor against the second broadcast camera such that the pixel orientation in the second image is known to the coordinate system of the first sensor, the calibration further including a relationship between the parameters of the first sensor and the second broadcast camera for different zoom levels; switching the broadcast video feed to the second video stream containing a second image of a second portion of the trajectory of a sports ball as the sports ball crosses the second field of view; and further inserting a tracer into the broadcast video feed based on the calibration between the first sensor and the second broadcast camera.

[0337] The Disclosure also relates to a method including: calibrating a first sensor to a first broadcast camera, the first sensor having a sensor field of view and configured to capture data corresponding to the trajectory of a sports ball; the first broadcast camera having a first field of view configured to capture a first video stream including a first image of a first portion of the trajectory of a sports ball; the first sensor and the first broadcast camera being calibrated such that the pixel orientation in the first image is known to the coordinate system of the first sensor, the calibration further including the relationship between parameters of the first sensor and the first broadcast camera at different zoom levels; generating a broadcast video feed using the first video stream including a first image of a first portion of the trajectory of a sports ball as the sports ball crosses the first field of view; determining trajectory parameters based on data of the trajectory of a sports ball as the sports ball crosses the sensor field of view; and inserting a tracer indicating the path of a sports ball into the broadcast video feed, the tracer being an insert generated based on the calibration between the first sensor and the first broadcast camera.

[0338] The Disclosure also relates to a processor coupled to a first sensor and a first broadcast camera configured to perform the following operations: calibrating the first sensor to the first broadcast camera, wherein the first sensor has a sensor field of view and is configured to capture data corresponding to the trajectory of a sports ball; the first broadcast camera has a first field of view configured to capture a first video stream containing a first image of a first portion of the trajectory of a sports ball; the first sensor and the first broadcast camera are calibrated such that the pixel orientation in the first image is known to the coordinate system of the first sensor, the calibration further comprising the relationship between parameters of the first sensor and the first broadcast camera at different zoom levels; generating a broadcast video feed using the first video stream containing a first image of a first portion of the trajectory of a sports ball as the sports ball crosses the first field of view; determining trajectory parameters based on data of the trajectory of a sports ball as the sports ball crosses the sensor field of view; and inserting a tracer indicating the path of a sports ball into the broadcast video feed, the tracer being an insert generated based on the calibration between the first sensor and the first broadcast camera.

[0339] The disclosure also relates to a system comprising: a tracking camera having a first field of view configured to capture images of a sports ball bouncing and rolling after an initial trajectory in a camera coordinate system, the tracking camera having associated intrinsic and extrinsic calibration parameters; a storage device comprising intrinsic and extrinsic calibration parameters and a three-dimensional (3D) model of at least a portion of the sports play area overlapping with the first field of view; and a processing device coupled to the tracking camera, configured as follows: to perform ball detection to detect the pixel position of the sports ball in the image; to determine the orientation of the sports ball in the 3D coordinate system, including a camera-ball line including a straight line passing through the camera, based on the intrinsic and extrinsic calibration parameters; to determine the intersection of the camera-ball line and the 3D model, based on the 3D model; and to output the intersection as the 3D position of the sports ball in the image.

[0340] The processing unit is further configured as follows: it determines that a sports ball is stationary by comparing the pixel positions of the sports ball across a series of images; and outputs the intersection as the 3D stationary position of the sports ball in the image. The captured image is associated with metadata including the time the image was acquired, the exposure time, and the cropped area. The system further includes a tracking radar having a second field of view that at least partially overlaps with a first field of view configured to capture radar data of the sports ball, and the processing unit is further configured to detect the sports ball in reliance on the radar data to narrow the area searched in the image and the number of images searched within a given time. Detection of the sports ball using radar data combined with images is used by the processing unit to improve the accuracy of the 3D positioning of the sports ball. The processing unit is further configured to detect the sports ball using a neural network trained to detect the sports ball in the image.

[0341] The neural network uses information from previous images in the search area to improve the accuracy of detecting the sports ball in the current image. The processing unit is further configured to determine the camera-ball line based on the image crop area. The sports play area is a golf course, and the 3D model of the golf course portion includes a surface model representing the terrain elevation at a given location on the golf course. The surface model further represents the terrain type at a given location on the golf course. The processing unit is further configured to determine the terrain type on which the sports ball will be detected based on the 3D model. The surface model includes a triangular mesh or a spline surface. Intersections are determined to solve the optimization problem using an iterative process or a numerical solver. The 3D location of the sports ball in the image is output to a database, a 3D graphics rendering engine, or a graphical diagram of the sports ball's 3D location. The 3D model is provided in a 3D coordinate system of the camera-ball line.

[0342] The present disclosure relates to a method comprising: performing ball detection to detect the pixel position of a sports ball in an image captured by a tracking camera, wherein the tracking camera has a first field of view configured to capture images in a camera coordinate system of the sports ball bouncing and rolling after an initial trajectory, and the tracking camera has associated intrinsic and extrinsic calibration parameters stored in a storage device; determining the orientation of the sports ball in a 3D coordinate system, a camera-ball line including a straight line passing through the camera, based on the intrinsic and extrinsic calibration parameters; determining the intersection of the 3D model and the camera-ball line based on a three-dimensional (3D) model stored in a storage device, wherein the 3D model includes at least a portion of the sports play area overlapping with the first field of view; and outputting the intersection as the 3D position of the sports ball in the image.

[0343] The disclosure also relates to a tracking camera and a processor coupled to a storage device configured to perform the following operations: performing ball detection to detect the pixel position of a sports ball in an image captured by the tracking camera, wherein the tracking camera has a first field of view configured to capture an image of the sports ball bouncing and rolling in a camera coordinate system after an initial trajectory; the tracking camera performs ball detection having associated intrinsic and extrinsic calibration parameters stored in the storage device; determining the orientation of the sports ball in a 3D coordinate system, a camera-ball line including a straight line passing through the camera, based on the intrinsic and extrinsic calibration parameters; determining the intersection of the 3D model and the camera-ball line based on a three-dimensional (3D) model stored in the storage device, wherein the 3D model includes at least a portion of the sports play area overlapping with the first field of view; and outputting the intersection as the 3D position of the sports ball in the image.

[0344] The disclosure also relates to a system comprising: a tracking camera having a first field of view configured to capture images of a sports ball bouncing and rolling after an initial trajectory in the image plane of a camera coordinate system, the tracking camera having associated intrinsic and extrinsic calibration parameters; a storage device containing the intrinsic and extrinsic calibration parameters; and a processing device connected to the tracking camera and storage device, configured as follows: to perform ball detection to detect the pixel position of the sports ball in the image plane in each series of images; to generate a first time series of ball detections in the image plane; to determine a camera-ball line for each detection of the sports ball in the first time series, based on the intrinsic and extrinsic calibration parameters, including a straight line passing through the camera in the direction of the sports ball in a 3D coordinate system; to generate a second time series of elevation angles for each camera-ball line; and to identify the bounce of the sports ball captured in the series of images based on the minimum value of the second time series.

[0345] The memory further stores a three-dimensional (3D) model of at least a portion of the sports play area overlapping with the first field of view, and the processing unit further configures: based on the 3D model, for each identified bounce, to determine the intersection of the 3D model and the camera-ball line; and output the intersection as the 3D position of the sports ball's bounce. The system further includes a tracking radar having a second field of view that at least partially overlaps with the first field of view, configured to capture radar data of the sports ball, and the processing unit further configures to identify bounces in relation to radar data by determining the discontinuity of the velocity in the radar data. The processing unit further configures: based on a physical model of the bounce, to distinguish between bounce and roll in the second time series; and to classify each minimum value in the second time series that has not been determined as a bounce as a roll. The 3D position of the sports ball in the image is output to a database, a 3D graphics rendering engine, or a graphical diagram of the sports ball's 3D position.

[0346] The Disclosure also relates to a method comprising: performing ball detection to detect the pixel position in the image plane of a sports ball in each of a series of images captured by a tracking camera, wherein the tracking camera has a first field of view configured to capture images in the image plane of a camera coordinate system of the sports ball bouncing and rolling after an initial trajectory, and the tracking camera has associated intrinsic and extrinsic calibration parameters stored in a memory device; generating a first time series of ball detections in the image plane; determining a camera-ball line for each ball detection in the first time series, based on the intrinsic and extrinsic calibration parameters, including a straight line passing through the camera in the direction of the sports ball in a 3D coordinate system; generating a second time series of elevation angles for each camera-ball line; and identifying the bounces of the sports ball captured in the series of images based on the minimum values ​​of the second time series.

[0347] The disclosure also relates to a tracking camera and a processor coupled to a storage device configured to perform operations including: performing ball detection to detect the pixel position in the image plane of a sports ball in each series of images captured by the tracking camera, wherein the tracking camera has a first field of view configured to capture images of the sports ball bouncing and rolling in the image plane of a camera coordinate system after an initial trajectory; the tracking camera performs ball detection having associated intrinsic and extrinsic calibration parameters stored in the storage device; generating a first time series of ball detections in the image plane; determining a camera-ball line for each ball detection in the first time series, based on the intrinsic and extrinsic calibration parameters, including a straight line passing through the camera in the direction of the sports ball in a 3D coordinate system; generating a second time series of elevation angles for each camera-ball line; and identifying the bounce of the sports ball captured in the series of images based on the minimum value of the second time series.

[0348] The disclosure also relates to a system including: a first sensor having a sensor field of view configured to capture data corresponding to the trajectory of a sports ball; a first broadcast camera having a first field of view configured to capture a first video stream containing a first image of a first portion of the trajectory of a sports ball; a second broadcast camera having a second field of view configured to capture a second video stream containing a second image of a second portion of the trajectory of a sports ball; and a processing unit connected to the first and second broadcast cameras, configured as follows: generates a broadcast video feed using the first video stream containing a first image of a first portion of the trajectory of a sports ball when the sports ball crosses the first field of view; determines trajectory parameters based on the data of the sports ball trajectory when the sports ball crosses the sensor field of view; detects events of the sports ball based on the trajectory parameters; and switches the broadcast video feed to use the second video stream depending on the event detected, so that the second broadcast camera captures a second image of a second portion of the trajectory of a sports ball when the sports ball crosses the second field of view.

[0349] An event includes positional reference parameters indicating that a sports ball has entered or is about to enter a second field of view. The positional reference includes a sports ball moving beyond a first distance from the launch position, or a launched ball moving beyond a second distance from a reference line. An event includes the launch of the sports ball. The second broadcast camera includes a robotic camera, and the operating state of the second camera is adjusted from a first state associated with the first camera parameters to a second state associated with the second camera parameters. The first and second camera parameters include crop, orientation, and zoom level for the robotic camera in the first and second states, respectively. The processing unit is further configured to provide a region of interest to the second broadcast camera in an area where the sports ball is likely to be found. The second field of view overlaps with the first field of view at least partially.

[0350] The second field of view does not overlap with the first field of view. The broadcast video feed is switched to use the second video stream after a predetermined delay following the detection of an event. The first sensor and the first broadcast camera are located in the same location on the same tracking unit. The first sensor and the first broadcast camera are the same device, including a tracking camera used additionally for broadcasting. The first sensor, the first broadcast camera, and the second broadcast camera are calibrated against each other so that a tracer mapping the trajectory of a sports ball in the broadcast video feed can be applied to the first and second video streams.

[0351] The Disclosure also relates to a method including: generating a broadcast video feed using a first video stream containing a first image of a first portion of the trajectory of a sports ball, wherein the first video stream is captured by a first broadcast camera having a first field of view, and the first video stream generates containing a first image of a first portion of the trajectory of a sports ball as the sports ball crosses the first field of view; determining trajectory parameters based on data of the sports ball's trajectory captured by a first sensor having a sensor field of view as the sports ball crosses the sensor field of view; detecting events of the sports ball based on the trajectory parameters; and switching the broadcast video feed to use a second video stream containing a second image of a second portion of the trajectory of a sports ball, wherein the second video stream is captured by a second broadcast camera having a second field of view, and the broadcast video feed is switched depending on an event to be detected so that the second broadcast camera captures a second image of a second portion of the trajectory of a sports ball as the sports ball crosses the second field of view.

[0352] The Disclosure also relates to a processor coupled to a first sensor and first and second broadcast cameras configured to perform operations including: generating a broadcast video feed using a first video stream containing first images of a first portion of the trajectory of a sports ball, wherein the first video stream is captured by a first broadcast camera having a first field of view, and the first video stream generates including first images of a first portion of the trajectory of a sports ball as the sports ball crosses the first field of view; determining trajectory parameters based on data of the sports ball's trajectory captured by the first sensor having a sensor field of view as the sports ball crosses the sensor field of view; detecting events of the sports ball based on the trajectory parameters; and switching the broadcast video feed to use a second video stream containing second images of a second portion of the trajectory of a sports ball, wherein the second video stream is captured by a second broadcast camera having a second field of view, and the broadcast video feed is switched depending on an event to be detected, such that the second broadcast camera captures second images of a second portion of the trajectory of a sports ball as the sports ball crosses the second field of view.

[0353] This disclosure also relates to a system including: a first sensor having a sensor field of view configured to capture data corresponding to the trajectory of a sports ball and the characteristics of a player; a first broadcast camera configured to capture a first video stream containing a first image; a second broadcast camera configured to capture a second video stream containing a second image; and a processing unit coupled to the first sensor and the first and second broadcast cameras, configured as follows: receives criteria for a custom broadcast video feed, the criteria relating to the prioritization of players or locations represented in the custom broadcast video feed; generates a first portion of the custom broadcast video feed using the first broadcast camera; detects events relating to preferred players or locations, the events based on the behavior or location of the preferred player or the trajectory parameters of one eye of a sports ball associated with the preferred player; and switches the custom broadcast video feed to use the second video stream depending on the detected events, such that the custom broadcast video feed includes a second image of the preferred player or the trajectory of a first sports ball associated with the preferred player.

[0354] An event includes the detection and identification of a preferred player in a second image from a second broadcast camera. The event further includes the position or orientation of the preferred player detected in the second image. The position or orientation of the preferred player includes the preferred player approaching the sports ball, standing on the sports ball, swinging a club, or hitting the sports ball. A neural network is used to detect the event. The event includes positional reference parameters indicating that a sports ball launched by the preferred player can be captured in the second image from the second broadcast camera. The positional reference includes a sports ball moving beyond a first distance from the launch position, or a sports ball moving beyond a second distance from a reference line, or the event includes the launch of the sports ball. The second broadcast camera includes a robotic camera, and the operating state of the second camera is adjusted from a first state using the parameters of the first camera to a second state using the parameters of the second camera.

[0355] The first and second camera parameters include crop, orientation, and zoom level for the robotic camera. The robotic camera is calibrated against the first sensor. The first sensor, the first broadcast camera, and the second broadcast camera are calibrated against each other so that a tracer mapping the trajectory of a sports ball in a custom broadcast video feed can be applied to the first and second video streams. Criteria for the custom broadcast video feed include: all shots from preferred players are displayed in the custom broadcast video feed; all shots from all players from a particular country / region are displayed in the custom broadcast video feed; or a specific player tier is identified and prioritized when determining which shots to display in the custom broadcast video feed. When it is determined that multiple preferred players are taking shots simultaneously, the video stream of one of the preferred players is buffered for broadcast after the video stream of another preferred player.

[0356] The Disclosure also includes a method comprising: receiving criteria for a custom broadcast video feed, criteria for prioritizing players or locations displayed in the custom broadcast video feed; generating a first portion of the custom broadcast video feed using a first broadcast camera configured to capture a first video stream containing first images; detecting events relating to preferred players or locations, the events being detected by a first sensor having a sensor field of view configured to capture data corresponding to the trajectory of a sports ball or player characteristics, based on the actions or location of a preferred player or trajectory parameters of a first sports ball associated with a preferred player; and switching the custom broadcast video feed to use a second video stream containing second images captured by a second broadcast camera, the custom broadcast video feed being switched depending on detected events so that the custom broadcast video feed contains second images of a preferred player or the trajectory of a first sports ball associated with a preferred player.

[0357] The Disclosure also relates to a first sensor and a processor coupled to first and second broadcast cameras configured to perform the following operations: receiving criteria for a custom broadcast video feed, the criteria relating to prioritization of players or locations represented in the custom broadcast video feed; generating a first portion of the custom broadcast video feed using a first broadcast camera configured to capture a first video stream containing first images; detecting events relating to preferred players or locations, the events being detected by a first sensor having a sensor field of view configured to capture data corresponding to the trajectory of a sports ball or player characteristics, based on the actions or location of a preferred player or trajectory parameters of a first sports ball associated with a preferred player; and switching the custom broadcast video feed to use a second video stream containing second images captured by a second broadcast camera, the custom broadcast video feed being switched depending on detected events so that the custom broadcast video feed contains second images of a preferred player or the trajectory of a first sports ball associated with a preferred player.

[0358] The disclosure also relates to a system including: a first sensor having a first field of view configured to capture first data corresponding to a first portion of a sports ball's trajectory, including the launch of a sports ball; a second sensor having a second field of view configured to capture second data corresponding to a second portion of a sports ball's trajectory; and a processing unit connected to the first and second sensors configured as follows: when the sports ball crosses the first field of view, it determines a three-dimensional (3D) position track of the sports ball based on the first data of the first portion of the sports ball's trajectory; it detects events of the sports ball based on one or more parameters obtained from the 3D position track; and it adjusts the operating state of the second sensor in response to a detected event so that the second sensor captures second data of the second portion of the sports ball's trajectory when the sports ball crosses the second field of view and the processing unit determines a 3D position track of the second portion of the sports ball's trajectory.

[0359] The event includes parameters related to the position of the sports ball relative to the second field of view. The operating state of the second sensor is adjusted to capture second data when it is determined that the sports ball has moved beyond a first distance from the launch position, or when it is determined that the sports ball has moved a second distance from the baseline. The operating state of the second sensor is adjusted from a low-power state to a full-power state. The operating state of the second sensor is adjusted from a non-tracking state to a tracking state. The second sensor includes a robot camera, and the operating state of the second sensor is adjusted from a first state using the parameters of the first camera to a second tracking state using the parameters of the second camera. The first and second camera parameters include crop, orientation, and zoom level for the robot camera. The operating state of the second sensor is adjusted from a first state where bounce and roll are not tracked using a dedicated bounce and roll tracking module to a second state where bounce and roll are tracked using a dedicated bounce and roll tracking module.

[0360] The processing unit is further configured to provide a region of interest to a second sensor that specifies an area where a sports ball is likely to be found. The second field of view overlaps with the first field of view, at least partially. The second field of view does not overlap with the first field of view. The processing unit determines a 3D position track for the first and second portions of the sports ball's trajectory in a world coordinate system. The first and second sensors are 3D Doppler radars. The first and second sensors are calibrated to a world coordinate system based on the GPS position of a reference point and the GPS positions of the first and second sensors. The first sensor is a 3D Doppler radar, and the second sensor is a tracking camera. The second sensor is calibrated to a world coordinate system based on the GPS position of a reference point, and the processing unit is further configured to identify one or more reference points in the image and determine the orientation of the second sensor based on them. The first and second sensors are located in the same location, and the orientation of the second sensor and the orientation of the first sensor, determined from image feature matching, are determined based on the orientation of the second sensor.

[0361] The Disclosure also relates to a method comprising: determining a three-dimensional (3D) position track of a sports ball based on the first data of the first portion of the sports ball's trajectory as the sports ball crosses the first field of view, from a first sensor having a first field of view configured to acquire first data corresponding to a first portion of the trajectory of a sports ball, including the launch of the sports ball; detecting an event of the sports ball based on one or more parameters obtained from the 3D position track; and adjusting the operating state of a second sensor in response to the detected event, wherein the second sensor has a second field of view configured to acquire second data corresponding to a second portion of the trajectory of a sports ball as the sports ball crosses the second field of view; and determining a 3D position track for the second portion of the trajectory of a sports ball.

[0362] The Disclosure also relates to a processor coupled to first and second sensors configured to perform operations including: determining a three-dimensional (3D) position track of a sports ball based on the first data of the first portion of the sports ball's trajectory as the sports ball crosses the first field of view, from a first sensor having a first field of view configured to acquire first data corresponding to a first portion of the sports ball's trajectory, including the launch of the sports ball; detecting events of the sports ball based on one or more parameters obtained from the 3D position track; and adjusting the operating state of a second sensor in response to the detected event, wherein the second sensor has a second field of view configured to acquire second data corresponding to a second portion of the sports ball's trajectory as the sports ball crosses the second field of view; and determining a 3D position track for the second portion of the sports ball's trajectory.

[0363] The disclosure also relates to a system including: a tracking camera having a first field of view configured to capture an image of a golf ball located on a green in a camera coordinate system, the tracking camera having associated intrinsic and extrinsic calibration parameters; a storage device including intrinsic and extrinsic calibration parameters and a three-dimensional (3D) model of at least a portion of the golf course overlapping the first field of view; and a processing device connected to the tracking camera and storage device, configured as follows: to perform ball detection to detect the pixel position of a stationary golf ball in the image plane; to determine the direction of the golf ball in the 3D coordinate system, including a camera-ball line passing through the camera, based on the intrinsic and extrinsic calibration parameters; to determine the intersection of the camera-ball line and the 3D model, based on the 3D model; to determine the 3D position of a stationary golf ball on the green; to simulate a series of putt trajectories based on various launch velocities and launch directions using knowledge of the green's stimpmeter and the 3D model; and to generate a putt break fan, which is a plot of combinations of the launch direction and launch velocity of the putt and the corresponding results of the putt, based on the series of simulated putt trajectories.

[0364] The puttbreak fan includes a plot where a combination of launch direction and launch velocity results in a successful putt into the hole, a first contour line where the combination of launch direction and launch velocity results in a stationary ball position at a first distance from the hole, and a second contour line where the combination of launch direction and launch velocity results in a stationary ball position at a second distance from the hole. The green stimpmeter is determined by tracking the previous putt and determining the velocity decay of the previous putt. The previous putt is tracked within a predetermined time before the current putt. The processing unit is further configured to overlay the puttbreak fan onto broadcast video. The processing unit is further configured to generate an output showing the probability of a successful putt depending on a given launch direction and a given launch velocity.

[0365] The disclosure also includes a method comprising: performing ball detection to detect the pixel position of a stationary golf ball in the image plane captured by the tracking camera, wherein the tracking camera has a first field of view configured to capture an image of a golf ball located on a green in a camera coordinate system, the tracking camera having associated intrinsic and extrinsic calibration parameters stored in a memory device; determining the direction of the golf ball in a 3D coordinate system, a camera-ball line including a straight line passing through the camera, based on the intrinsic and extrinsic calibration parameters; determining the intersection of the camera-ball line and the 3D model based on a three-dimensional (3D) model stored in a memory device that overlaps with the first field of view; determining the 3D position of the golf ball stationary on the green; simulating a series of putt trajectories based on various launch velocities and launch directions using knowledge of the green's stimpmeter and the 3D model; and generating a putt break fan, which is a plot of combinations of the launch direction and launch velocity of the putt and the corresponding results of the putt, based on the series of simulated putt trajectories.

[0366] The disclosure also relates to a tracking camera and a processor coupled to a memory device configured to perform the following operations: the tracking camera has a first field of view configured to capture an image of a golf ball located on a green within a camera coordinate system, and the tracking camera performs ball detection to detect the pixel position of a stationary golf ball in the image plane captured by the tracking camera, the tracking camera has associated intrinsic and extrinsic calibration parameters stored in a memory device; based on the intrinsic and extrinsic calibration parameters, the camera-ball line, including a straight line passing through the camera, the direction of the golf ball in a 3D coordinate system; based on a three-dimensional (3D) model stored in the memory device that overlaps with the first field of view, the intersection of the camera-ball line and the 3D model; the 3D position of a stationary golf ball on the green; using knowledge of the green's stimpmeter and the 3D model, the simulation of a series of putt trajectories based on various launch velocities and launch directions; and based on a series of simulated putt trajectories, the generation of a putt break fan, which is a plot of the combination of the launch direction and launch velocity of the putt and the corresponding result of the putt.

[0367] It will be apparent to those skilled in the art that various modifications can be made to this disclosure without departing from the intent or scope of this disclosure. Accordingly, this disclosure is intended to cover modifications and variations of this disclosure, provided that they fall within the scope of the appended claims and their equivalents.

Claims

1. It is a system, A robotic camera configured to capture data corresponding to the position of a sports ball, and further configured to automatically adjust the orientation and zoom level of the robotic camera in response to the captured data or an external command; and A processing unit coupled to the robot camera, The robot camera is pre-calibrated so that its initial orientation is known in the world coordinate system, and each of the multiple different zoom levels used by the robot camera is associated with its respective intrinsic parameter value; In the image coordinate system of the first image, the first image position of the sports ball on the surface of the sports play area is detected, and the first image is captured by the robot camera in a first orientation; From the robot camera, read the first zoom level associated with the first image and the first intrinsic parameter value associated with the first zoom level; Based on the pan and tilt of the robot camera relative to the initial orientation, the first orientation of the robot camera is determined; Based on the detected first image position of the sports ball, the determined first orientation, and the first intrinsic parameter value of the first zoom level, a three-dimensional line passing through the robot camera and the sports ball in the world coordinate system is determined; and The processing device is configured to determine the three-dimensional position of the sports ball in the world coordinate system located along the three-dimensional line, based on external information from the robot camera. A system in which external information to the robot camera includes information received from a three-dimensional model of at least a portion of the surface of the sports play area encompassing the three-dimensional position of the sports ball, and the processing device determines the three-dimensional position of the sports ball by positioning the intersection of the three-dimensional line and the three-dimensional model.

2. The robot camera further comprises pan and tilt sensors fixed to it, The system according to claim 1, wherein the processing device is further configured to read the pan and tilt of the robot camera relative to the initial orientation from the pan and tilt sensors.

3. The system according to claim 1, wherein the pan and tilt of the robot camera are determined from a control signal used to adjust the robot camera to the first orientation, the control signal being generated in response to tracking data of the sports ball captured before the capture of the first image.

4. The system according to claim 1, wherein the robot camera is further configured to automatically adjust the crop in response to the captured data or the external command.

5. The system according to claim 1, wherein the intrinsic parameters include focal length, principal point, and lens distortion.

6. The system according to claim 1, wherein the external information for the robot camera includes information from a sensor regarding the distance of the sports ball from the robot camera.

7. It is a method, Pre-calibrating a robotic camera such that its initial orientation is known within a world coordinate system, and associating each of a plurality of zoom levels used by the robotic camera with its respective intrinsic parameter value, wherein the robotic camera is configured to capture data corresponding to the position of a sports ball, and the robotic camera is further configured to automatically adjust its orientation and zoom level in response to the captured data or an external command; The detection of a first image position of the sports ball on the surface of the sports play area in the image coordinate system of the first image, wherein the first image is captured by the robot camera in a first orientation; Reading a first zoom level from the robot camera and a first intrinsic parameter value for the first zoom level of the robot camera corresponding to the first image; Determining the first orientation of the robot camera based on the pan and tilt of the robot camera relative to the initial orientation; Based on the detected first image position of the sports ball, the determined first orientation, and the first intrinsic parameter value of the first zoom level, a three-dimensional line passing through the robot camera and the sports ball in the world coordinate system; and This includes determining the three-dimensional position of the sports ball in the world coordinate system located along the three-dimensional line, based on external information from the robot camera, A method wherein the external information for the robot camera includes information received from a three-dimensional model of at least a portion of the surface of the sports play area encompassing the three-dimensional position of the sports ball, and the three-dimensional position of the sports ball is determined by positioning the intersection of the three-dimensional line and the three-dimensional model.

8. A processor coupled to a robot camera, Pre-calibrating the robot camera such that its initial orientation is known within a world coordinate system, and associating each different zoom level used by the robot camera with its respective intrinsic parameter value, wherein the robot camera is configured to capture data corresponding to the position of a sports ball, and the robot camera is further configured to automatically adjust its orientation and zoom level in response to the captured data or an external command; Detecting a first image position of the sports ball on the surface of a sports play area in the image coordinate system of a first image, wherein the first image is captured by the robot camera in a first orientation; Reading a first zoom level from the robot camera and a first intrinsic parameter value for the first zoom level corresponding to the first image; Determining the first orientation of the robot camera based on the pan and tilt of the robot camera relative to the initial orientation; Based on the detected first image position of the sports ball, the determined first orientation, and the first intrinsic parameter value of the first zoom level, a three-dimensional line passing through the robot camera and the sports ball in the world coordinate system; and The robot is configured to perform an operation that includes determining the three-dimensional position of the sports ball in the world coordinate system located along the three-dimensional line, based on external information from the robot camera. A processor that determines the three-dimensional position of a sports ball by positioning the intersection of the three-dimensional lines and the three-dimensional model, wherein the external information to the robot camera includes information received from a three-dimensional model of at least a portion of the surface of the sports play area encompassing the three-dimensional position of the sports ball.

Images