Tracking three-dimensional objects using a combination of radar velocity data and two-dimensional image data.

Abstract

Methods, systems, and apparatus for 3D flight tracking of objects, including media-encoded computer program products, in at least one aspect, a method includes: acquiring 2D camera image data of a golf ball; modeling a 2D trajectory of the golf ball using the 2D image data; acquiring radar velocity data of the golf ball; modeling the velocity of the golf ball using the radar velocity data, wherein modeling the velocity of the golf ball includes fitting a polynomial function to a first portion of the radar velocity data and fitting an exponential function to a second portion of the radar velocity data; combining the modeled velocity of the golf ball with the modeled 2D trajectory of the golf ball to form a 3D trajectory of the golf ball; and outputting the 3D trajectory of the golf ball in 3D space for display.

Description

BACKGROUND

[0001] This specification relates to tracking objects in flight, such as golf balls, using data obtained from different image sensor technologies.

[0002] Systems and methods for tracking the flight of a golf shot using sensors include launch monitoring, full flight two-dimensional (2D) tracking, and full flight three-dimensional (3D) tracking. Common sensor types are cameras, Doppler radar, and phased array radar. The launch monitoring approach is based on measuring a set of parameters that can be observed during the swing of a golf club and the first few inches of the ball flight after the club has struck the ball. The measured parameters are then used to infer the expected ball flight using mathematical and physical modeling.

[0003] In contrast, full flight 3D tracking systems are characterized by designs that attempt to track the full flight of a golf shot rather than inferring from launch parameters. Further, from a particular perspective, full flight 2D tracking systems track the shape of a golf shot, but will not produce 3D information, and generally cannot be used to determine key parameters such as the distance traveled by the ball. Finally, full flight 3D tracking using a combination of camera and Doppler radar data is described in U.S. Patent Publication No. 2019-0111315-A2. SUMMARY

[0004] This specification describes techniques relating to tracking objects in flight, such as golf balls, using data obtained from camera and radar sensor technologies, and more specifically to full flight 3D tracking systems using a combination of camera and radar data and a two-part model for the radar data.

[0005] In general, one or more aspects of the subject matter described in this specification can be embodied in one or more methods, including obtaining two-dimensional image data of a golf ball in flight, the two-dimensional image data derived from a camera; modeling a two-dimensional trajectory of the golf ball in flight using the two-dimensional image data; obtaining radar velocity data of the golf ball in flight, the radar velocity data derived from a radar device associated with the camera; modeling a velocity of the golf ball in flight using the radar velocity data, wherein modeling the velocity of the golf ball includes fitting a polynomial function to a first portion of the radar velocity data and fitting an exponential function to a second portion of the radar velocity data; combining the modeled velocity of the golf ball in flight with the modeled two-dimensional trajectory of the golf ball in flight to form a three-dimensional trajectory of the golf ball in flight; and outputting the three-dimensional trajectory of the golf ball in flight in three-dimensional space for display. Other embodiments of this aspect include corresponding systems, apparatus, and computer program products.

[0006] A system may include: a camera; a radar device; and a computer including a hardware processor and memory coupled to the hardware processor, the memory encoding instructions configured to cause the hardware processor to perform operations according to any methods described herein. The radar device may be a single-antenna Doppler radar device designed to provide velocity data rather than range data. The camera and radar device may be aligned with each other and integrated into a shared sensor housing. The system may include a broadcast camera. Furthermore, a non-transitory computer-readable medium may encode instructions that cause data processing means associated with the camera and radar device to perform operations according to any methods described herein.

[0007] The radar device can be designed to provide radar velocity data of a golf ball in flight, rather than distance data, and the combination can include: deriving the radial distance from the golf ball by integrating the velocity value obtained from modeling the velocity of the golf ball; and calculating the three-dimensional position of the golf ball in space, including using the radial distance to derive the depth distance from the golf ball, and using horizontal and vertical values ​​obtained from modeling a two-dimensional trajectory, as well as the depth distance, to derive the horizontal and vertical distances from the golf ball, at least based on the camera's focal length.

[0008] The polynomial function can be a quadratic function, and modeling the speed of a golf ball in flight using radar velocity data can include using one or more weighted models for a leading portion of the radar velocity data. Fitting the polynomial function to the first portion of the radar velocity data can include using random sampling to exclude outliers in the radar velocity data.

[0009] Modeling the speed of a golf ball in flight using radar speed data may include: fitting an exponential function to initial values ​​of the radar speed data to form an exponential model of the radar speed data; using the exponential model of the radar speed data to identify a first portion of the radar speed data that does not match the exponential model; performing a polynomial function fitting to one or more values ​​of a second portion of the radar speed data and the first portion of the radar speed data to form a polynomial model that fits the first portion of the radar speed data and satisfies the exponential model of the radar speed data at the transition point between the first and second portions of the radar speed data; and updating the exponential model by performing an exponential function fitting to the second portion of the radar speed data when additional values ​​in the radar speed data are received from the radar equipment.

[0010] Fitting a polynomial function to one or more values ​​of a second portion of the radar velocity data and the first portion of the radar velocity data may include iteratively including more values ​​from one or more values ​​of the second portion of the radar velocity data until a threshold level of continuity between the polynomial model and the exponential model is reached at a transition point. The initial values ​​for fitting the exponential function to the radar velocity data may include using random sampling to exclude outliers in the radar velocity data, and updating the exponential model includes using the exponential model to exclude one or more additional values ​​from being included in the second portion of the radar velocity data.

[0011] Modeling the two-dimensional trajectory of a golf ball in flight using two-dimensional image data may include: deriving an initial version of the two-dimensional trajectory of the golf ball in flight; receiving an initial version of the three-dimensional trajectory of the golf ball in flight from a combined view; extending the initial version of the three-dimensional trajectory in three-dimensional space according to the physical world conditions associated with the golf ball in flight to derive at least one three-dimensional position beyond the initial version of the three-dimensional trajectory; projecting the at least one three-dimensional position onto the two-dimensional image plane of the camera to locate a two-dimensional region; and processing the two-dimensional region in the two-dimensional image data to extend the two-dimensional trajectory of the golf ball in flight.

[0012] Extending an initial version of a 3D trajectory in 3D space may include: modifying the physical world conditions associated with a golf ball in flight to form two or more sets of physical world conditions; modeling the flight of the golf ball in 3D space based on the two or more sets of physical world conditions to generate two or more ball flights in 3D space; projecting each of the two or more ball flights in 3D space onto a 2D image plane of a camera to form two or more 2D paths of the golf ball in flight; comparing the two or more 2D paths with at least a portion of 2D image data corresponding to the initial version of the 2D trajectory; and, based on the comparison, selecting one of the two or more sets of physical world conditions to extend the initial version of the 3D trajectory in 3D space. Furthermore, modeling the 2D trajectory of a golf ball in flight using 2D image data may include determining the size of the 2D region based on an error estimate of the initial version of the 3D trajectory in 3D space.

[0013] Finally, the system and / or the methods and operations of the encoding instructions may include: acquiring a set of two-dimensional image data from a camera; modeling a two-dimensional trajectory of a flying object using the set of two-dimensional image data; acquiring a set of radar velocity data from a radar device associated with the camera; modeling the velocity of a flying object using the set of radar velocity data; combining the radar model of the velocity of the flying object with an optical model of the two-dimensional trajectory of the flying object to form a three-dimensional trajectory of the flying object; comparing an initial portion of the three-dimensional trajectory of the flying object with data representing a typical golf ball launch in three dimensions; and identifying the three-dimensional trajectory of the flying object as not a golf ball when the initial portion of the three-dimensional trajectory of the flying object differs from the data representing a typical golf ball launch in three dimensions by a threshold amount.

[0014] Various embodiments of the subject matter described in this specification can be implemented to achieve one or more of the following advantages. From the perspective of the radar device, a more accurate model of radar velocity data enables improved radar data modeling, and thus allows for improved hybrid tracking of objects in flight using both radar data and 2D image data. The accuracy of ball launch detection and in-flight ball tracking can be improved. Furthermore, a more accurate model of radar velocity data provides a better understanding of the ball's overall flight, which can improve the data acquired and presented for each ball's flight, such as launch velocity and launch angle.

[0015] Details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages of the invention will become apparent from the description, drawings, and claims. Attached Figure Description

[0016] Figure 1A An example of a system for performing 3D tracking of a golf ball in flight is shown.

[0017] Figure 1B and 1C This is a diagram showing 3D tracking of a golf ball in flight using a camera and Doppler radar equipment.

[0018] Figure 2A This is a schematic diagram of a data processing system.

[0019] Figure 2B This is a diagram illustrating an example of radar data processing.

[0020] Figure 3A It is shown that, as can be, Figures 1A to 2B The flowchart illustrates an example of the process of performing 3D tracking of a golf ball in flight implemented in the system.

[0021] Figure 3BAn example is shown where a golf ball is launched from a location close to but spaced apart from the radar equipment.

[0022] Figure 3C Show Figure 3B An example of the radial velocity curve of a golf ball launch.

[0023] Figure 4A This is a flowchart illustrating an example of the process of modeling the speed of a golf ball in flight using radar speed data.

[0024] Figure 4B This is a flowchart illustrating an example of the process of fitting a quadratic function to the leading part of a golf ball's flight.

[0025] Figure 5A This is a flowchart illustrating an example of the process of modeling the 2D trajectory of a golf ball in flight using 2D image data and an initial 3D trajectory derived from a combination of 2D image data and radar velocity data.

[0026] Figure 5B This is a flowchart illustrating an example of the process of extending the current version of the 3D trajectory in order to extend the 2D trajectory of a golf ball's flight.

[0027] Similar reference numerals and names in the various figures indicate similar elements. Detailed Implementation

[0028] Figure 1A An example of a system for performing 3D tracking of a golf ball in flight is shown. The system includes at least one sensor unit 110, which is communicatively coupled to one or more server computers 120, one or more personal computers 122, or a combination thereof. Sensor unit 110 includes both an object tracking camera and a radar device (e.g., a Doppler radar device). The camera and radar device can be aligned with each other and integrated into a shared sensor housing, as shown in the figure (note that alignment does not need to be precise or calibrated in any way, in some embodiments because the radar device does not need to be used to determine any orientation information, as described further in detail below). Additionally, although in Figure 1A While physical wires / cables are used to represent this, it will be understood that wireless technologies such as Near Field Communication (NFC), Bluetooth, WiFi, and / or one or more mobile phone technologies can be used to achieve communication coupling between sensor unit 110 and one or more computers 120, 122. Furthermore, in some embodiments, one or more of the computers 120, 122 may be integrated into sensor unit 110.

[0029] exist Figure 1AIn one example, the camera and radar equipment are integrated together into a common sensor unit 110 associated with the additional camera 130, and both the sensor unit 110 and the additional camera 130 are oriented towards the golfer 140, who has struck a golf ball 155 from a starting position 150 into the flight path toward the hole or other target. The additional camera 130 may be a television (TV) camera adapted to generate signals for live transmission or recording. In some embodiments, the sensor unit 110 is located in the same position as the additional camera 130 and is mechanically attached to the additional camera 130. In other embodiments, the sensor unit 110 is in the same general area as the additional camera 130, but is not specifically attached to it. Other variations are also possible, such as those described in further detail below.

[0030] In these various implementations, different types of modeling are used for different parts of the ball's flight observation, such as the initial portion 151 and the later portion 152 of the ball's flight. A hybrid of radar and optical tracking combines camera-based full-flight 2D tracking with data from radar to produce full-flight 3D tracking. Golf ball tracking for TV production typically requires low latency to allow producers to switch to the receiving camera as the ball lands, while still obtaining the trajectory from the ball's starting position 150. Therefore, rapid and accurate detection of ball launch is important in the case of TV golf ball tracking, but it is also important in other cases, such as 3D tracking of a golf ball being hit at a driving range.

[0031] In some embodiments, the distance 145 between the sensor unit 110 and the initial position 150 of the golf ball 155 is provided information (e.g., a predetermined starting point from which the golf ball 155 is hit, or a distance parameter input as input to one or more computers 120, 122). In some embodiments, the distance 145 is determined automatically (e.g., by identifying the golfer 140 in image data using computerized object classification, calculating the distance to the identified golfer 140 using trigonometry and the known or assumed height of the golfer 140, and treating the distance to the golfer 140 as equal to the distance 145 from the initial position 150 of the golf ball 155). Note that the distance 145 can also be used to provide a ball size standard to assist in identifying the golf ball 150 in the image data generated by the sensor unit 110. Furthermore, in some embodiments, the initial position 150 of the golf ball is not observed by the camera in the sensor unit 110, and the initial position 150 must be determined by backward inference of the 3D path of the ball's flight.

[0032] Furthermore, the system can be designed to convert the ball position recorded by sensor unit 110 into a corresponding position in video data acquired by additional camera 130, and using this conversion, a graphical representation of the ball's flight path can be overlaid onto the video data for transmission and / or recording. This conversion between the view of the camera at sensor unit 110 and that of additional camera 130 can be performed using homography techniques, such as those described in R. Hartley & A. Zisserman's *Multiple View Geometry in Computer Vision*, 2nd edition, Cambridge University Press, March 2004. Additionally, in some embodiments, this system and technology are combined with the 2D ball flight tracking system and technology described in U.S. Patent No. 8,077,917, published December 13, 2011, entitled "SYSTEM AND METHODS FOR ENHANCING IMAGES IN A VIDEO RECORDING OF A SPORTSEVENT," which is incorporated herein by reference.

[0033] In some implementations, the additional camera 130 is not included, and the 3D ball position determined using sensor unit 110 can be used to supplement other data or media. For example, the determined 3D ball position can be used to generate a 3D representation of the ball's flight path within a 3D computer model 124 of the environment of the golf ball 155. This environment can be a representation of the actual physical environment where the golfer 140 is standing (e.g., a specific hole on a particular golf course), or it can be a virtual environment that exists only in the computer model 124. For example, a flying drone 126 can be used to create a 3D model of a physical golf course, which is then used as a virtual environment to display 3D ball tracking data.

[0034] In any case, the object tracking camera and radar device within sensor unit 110 provide data, which is combined using one or more computers 120, 122 to provide 3D flight tracking of the golf ball 155. Data from this hybrid sensor 110 is aligned with a predefined coordinate system (e.g., a virtual golf course) so that the output data can be combined with other data (e.g., from camera 130) before viewing. Typically, the position and orientation of sensor unit 110 need to be determined. Additionally, when the object tracking camera and radar device are not co-located within a common housing 110, some reference frame alignment between the two sensors is also required, as described in further detail below.

[0035] Figure 1BIt is a diagram representing an object tracking camera 160 (e.g., a digital camera based on a single CMOS (complementary metal-oxide-semiconductor)) that generates 2D image data 165 (e.g., for 2D flight tracking of a golf ball 155). Figure 1C This is a diagram representing a Doppler radar device 180 (e.g., a single-antenna continuous wave (CW) or linear frequency modulated Doppler radar device) that generates radar data 185 of a golf ball 155 in flight, wherein the radar data 185 is used to supplement 2D image data 165 to provide a complete 3D tracking system capable of full-flight 3D tracking of the golf ball.

[0036] In some embodiments, camera 160 and radar device 180 are attached to each other such that they are both aimed in substantially the same direction (e.g., camera 160 and radar device 180 may be aligned with each other and integrated into a common housing). In other embodiments, camera 160 and radar device 180 are not attached to each other, and a specific orientation of radar device 180 is not required. For example, the position of camera 160 and the target may be known (or determined from the observation of camera 160), and assuming the relative position of radar device 180 with respect to camera 160 is known, data streams 165, 185 can be easily combined. Nevertheless, it should be noted that both camera 160 and radar device 180 are generally aimed downwards at the expected trajectory of the golf shot, such that, in contrast to the launch monitor method, each of camera 160 and radar device 180 contributes to the observation of the entire flight of the ball, where the camera is aimed at the ball and perpendicular to the expected trajectory of the golf shot (i.e., the camera only sees the small initial portion of the ball's flight).

[0037] Furthermore, while co-positioning and alignment are not required, it should be noted that co-positioning and aligning the radar device 180 and camera 160, so that they have substantially the same location and orientation, allows for a significant reduction in system complexity, as the camera 160 and radar device 180 can be viewed as a single sensor device in a single location providing two different data streams 165, 185. Moreover, the radar device 180 can be configured to communicate its readings to the camera 160 (radar data 185, for example, goes to the camera 160 via a short serial cable), and the embedded circuitry in the camera 160 can be updated (e.g., by writing new firmware for the camera 160) so that those radar readings are available through the same API (Application Programming Interface) and the same physical cables as the camera 160. This eliminates the need for a second data path from the camera + radar unit 110, which is a considerable benefit in golf ball production, where the sensor 110 can be deployed a mile or more away from the TV production facility (where the analysis computer may be located).

[0038] 2D image data 165 from camera 160 is used to identify observations 162, 164, and 166 of the golf ball in 2D frames 170 of camera 160. These observations include the initial observation 162 of the ball before it is struck, and the observations 164 and 166 of the ball in flight after it is struck. Note that for ease of description, only two in-flight observations are shown, but in practice, many in-flight observations of the ball will exist when the optical subsystem tracks the 2D flight of the golf ball as seen by camera 160 within 2D frames 170. Due to the frame rate of camera 160 (e.g., 30 frames per second, 60 frames per second, 120 frames per second, etc.), the golf ball will typically be observed hundreds of times in video stream 165 during a single 2D tracking process of a golf shot. Additionally, in some embodiments, camera 160 includes one or more inputs 161 to receive settings and / or programming to configure camera 160 for operation in a given system. For example, in some embodiments, the camera has a predetermined frame rate that is programmable via input 161.

[0039] For in-flight observations 164 and 166, 2D image data 165 is processed to determine angles 175 for in-flight observations 164 and 166. In this example, each of angles 175 (θ1 and θ2) is the angle between the center of the camera sensor and the position of the sphere observed on the camera sensor. A pinhole model is used to observe camera 160, with sensor region 170 being part of camera 160, but the camera also includes a virtual “pinhole” 172 (created by the lens of camera 160) located one focal length in front of the sensor. All rays pass through pinhole 172 before hitting the sensor, and angle 175 is the angle (in 3D) between a particular ray and a reference vertical ray that passes through pinhole 172 and hits the center of the sensor. Therefore, angle 175 is the angle between the “observation ray” associated with in-flight observations 164 and 166 and the center of sensor region 170, related to the virtual pinhole 172 created by the camera lens (the focal length of camera 160 is the distance from sensor plane 170 to pinhole 172).

[0040] Other angles can be used. The key point is that the 2D image tracking subsystem is used to determine the angle of observation toward the ball, rather than using radar data to determine such an angle. This allows the radar device 180 to be a much simpler device, potentially including radar devices that only provide velocity information (note that velocity-only radar devices are generally simpler and cheaper than radar devices that provide distance information). In some implementations, the radar device 180 is a phased array radar device, but even in such cases, it is not relied upon to accurately measure the angle toward the ball using a phased array radar device 180. This allows the radar device 180 to be miniaturized, and the antenna array (when used) does not need to have a specific width and height capable of accurately measuring angles. Since the system's angular resolution is provided by the camera 160, details about the shape of the ball in flight can be easily detected without using a larger and / or more complex radar device 180.

[0041] In some embodiments, radar device 180 is a single-antenna radar device that only provides velocity data 185 (e.g., a beat sensor that does not know the angle or distance to the object). In some embodiments, radar device 180 is a CW Doppler radar device. In some embodiments, radar device 180 provides range data 185. In various embodiments, radar device 180 may be a pulse radar device, a frequency modulated continuous wave (FMCW) radar device, a phased array radar device, etc. In some embodiments, radar device 180 provides angle data 185 about the object.

[0042] In some implementations, data 185 is collected from radar device 180 by polling radar device 180 multiple times per second. Furthermore, radar device 180 may include one or more inputs 181 to receive settings and / or programming to configure radar device 180 to operate in a given system. For example, settings can be entered into radar device 180 via input 181 such that device 180 will report a velocity reading for each measurement. As another example, when radar device 180 includes the ability to determine the angle to a ball, radar device 180 can be programmed via input 181 to control the use of angle information to aim the radar beam in the direction of the object being tracked. Note that even in this case, the angle to the ball used to determine the 3D position will still be determined using image data 165, as this data will provide better angular resolution.

[0043] Radar device 180 detects the flight of a ball by providing measurements 182, 184, 186, and 188 of the ball as it flies through the space 190 in front of radar device 180. As described above, these measurements 182, 184, 186, and 188 may include velocity measurements or distance measurements. In either case, due to the difference between the frame rate of camera 160 and the measurement timing of radar device 180, or due to gaps in the data 185 provided by radar device 180, there will generally not be radar data available at exactly the same time as camera observations 164 and 166. To solve this problem, interpolation techniques can be used on measurements 182, 184, 186, and 188 to derive intermediate data points 194 and 196 that match camera observations 164 and 166.

[0044] As will be understood, since data points 194, 196 match camera observations 164, 166, the time 198 between these data points 194, 196 is equal to the reciprocal of the predetermined frame rate of camera 160, which facilitates the combination process of the two data streams 165, 185. In some embodiments, curve fitting techniques are used to infer radar data 185 to a continuous function 192 for the time of an object (e.g., a golf ball). By fitting a curve to the data and potentially updating the curve fit as additional data comes in, radar measurements at any point in time can be readily obtained by interpolating time values ​​into the continuous function 192. Furthermore, in some embodiments, in addition to constructing a model of the ball's velocity based on radar data 185, a model of the ball's 2D path is also constructed based on 2D image data 165.

[0045] Figure 2A This is a schematic diagram of a data processing system including a data processing device 200, which can be programmed as a client or server. The data processing device 200 is connected to one or more computers 290 via a network 280. Although in Figure 2A Only one computer is shown as the data processing device 200, but multiple computers can be used. Therefore, Figure 1A One or more of the computers 120 and 122 can be implemented using the data processing device 200.

[0046] Data processing device 200 includes various software modules that can be distributed between the application layer and the operating system. These may include executable and / or interpretable software programs or libraries, including program 230, which operates as a 3D object flight tracking system that analyzes and combines camera and radar data, and modeling the radar data using a two-part radar model, as described herein. The number of software modules used can vary from one implementation to another. Furthermore, the software modules may be distributed across one or more data processing devices connected via one or more computer networks or other suitable communication networks.

[0047] The data processing apparatus 200 also includes hardware or firmware devices including one or more hardware processors 212, one or more auxiliary devices 214, a computer-readable medium 216, a communication interface 218, and one or more user interface devices 220. Each processor 212 is capable of processing instructions for execution within the data processing apparatus 200. In some embodiments, the processor 212 is a single-threaded processor or a multi-threaded processor. Each processor 212 is capable of processing instructions stored on the computer-readable medium 216 or on a storage device (such as one of the auxiliary devices 214). The data processing apparatus 200 communicates with one or more computers 290 using its communication interface 218, for example, via a network 280. Thus, in various embodiments, the described processes can run in parallel or serially on single-core or multi-core computing machines and / or on computer clusters / clouds, etc.

[0048] Examples of user interface devices 220 include displays, touchscreen displays, cameras, radar devices, speakers, microphones, haptic feedback devices, keyboards, and mice. Data processing device 200 may store instructions for implementing operations associated with the modules described herein, for example, on computer-readable medium 216 or one or more auxiliary devices 214 (e.g., one or more of floppy disk devices, hard disk devices, optical disk devices, magnetic tape devices, and solid-state storage devices).

[0049] To acquire radar and camera data, the data processing device 200 includes one or more communication interfaces 218, such as those for the above combination. Figure 1AThe described interface is a wired or wireless technology. In some embodiments, the data processing device 200 also includes a camera 160 and a radar device 180 (e.g., integrated into a single housing 110), in which case the camera 160 and radar device 180 also form part of a structure for acquiring radar and camera data. For example, in some embodiments, the single device 200 includes a single camera 160, a single-antenna Doppler radar device 180, and a computer programmed to retrieve, analyze, interpolate (as needed), and mix camera and radar data to form three-dimensional position information of an object (e.g., a golf ball) as it flies in front of the 3D flight tracking device 200.

[0050] When radar data only provides the ball's velocity, the velocity reported by the radar equipment will (due to the nature of the Doppler effect) not represent the actual speed of the ball through the air. Instead, the radar data will correspond to the radial velocity of the ball's distance from the radar equipment. Imagine a circle at a certain distance R from the radar equipment, and as the ball travels along the circumference of this circle, the radar equipment will report zero radial velocity because the ball itself has no distance from the radar equipment. However, if the ball travels on any other path, the distance from the radar equipment will change, and the radar Doppler frequency shift will detect this change in velocity, and the radar equipment will report this radial velocity.

[0051] Figure 2B This is a diagram illustrating an example of radar data processing. Radar antenna 260 is at a distance 265 from golf ball 250 at a specific time. The velocity measurement obtained using radar antenna 260 is radial velocity 232, but this is merely the radial component of the ball's velocity 234 along its trajectory 236. The velocity measurement obtained using radar antenna 260 does not include the tangential velocity component 238. Since the time between each camera observation of the ball is known (as mentioned above, it is determined by the camera frame rate), the radial velocity 232 of each observation can be used to determine how the distance between the ball 250 and the radar device (and therefore the camera, e.g., when co-located with the radar device) increases with each new observation of the ball, according to the following equation:

[0052] (1)R n =R n-1 +S n *dt,

[0053] Where R n It refers to the distance S from the radar at point n. ndt is the radial velocity at observation n, and dt is the time between any pair of consecutive observations. R0 is the distance between the ball and the radar before the ball is struck. R0 is preferably determined before the ball is struck, as it is usually the input for the distance calculation of the first observation of the ball. There are several possible ways to determine or estimate this distance. For example, the distance between the camera and the golf ball can be measured manually and input into the system before it is used, or the distance can be determined from the observed size of the ball and / or the golfer in the camera data. Furthermore, for ease of interpretation, the example of equation (1) above shows linear interpolation, but as will be understood, exponential and polynomial interpolation of the data can be used.

[0054] Once the distance R between the radar / camera and the sphere 250 is known under a given observation, for example, using equation (1) and the iterative processing in procedure 230, the 3D position of the sphere 250 can be determined using the derived radial distance 265 and details of the camera observation, such as by using the following linear algebra in procedure 230:

[0055] (2)Z n =R n *cos(θ n ),

[0056] (3)X n =Z n *dx n / f, and

[0057] (4)Y n =Z n *dy n / f,

[0058] Where n is the number of observations, f is the focal length of the camera optics (e.g., measured in pixels), θ is the angle between the center of the camera sensor and the observed position of the ball on the camera sensor, dx is the horizontal offset between the ball's position on the sensor and the sensor center (e.g., in pixels), and dy is the vertical offset between the ball's position on the sensor and the sensor center (e.g., in pixels). Note that (X n Y n Z nThe coordinate system describes the 3D position of the ball at observation number n, so the entire flight of the golf ball 250 can be described in 3D coordinates. In this example, the coordinate system will have its origin at the co-located radar and camera, the Z-axis will point in the direction of the camera, the X-axis will be horizontal from left to right, and the Y-axis will be perpendicular to the Z-axis and point downwards. Depending on the orientation of the image and camera in the final coordinate system, other arrangements are also possible. For example, the origin of the image could be in the top left corner, in which case Y points down and X points right. However, flipping only Y would also flip the system by hand, which could introduce complexity to a given implementation. For example, OpenGL has flipped Z instead of Y (i.e., the camera is looking down at the negative Z-axis, and Y points up and X points right), and while mathematical literature often uses a right-handed system, some code, such as DirectX, optionally uses a left-handed system.

[0059] As will be understood, the coordinate system is derived from the pinhole camera model described above. Additionally, in some implementations, a specific direction is chosen for the coordinate system that differs from the angle the camera is pointing towards. For example, if the trajectory data will be used by another system such as a TV broadcasting system or a virtual gaming environment system, it is important to establish a common understanding of the orientation of the coordinate system and how it relates to the real world. This can be achieved using known coordinate transformation techniques. For example, an agreement can be reached on a target object in the camera view, and the orientation of that target can then be used to define “north” in the coordinate system. Rotating the coordinate system to match the selected target may involve determining the angles of the target relative to the camera center in the horizontal and vertical planes, and then rotating the output data to compensate for these angles. Assuming the focal length and pixel size of the camera system are known, the target angles can be readily derived from the pinhole camera model. For additional details on coordinate system transformations, see R. Hartley & A. Zisserman, *Multiple View Geometry in Computer Vision*, 2nd edition, Cambridge University Press, March 2004.

[0060] While the above example focuses on a co-located camera and radar where the radar device only measures velocity, other implementations are possible. The radar device 180 can be separated from the camera 160 used for 2D tracking. When the radar device 180 and camera 160 are not at the same point, similar but slightly more complex calculations are used to mix the radar and camera data, but using the same basic principles described herein. Note that both camera 160 and radar device 180 should be positioned so that they have a considerable chance of capturing the main part of the ball's flight. In practice, sensors 160 and 180 should both be positioned near the golfer and aimed downwards "along the line of the golf shot."

[0061] When radar device 180 and camera 160 are not in the same position, their data can be combined as follows. Radar data is used to track the distance R between radar device 180 and the sphere at each time point. This information can be viewed as a 3D sphere of radius R, and the radar indicates that the sphere is located somewhere on the surface of that sphere. Simultaneously, camera 160 tracks the same sphere and knows the angles (β and γ) of the straight line between camera 160 and the sphere at each time point. If the positions of radar device 180 and camera 160 in 3D space are known and the angle pointed by camera 160 is known, the 3D position of the sphere can be determined by using appropriate mathematical calculations to find the intersection point between the sphere around radar device 180 and the straight line between camera 160 and the sphere. In some cases, the line will intersect the sphere twice, and a trial-and-error method can be used to determine which of the two intersections represents the true position of the sphere.

[0062] Figure 3A It is shown that, as can be seen, Figures 1A to 2B A flowchart illustrating an example of the process of performing 3D tracking of a golf ball in flight implemented in the system. (E.g., via data processing device 200) Two-dimensional image data of the golf ball in flight (e.g., from camera 160) and radar velocity data (e.g., from radar device 180) are obtained. In some embodiments, radar velocity data is received directly from the radar device (e.g., when the radar device is a single-antenna Doppler radar device that outputs velocity data). In some embodiments, the radar device provides range data in addition to velocity data, and the velocity data can be extracted from the radar device output.

[0063] (For example, via data processing device 200) The acquired data is analyzed to identify golf shots. In some embodiments, radar data is compared to one or more criteria to identify golf shots, such as ball velocity criteria indicating that the radar time series can only begin within a specific velocity range, corresponding to the velocity range that might be used for a golf ball that has just been hit (e.g., 22 to 112 m / s). Thus, long-distance golf shots flying at low speeds and other objects, such as birds and aircraft, can be easily ignored. Additional criteria can be used in a series of radar measurements because these measurements are received in real time. For example, the radar measurement series of golf shots should have a velocity that decreases over time, and this fact can be used to identify golf shots in the radar data.

[0064] In some implementations, Doppler radar equipment provides radar readings at a fixed rate, some of which may be empty readings, i.e., when the radar is not tracking anything. In addition to considering ball speed criteria to exclude velocity readings that did not fall within a reasonable range of a golf ball launch, as mentioned above, various factors can be considered. Velocity readings from the same golf ball should not be considered as new launches; therefore, velocity readings that continue to arrive after a verified ball launch in the radar data can be compared to previous velocity readings from that ball launch, and although those velocity readings have sufficiently similar radial velocities, the data may be part of the same ball launch. Furthermore, care should be taken in cases where the radar picks up a golf club or another ball in flight in the background to allow launches that are in the middle of several other radar readings not from a golf ball.

[0065] This can be achieved by finding a sufficiently large radial velocity jump for a new launch. Simply looking for changes in radial velocity is insufficient, as there may be two consecutive ball launches where the last reading of the first launch matches the initial launch velocity of the second. In this case, the check can be performed within the minimum duration of the empty reading between the last accepted reading and the potential launch. For radar equipment other than CW-type radars, ball launches can be detected by comparing single object velocities reported in each measurement (one reading at a time, continuously) with multiple (e.g., five or more) readings that are temporally close (e.g., checking velocity and / or spatial proximity, depending on whether the radar equipment provides range information). Thus, an FMCW-type radar that reports multiple object velocities (and distances) in each measurement can be used to detect ball launches by checking readings that are sufficiently close to each other, with a decreasing or tilting radial velocity (depending on the expected preamble effect) and an increasing radial distance within a specified threshold.

[0066] To prevent outlier-generated false ball launches, outlier rejection can be used in conjunction with a radar model for speed control, deceleration, and acceleration. Note that speed acceleration is required to handle the lead-in, as described further below. Furthermore, it should be noted that although in Figure 3A The modeling of the golf ball's velocity 308 is presented as occurring after the detection of a golf shot, but one or more models of radar data are used to identify 304 the golf shot and subsequently track it. Therefore, obtaining 302 image data and radar data can be an ongoing process of adding new data to be used by modeling 306, 308, and combination 310.

[0067] In any case, using a model of radar velocity data, each radar reading can be examined to see if it is an outlier among its neighboring readings (e.g., each reading is at most 5 meters per second away from a previous reading). If most surrounding readings could be part of a ball launch from the same model, the radar reading is a possible launch; otherwise, the radar reading can be discarded. Isolated readings can be discarded; in other words, a radar launch may require a minimum number of nearby matching readings. Note that various types of outlier rejection can be used when identifying launches and after a launch has been verified; for example, any reading that is more than two, three, or four standard deviations away from the model can be considered an outlier.

[0068] Regardless of the source of the raw data, it should be filtered to avoid corrupting the model. When tracking a golf ball using a radar-optical mixer, adding outliers to either of the two models can cause premature loss of ball tracking or simply produce an incorrect trajectory. To generate parameters associated with the trajectory, such as launch velocity and angle, as well as carry, an accurate 3D path is required. Therefore, it is important to utilize all available data without corrupting it with outliers. A single outlier at an incorrect location can cause either model to deviate from its path, and this will be very noticeable in the presented parameters.

[0069] Ideally, the ball velocity should decrease strictly with increasing t, where t is the duration from the launch time. However, the raw data is noisy, so instead of this direct check on the ball velocity standard, a downward trend can be checked. For example, a linear fit can be performed on ten uniformly distributed samples between 0 and 0.4 seconds after launch, and the slope of the line from this linear fit should decrease with increasing T. However, this is not always possible due to outliers and / or imperfect leading edges. Therefore, the standard deviation from the linear fit can be checked; if the deviation is too large after rejecting outliers, the data may be a spurious launch.

[0070] Additionally, while false ball launches can still be identified in radar data, these will be relatively rare and can be prevented from causing problems by cross-checking ball launches using image data from the camera. Camera observations of golf shots can be analyzed to identify 2D image data. For example, streaming camera data 165 can be processed in real time (e.g., using an object classifier) ​​to identify various objects (e.g., golfers and candidate balls) in the video stream, and a golf shot 304 can be identified when a series of candidate balls across a set of video frames meets or exceeds one or more established criteria for golf shots. In some implementations, the analysis involves automatically adjusting one or more thresholds (e.g., optimized thresholds per pixel) to maximize sensitivity for objects of interest (e.g., objects that look like golf balls) and real-time filtering to enable golf shot identification before all 2D image data of the golf shot is received. Furthermore, when a golf shot is identified in radar data, a signal can be sent to trigger adjustments to one or more criteria used in analyzing the 2D image data. In some implementations, potential ball launches are identified using only radar readings, and the timestamps of these launches are then provided to an optical tracking component, which can then apply its own ball launch criteria to confirm that the launch is genuine. For further details, see U.S. Patent Publication No. 2019-0111315, entitled “System and Method for Three Dimensional Object Tracking Using Combination of Radar and Image Data,” published April 18, 2019, which is incorporated herein by reference.

[0071] Modeling the 2D trajectory of a golf ball in flight using 2D image data 306. In some implementations, the model is entirely two-dimensional, where 2D tracking works by estimating where the ball should be in the 2D frame and searching for the expected area of ​​the 2D observation. While this is effective, modeling the flight of the golf ball can become difficult once it is projected onto the image plane of the camera frame. Therefore, for 2D tracking to work correctly, it should be appropriate to accept where the next observation is allowed if the flight is modeled entirely in 2D. Furthermore, because accepting outlier 2D observations can cause problems, the model can be improved by imposing stricter constraints when searching for the next observation. In some implementations, this is achieved by modeling the flight in 3D instead of 2D, as defining the shape of the golf ball in 3D is much easier and therefore allows for a much better model. The following combines... Figure 5A and 5B Further details regarding this implementation are provided.

[0072] Furthermore, note that ball observations occur at different times, as determined by the camera's frame rate and the detection of potential loss of the golf ball(s) in one or more frames. Therefore, a model of the ball's flight can be used to fill the gaps between ball observations, thus defining the 2D path (or trajectory) of the golf ball in the 2D image data. Note that this facilitates the combination of 2D image data with radar data, since camera and radar equipment do not necessarily generate data at the same rate. Therefore, there is no guarantee that 2D observations will have matching radar readings at the same point in time. Neither radar nor camera can guarantee observation of the entire flight of the golf ball.

[0073] Typically, one of the camera or radar devices loses tracking of the ball before the other, and / or for a short period during the ball's flight. Therefore, models for 2D observation and radar readings can be used. The model used for 2D observation is called the optical model, and the model used for radar readings is called the radar model. For timely synchronization of these two models, it is helpful to have a common timestamp for both 2D observation and radar readings. To obtain such a timestamp, a precise difference between the duration required for the camera to generate a frame and the duration required for the radar device to generate a radar reading is needed. This difference can be derived experimentally for specific combinations of camera and radar devices. Generally, the timing between radar readings and the camera's 2D observation should be synchronized so that the optical model and radar model can operate in a shared time dimension.

[0074] The velocity of the golf ball in flight is modeled using radar velocity data and the first and second model components 308. As mentioned above, the velocity measurement obtained using radar antenna 260 does not include the tangential velocity component 238. Therefore, the radar model models the radial velocity of the golf ball rather than its actual velocity. Consequently, a simple radar model that models the velocity of the golf ball as a descending exponential function will not be entirely accurate relative to the velocity data.

[0075] Figure 3B An example of a golf ball being launched from a position 340 adjacent to but separated from radar device 330 (e.g., ten meters away). Figure 3B In the diagram, each concentric circle 332 represents an additional three meters of radius around the radar device 330, and the golf shot 342 represents a shot from position 340 with an initial ball velocity of 75 m / s and a maximum speed of 30 m / s. 2The ball's launch is subject to resistance. As shown in the figure, the position and orientation of the golf ball 342 relative to the radar device 330 mean that even though the golf ball travels at a speed of 75 m / s immediately after being struck from position 340, its radial velocity (its velocity along the line of sight from the golf ball to the radar device 330) is almost zero. Therefore, in the short duration after launch, the radial velocity of the golf ball actually increases rapidly due to the geometry of its flight relative to the radar device. Due to the geometry of its flight relative to the radar device, the radar velocity data behaves very much like an exponential model, except for the short duration after launch.

[0076] Figure 3C It shows Figure 3B An example of the radial velocity curve of a golf ball launch at 350. Figure 3C In the diagram, the X-axis represents time in seconds, and the Y-axis represents radial velocity in meters per second (m / s). As shown, in the first 0.4 seconds of the ball's flight, the radial velocity increases rapidly before it begins to decrease slowly. This initial change in radial velocity from an exponential function immediately after launch is referred to here as the leader, and its effect on the radar model is called the leader effect. Note that in this example, the leader lasts for approximately 0.4 seconds, but the length of the leader will vary depending on the relative positions of the launch location and the radar equipment, as well as the direction of the golf ball's impact. The leader effect is most pronounced when the ball is close to the radar and is generally insignificant after less than one second.

[0077] Typically, each golf shot will have a first part 352 that generates a certain amount of leading effect and a second part 354 that can be modeled as an exponential curve. Therefore, radar models can be constructed from two simpler models: the leading model and the exponential model. (See again...) Figure 3A Modeling the velocity of a 308 golf ball can include fitting a polynomial function to a first portion of the radar velocity data and an exponential function to a second portion of the radar velocity data. In some implementations, the polynomial function is a quadratic function (quadratic polynomial function). In some implementations, higher-order polynomial fitting is used, provided sufficient radar readings are available.

[0078] To use the two models as a single model, a transition point (also known as the leader split point) from the leader to the exponential model should be derived. When a quadratic polynomial is used to model the leader for each golf shot, the leader portion of the model may not fit radar readings well near the transition point. Therefore, to achieve a cleaner transition between the two models, the leader can “borrow” several radar readings from the exponential model. Note that this borrowing means that radar readings from the second part are used to form the leader model, even if they are not part of the leader. In some implementations, modeling 308 includes using a weighted model of the leader portion used for radar velocity data. Using a weighted model as the leader model is useful for obtaining a clean transition to the exponential model and forcing a better fit.

[0079] For example, a weighted model could be a model that fits radar readings to a (quadratic) function representing the leader, so that (1) the leader model is aligned with the exponential model at the transition point, and (2) not only the data points near the transition point, which typically do not produce a usable function, are used. Therefore, the weighted model (1) places higher values ​​on data points closer to the transition point, and (2) also places higher values ​​on later data points on the earliest data points, since these have the greatest impact on the leader's outcome. Thus, the values ​​of radar readings near the launch point and near the transition point are greater than those of data points in between. A weighted fitting model can be used to achieve this by imposing more weights on these data points; thus, when a fit is produced, the algorithm will preferentially use these radar readings. An example of such a weighted fitting is weighted least squares. The following is combined with... Figures 4A to 4B Further details are provided regarding modeling radar velocity data.

[0080] A radar model of the velocity of a flying golf ball is combined with an optical model of its two-dimensional trajectory to form a three-dimensional trajectory. This may involve integrating the velocity values ​​obtained from the radar model of the golf ball's velocity to derive the radial distance to the golf ball and calculating the golf ball's three-dimensional position in space. This calculation may include using the radial distance to derive the depth distance to the golf ball, and using horizontal and vertical values ​​(angles or data representing angles) obtained from the optical model, along with the depth distance, to derive the horizontal and vertical distances to the golf ball, at least based on the camera's focal length.

[0081] This process can be understood based on the ray and distance. The internal camera parameters can be used with 2D pixel coordinates to generate a 3D ray passing through the origin in the camera's coordinate system. In general, this ray has at most two 3D points at any given distance from the radar. In the simple case, where the camera and radar are positioned in the same location, there are exactly two such 3D points, only one of which is in front of the camera. Therefore, for each 2D observation and distance pair, there exists a single possible 3D observation. If distortion in the camera lens is ignored, at least the chip size, focal length, and pixel size of the (internal) camera parameters are used. Further details are provided above in conjunction with equations (1) through (4). Furthermore, the radar device does not need to be co-located with the camera.

[0082] Output 312 the 3D trajectory of a golf ball flying in 3D space for display. This can involve outputting the 3D trajectory directly to a display device, or sending the 3D trajectory to another computer or process that will display information about the 3D trajectory. For example, the 3D trajectory data can be output to one or more computers 120, 122 to be included in a TV broadcast of the golf ball's flight. As another example, the 3D trajectory data can be output to one or more computers 120, 122 to be included in a virtual environment. The virtual environment can be, for example, a computer model of a golf course generated by drone 126, which can be displayed as part of a TV broadcast or as part of a simulated golf game displayed on a display device at a golf driving range facility, where data from actual golf shots are integrated into the simulated golf game. Furthermore, in some cases, the simulated golf game does not need to match or include any model of a traditional golf course in a traditional golf game. The virtual environment can represent a completely artificial game world, where data from actual golf shots are integrated into the 3D space of the artificial game world.

[0083] Furthermore, as mentioned above, acquiring image data 302 and radar data can be an ongoing process of continuously adding new data to be used by modeling 306, 308, and combination 310. Similarly, output 312 can be an ongoing process of continuously outputting 3D trajectory data while actively tracking a golf shot using modeling 306, 308, and combination 310. Likewise, the output data generated by the operation of modeling 306, 308, and / or combination 310 can be used as input to golf shot launch detection 304 before the golf ball launch is verified, thereby improving the ability to detect false ball launches (detected flying objects that are not golf balls).

[0084] For example, (by acquiring 302 2D image data and radar data, modeling the 2D trajectory of a flying object using the 2D image data 306, modeling the velocity of the flying object using radar velocity data 308, and combining the radar model of the object's velocity with the optical model of the object's 2D trajectory 310), the initial portion of the 3D trajectory can be compared with a ball launch standard 360. This ball launch standard represents one or more features of a typical golf ball launch in three dimensions, and is therefore different from ball launch standards applied only in 2D space from image data from a camera or applied to velocity data from a radar device. If the initial portions of the 3D trajectories differ by at least a threshold amount 362, then this identifies the 3D trajectory of the flying object as not a golf ball. Therefore, based on the standard applied in 3D space, a previously identified ball launch can be rejected as a golf ball launch, or a previously identified ball launch can be verified as a golf ball launch based on the standard applied in 3D space.

[0085] For example, a comparison can be made by calculating the angle between two rays: a first ray r1, generated from 3D data points p0 and p1, and a second ray r2, generated from 3D data points p1 and p2. Here, p0 is the launch position, p1 is x seconds after launch, and p2 is 2x seconds after launch. If the angle between r1 and r2 exceeds y degrees, the trajectory is not straight enough to be considered a golf shot, and the object is identified as not a golf ball. If the angle between r1 and r2 is less than or equal to y degrees, the trajectory is straight enough to be considered a golf shot, and the object is identified as a golf ball and further tracked. In some implementations, x is set between 0.2 seconds and 0.25 seconds, and y is set between 1 degree and 5 degrees. Note that the x and y parameters can be set based on a given implementation after appropriate tuning and testing. Furthermore, these values ​​can change for a given golf shot after receiving further data (such as when 3D information from further away (e.g., from the fairway on the golf course) indicates that the shot is not long), meaning a larger angle can be used to advance the shot.

[0086] Furthermore, if the 3D path has been determined, it is easier to interpolate between two observations or to infer where the ball should be in 2D and 3D at any given time point (e.g., backwards to the launch position). It is also simpler to obtain a good estimate of the launch time using the determined 3D path when the launch position is known, and knowing the launch time is important when calculating launch velocity, as the ball decelerates at approximately 45 miles per second, so a 22-millisecond error in launch time results in an error of approximately one mile per hour, even with a perfect understanding of the ball's flight path.

[0087] In addition to using output data from process operations 306, 308, and 310 to improve golf shot detection 304 as discussed above, output data 370 from one or more of process operations 302, 306, 308, and 310 can be used as input to improve the functionality of one or more other process operations 302, 306, 308, and 310. For example, a 2D ball tracker can use radial velocity readings from a radar device (instead of a fixed parameter ball velocity) to form a rapid estimate of the ball's motion in 3D, which can then be projected back onto the camera's 2D image plane to help identify the ball and track its motion in 2D. The following combines... Figure 5B Further details are provided regarding the use of 3D motion estimation to perform 2D tracking; however, it should be noted that it is not necessary to construct a high-level and accurate model of the radar readings for this radar velocity data to be useful for a 2D tracker.

[0088] For example, radar readings can be integrated to generate a sphere around the radar device, where the sphere should be positioned somewhere on the surface, and the camera's projection matrix can be used to extend the 2D observation into rays extending from the camera to generate a 3D observation where the rays intersect the sphere. These 3D observations can then be used with an existing physical model of the sphere's flight in 3D. Note that the set of spheres used may vary because the integration should begin at the time of the sphere's flight. In other embodiments, a high-level and accurate model of the radar readings is used to determine the velocity of the sphere used in 2D tracking.

[0089] Figure 4A This is a flowchart illustrating an example of the process of modeling the speed of a golf ball in flight using radar velocity data (i.e., the example of modeling 308). An exponential function is fitted to the initial values ​​of the radar velocity data to form an exponential model of the radar velocity data. Fitting the exponential function to the initial values ​​of the radar velocity data may involve using random sampling to exclude outliers in the radar velocity data. For example, the Random Sample Consensus (RANSAC) algorithm can be used to avoid outliers and help improve the fit. Other outlier rejection methods, such as heuristics or hard-coded outlier rejection, can be used. Moreover, if the fit produces an unreasonably large number of outliers, for example, more than one-third or more than half of the input data points, the fit can be rejected.

[0090] This (initial) exponential model uses the 405 radar velocity data to identify the first portion of the radar velocity data (which does not match the exponential model), i.e., to derive the leader breakpoint. To place the leader breakpoint, a starting position is needed. An exponentially fitted model can be used to derive such a position, starting far enough back that a leader is impossible. For example, in most implementations, the leader effect is negligible 0.5 seconds after ball launch, so it can be assumed that the breakpoint occurs before that time. Once the initial exponential model is derived, it is extended backwards to the leader while tracking how well the model matches the next point. An example is shown in tabular form below.

[0091] Matched: 0 0 1 0 0 0 1 1 0 1 1 1 1 1 1 1 Index: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

[0092] Table 1

[0093] In Table 1, the first row contains 1 if the exponentially fitted model matches the radar readings, and 0 otherwise. The second row contains the data point indices. In this case, the leading breakpoint is simply defined at approximately index 6, where one-fifth of the radar readings match the exponentially fitted model before, and nine-tenths match after.

[0094] Once the leading portion (405) is identified, a polynomial function (e.g., a quadratic function) is fitted to this first portion of the radar velocity data (410) to form a polynomial model fitting the first portion of the radar velocity data. The fitting of 410 can also employ outlier rejection, such as random sampling, to exclude outliers in the radar velocity data. For example, the RANSAC algorithm can be used to construct the leading model (410). Other outlier rejection methods, such as heuristics or hard-coded outlier rejection, can be used. Furthermore, if the fitting produces an unreasonably large number of outliers, for example, more than one-third or more than half of the input data points, the fit can be rejected.

[0095] Furthermore, fitting 410 may include one or more values ​​from the second (exponential) portion of the radar velocity data to form a polynomial model that fits the first portion of the radar velocity data and also satisfies the exponential model of the radar velocity data at the transition point between the first and second portions. Using several additional radar readings from the second (exponential) portion of the radar velocity data in fitting 410 helps ensure a close match between the two models (the polynomial model and the exponential model) at the leading split point.

[0096] The 420 exponential model is then updated by fitting an exponential function to a second portion of the radar velocity data. This involves receiving additional values ​​from the radar velocity data from the radar equipment and adding them to the second portion. Furthermore, in some implementations, fitting the 420 exponential function to the second portion of the radar velocity data also includes using random sampling to exclude outliers in the radar velocity data. For example, the RANSAC algorithm can be used to avoid outliers and help improve the fit. Other outlier rejection methods, such as heuristics or hard-coded outlier rejection, can be used. Moreover, if the fit produces an unreasonably large number of outliers, for example, more than one-third or more than half of the input data points, the fit can be rejected. In any case, the two models (the multinomial model and the exponential model) are combined to form the complete radar model.

[0097] Furthermore, this complete radar model can be used to define outliers by calculating the standard deviation of the added radar readings. Using multiples of the standard deviation is a quick way to locate outliers; multiples of four make false positives very unlikely (approximately 0.006% to 4% chance), multiples of three make false positives impossible (approximately 0.3% to 6.25% chance), and multiples of two make false positives slightly unlikely (approximately 5% to 11.1% chance). Here, the range of chance depends on how close the normal distribution of the data obtained is to the normal distribution of the ideal model, and in practice, Chebyshev's inequality is a good approximation of the chance. Note that adjusting the cutoff value allows for a careful balance between rejecting normal values ​​and accepting outliers, and using two to four standard deviations as the cutoff value typically results in 11.1% to 4% of normal values ​​being incorrectly removed as outliers.

[0098] Standard deviation is typically a very small number, so having a lower bound for this constraint is reasonable; similarly, the constraint has an upper bound to avoid following slow trends, such as transitions to a leading model. For added future points, this constraint can then be used to determine if they match the model, and nearby radar readings generally reject outliers. In addition to using standard deviation to check for outliers, the process can check the total number of outliers (rejected data points) when deciding whether the fit is acceptable. For example, it can be checked whether the number of outliers is less than half or less than one-third of all data points input to the fit, and also the standard deviation of the outliers is checked to determine how good the fit is, and therefore whether to accept it.

[0099] Figure 4BA flowchart showing an example of the process of fitting a quadratic function to the leading portion of a golf ball flight (i.e., an example of the fit 410). In this example, fitting a polynomial function to one or more values of the second portion of the radar velocity data and the first portion of the radar velocity data involves iteratively including more values from one or more values of the second portion of the radar velocity data until a threshold level of continuity between the polynomial model and the exponential model is reached at the transition point. Additionally, in some cases, since the lead may be much less than half a second long, there may not be enough radar readings in the leading portion to separately build a polynomial model. Therefore, in some embodiments, a check may be made to see if there are enough data points to build a leading model before borrowing any data points from the second portion of the radar velocity data.

[0100] To create a leading model, one or more values can be added 450 from the second portion of the radar velocity data to the first portion. A quadratic function can be fit 455 to the leading portion plus the added data values. Continuity between the two radar models (quadratic model and exponential model) is checked 460 at the transition point, and more data values are added 450 until a threshold level of continuity is achieved. For example, the nominal values and first derivatives (slopes) provided by the two models at the transition point can be compared to see if they are within a threshold distance (s) of each other. In some embodiments, the goal is a single “virtual” function, i.e., C 0 and C 1 continuous, even though it is modeled as two separate functions.

[0101] For example, let e(t) be the exponential model, p(t) be the leading model, and t0 be the transition point. Then, the combined model can be f(t) = {p(t) if t < t0 | e(t) if t >= t0}. It is also assumed that both e(t) and p(t) are C 0 and C 1 continuous. If p(t0) = e(t0), then C 0 for f is achieved. But to avoid a sharp edge in f at t0, C 1 continuity should also be sought, which means p’(t0) = e’(t0), where p’ and e’ are the derivatives of p and e, respectively. But since these are two completely different functions, an exact match is not required for either of the continuity checks. Therefore, for sufficiently small values of ε0 and ε1, the check can be |p(t0) - e(t0)| < ε0 and |p’(t0) – e’(t0)| < ε1, where the discontinuity is not as visible. Note that higher-order continuity checks are also possible. The continuity check ensures that there will be a smooth transition between the two models. Then, the quadratic model of the leading portion of the radar velocity data is output 465 for subsequent use in modeling the full flight of the golf ball.

[0102] Figure 5A This is a flowchart illustrating an example of the process of modeling the 2D trajectory of a golf ball in flight using 2D image data (i.e., the example of modeling 306). An initial version of the 2D trajectory of the golf ball in flight is derived in 500. As mentioned above, this involves identifying the ball observation in the camera data of the next frame. Furthermore, note that optical tracking should be highly accepting of new ball observations to function properly, which increases the risk of adding outliers to the optical model. Therefore, in some implementations, outlier rejection for optical tracking is improved by using a 3D model instead of the optical model to detect outliers, since the 3D model better describes how the ball moves.

[0103] Therefore, an initial version of the three-dimensional trajectory of a golf ball in flight is received by combining camera data and radar data. As mentioned above, the current radar model can be used to extend 2D ball observations to 3D observations, and these 3D observations then form the basis for a 3D path, which is more easily extended to the future using a simple 3D model. This model only needs to be accurate for a small fraction of a second in the future to locate the next 2D observation, making it less affected by physical world conditions such as wind. However, in some implementations, the initial version of the three-dimensional trajectory in three-dimensional space is extended according to the physical world conditions associated with the golf ball in flight to derive at least one three-dimensional position beyond the initial version of the three-dimensional trajectory.

[0104] The at least one 3D position is projected 515 onto the camera's 2D image plane to locate a 2D region. The 2D region in the 2D image data 520 is processed to expand the 2D trajectory of the golf ball in flight. Note that this may include determining the size of the 2D region based on an estimate of the error of an initial version of the 3D trajectory in 3D space. Since the ball's flight is modeled in 3D, this model can be used to infer where the ball should be in 3D. However, since the model is not perfect, the region around the modeled position is used based on the expected maximum error in the model. The modeled position and its error produce a 3D shape, which can be projected onto a 2D region in the camera's image plane using the camera's intrinsic parameters. Note that since the 3D position is in the camera's coordinate system at this stage, only the intrinsic parameters are needed. The 2D shape can then be searched for possible 2D views of the golf ball. The algorithm can preferentially select the 2D view closest to the modeled position, and image processing time can be reduced since it is not necessary to search all portions of each image frame.

[0105] Figure 5BThis is a flowchart illustrating an example of the process of extending the current version of the 3D trajectory to facilitate the extension of the 2D trajectory of the golf ball's flight (i.e., examples of extension 510, projection 515, and processing 520). Modification 550 modifies the physical world conditions associated with the golf ball in flight to form two or more sets of physical world conditions. Physical world conditions may include wind speed and direction, ball spin, ball axis of rotation, temperature, humidity, etc. Note that some physical world conditions, such as wind speed and direction, may have initial values ​​set by data received from sensors at the golf ball's impact point.

[0106] Modeling the flight of a golf ball in three-dimensional space based on two or more sets of physical world conditions 555 generates two or more ball flights in three-dimensional space. This involves using a physical modeler that knows how the ball should fly through the air given a set of physical world conditions. Each of the two or more ball flights in three-dimensional space is projected 560 onto a two-dimensional image plane of a camera to form two or more two-dimensional paths of the golf ball in flight. The two or more two-dimensional paths are compared with at least a portion of two-dimensional image data corresponding to an initial version of the two-dimensional trajectory, and (based on the comparison) one of the two or more sets of physical world conditions is selected to extend the initial version of the three-dimensional trajectory in three-dimensional space.

[0107] Note that the choice here doesn't have to be a formal choice of one 2D path being superior to another. Instead, the algorithm can formulate many hypotheses (options about the conditions of the physical world) and continue processing all existing hypotheses until they "fail" due to the input data. Note that a hypothesis can fail for a variety of different reasons, but it will eventually fail if nothing is observed in the expected (inferred) region proposed by the given hypothesis. Furthermore, each hypothesis can fail or generate further hypotheses, and most hypotheses will eventually fail, usually leaving one surviving best hypothesis for the ball's flight.

[0108] For example, each hypothesis can have its 2D projection checked 565 to determine if the criteria for finding a ball observation are met for the current 2D image data. If so, then 570 more 2D image data and radar velocity data are obtained to model the flight of the golf ball in 3D based on that hypothesis. If the criteria for a given hypothesis are not met 565, then a check 575 can be performed to determine if the given hypothesis has failed. For example, if no ball observation was found for the given hypothesis in the last four image frames, then the hypothesis has failed. In this case, 580 the hypothesis can be removed from two or more sets of physical world conditions before the algorithm proceeds to obtain 570 more 2D image data and radar velocity data. Note that this process can also be used to discard spurious launches; if a 3D observation does not produce a valid 3D trajectory, it can be discarded as a spurious launch.

[0109] If a given hypothesis is not failed 575, but does not meet 565 criteria, 550 physical world conditions can be modified to attempt to improve the hypothesis regarding the 3D ball's flight. Note that weather conditions can make it difficult to infer the 3D path. Therefore, in windy weather, early loss of tracking of the ball can be disastrous for determining flight, and thus simultaneously checking many different hypotheses about various physical world conditions (such as different wind speeds and directions, including changes in these wind speeds and directions during different parts of the ball's flight) can maximize the system's ability to accurately track the ball. In some implementations, the current hypothesis is modified and checked relative to previous camera frames. In some implementations, a hypothesis triggers one or more new hypotheses because the current hypothesis might perform better with more data coming in, rather than being removed after more data comes in 580. Furthermore, in some implementations, checks are performed to ensure that the number of hypotheses does not become too high, otherwise the computer will not be able to handle them. This is most important for shorter paths, as there should not be many persistent hypotheses, such as leaves, birds, flies, or anything else that can move at any point in time like a golf ball in 2D and happen to coincide with reasonable radar readings, but ultimately any of these data sources should cause the hypothesis to fail. Therefore, if the system receives 2D observations and radar readings that form a reasonable 3D golf ball path that takes more than a second, it is unlikely to be anything other than a real golf ball.

[0110] Embodiments of the subject matter and functional operation described in this specification can be implemented in digital electronic circuits or computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or one or more combinations thereof. Embodiments of the subject matter described in this specification can be implemented using one or more modules of computer program instructions encoded on a computer-readable medium for execution or control of its operation by a data processing apparatus. The computer-readable medium can be an article of manufacture, such as a hard disk drive in a computer system or an optical disc sold through retail channels, or it can be an embedded system. The computer-readable medium can be separately acquired and subsequently encoded with one or more modules of computer program instructions, such as by transmitting one or more modules of computer program instructions over a wired or wireless network. The computer-readable medium can be a machine-readable storage device, a machine-readable storage substrate, a storage device, or a combination of one or more of these.

[0111] The term "data processing apparatus" encompasses all means, devices, and machines used for processing data, including, for example, programmable processors, computers, or multiple processors or computers. In addition to hardware, the apparatus may also include code that creates an execution environment for the computer program in question, such as code that constitutes processor firmware, protocol stacks, database management systems, operating systems, runtime environments, or combinations thereof. Furthermore, the apparatus can employ a variety of different computing model infrastructures, such as network services, distributed computing, and grid computing infrastructures.

[0112] Computer programs (also known as programs, software, software applications, scripts, or code) can be written in any suitable programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any suitable form, including as standalone programs or as modules, components, subroutines, or other units suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program can be stored as part of a file containing other programs or data (e.g., one or more scripts stored in a markup language document), a single file dedicated to the program in question, or multiple coordinating files (e.g., files storing one or more modules, subroutines, or portions of code). Computer programs can be deployed to execute on one computer or on two or more computers located at a site or distributed across multiple sites and interconnected by a communication network.

[0113] The processes and logic described in this specification can be executed by one or more programmable processors, which execute one or more computer programs to perform functions by manipulating input data and generating output. The processes and logic can also be executed by dedicated logic circuitry, and the device can be implemented as dedicated logic circuitry, such as an FPGA (Field-Programmable Gate Array) or an ASIC (Application-Specific Integrated Circuit).

[0114] As an example, processors suitable for executing computer programs include general-purpose microprocessors and special-purpose microprocessors, as well as any one or more processors in any kind of digital computer. Typically, the processor receives instructions and data from read-only memory or random access memory, or both. The basic components of a computer are a processor for executing instructions and one or more storage devices for storing instructions and data. Typically, a computer will also include or be operatively coupled to one or more mass storage devices (e.g., magneto-optical, magneto-optical, or optical discs) for storing data, to receive data from, or to transfer data to, or both. However, a computer does not necessarily need to have such devices. Furthermore, to name just a few, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive). Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, for example, semiconductor storage devices (e.g., EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory)) and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROMs and DVD-ROMs. Processors and memory may be supplemented by or integrated into dedicated logic circuitry.

[0115] To provide interaction with the user, embodiments of the subject matter described in this specification can be implemented on a computer having: a display device, such as an LCD (liquid crystal display), OLED (organic light-emitting diode), or other monitor, for displaying information to the user; and a keyboard and pointing device, such as a mouse or trackball, through which the user can provide input to the computer. Other types of devices may also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including sound, speech, or tactile input.

[0116] A computing system may include clients and servers. Clients and servers are typically geographically separated and typically interact via a communication network. The client-server relationship is created by computer programs running on their respective computers and having a client-server relationship with each other. Embodiments of the subject matter described in this specification can be implemented in a computing system that includes back-end components (e.g., as a data server), or middleware components (e.g., an application server), or front-end components (e.g., a client computer with a graphical user interface or a web browser through which a user can interact with embodiments of the subject matter described in this specification), or any combination of one or more such back-end components, middleware components, or front-end components. Components of the system may be interconnected via digital data communication (e.g., a communication network) of any form or medium. Examples of communication networks include local area networks (“LANs”) and wide area networks (“WANs”), the Internet (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks).

[0117] While this specification contains numerous implementation details, these should not be construed as limiting the scope of the invention or its claimable content, but rather as descriptions of features specific to particular embodiments of the invention. Certain features described in this specification within the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although the features described above may be described as functioning in certain combinations and even initially claimed in this way, in some cases one or more features of the claimed combination may be removed from the combination, and the claimed combination may be for sub-combinations or variations thereof. Therefore, unless expressly stated otherwise, or unless expressly stated otherwise by knowledge of ordinary skill in the art, any feature of the embodiments described above may be combined with any other feature of the embodiments described above.

[0118] Similarly, although operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring such operations to be performed in the specific order shown or sequentially, or performing all shown operations to achieve the desired result. In some cases, multitasking and / or parallel processing may be advantageous. Furthermore, the separation of various system components in the embodiments described above should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0119] Therefore, specific embodiments of the invention have been described. Other embodiments are within the scope of the following claims. For example, the above description focuses on tracking golf shots, but the described systems and techniques can also be applied to tracking other types of objects in flight, such as baseball or frisbee shots, and non-sports applications. Furthermore, in some embodiments, tracking an object "in flight" may include tracking the object as it bounces back and / or rolls along the ground. Additionally, the actions described in the claims may be performed in different orders and the desired result may still be achieved.

Claims

1. A tracking method comprising: obtaining two-dimensional image data of a golf ball in flight, the two-dimensional image data derived from a camera; modeling a two-dimensional trajectory of the golf ball in flight using the two-dimensional image data; obtaining radar velocity data of the golf ball in flight, the radar velocity data derived from a radar device associated with the camera; modeling a radial velocity of the golf ball in flight using the radar velocity data, wherein, modeling the radial velocity of the golf ball comprises fitting a polynomial function to a first portion of the radar velocity data associated with an initial portion of the flight of the golf ball and fitting an exponential function to a second portion of the radar velocity data associated with a later portion of the flight of the golf ball, wherein the first portion is associated with a rapid increase in the radial velocity before the radial velocity begins to decrease; combining the modeled velocity of the golf ball in flight with the modeled two-dimensional trajectory of the golf ball in flight to form a three-dimensional trajectory of the golf ball in flight; and outputting the three-dimensional trajectory of the golf ball in flight in three-dimensional space for display.

2. The tracking method of claim 1, wherein, the radar device is designed to provide the radar velocity data rather than range data of the golf ball in flight, and the combining comprises: deriving a radial distance of the golf ball by integrating velocity values obtained from the modeled velocity of the golf ball; and computing a three-dimensional position of the golf ball in space, the computing comprising: deriving a depth distance from the golf ball using the radial distance, and deriving a horizontal distance and a vertical distance from the golf ball based on at least a focal length of the camera using horizontal and vertical values obtained from the modeled two-dimensional trajectory and the depth distance.

3. The tracking method of claim 1, wherein, the polynomial function is a quadratic function, and modeling a velocity of the golf ball in flight using the radar velocity data comprises using one or more weighted models for a leading portion of the radar velocity data.

4. The tracking method of claim 1, wherein, fitting the polynomial function to the first portion of the radar velocity data comprises: using random sampling to exclude outliers in the radar velocity data.

5. The tracking method of claim 1, wherein, modeling the velocity of the golf ball in flight using the radar velocity data comprises: fitting the exponential function to initial values of the radar velocity data to form an exponential model of the radar velocity data; using the exponential model of the radar velocity data to identify the first portion of the radar velocity data that does not match the exponential model; performing a fitting of the polynomial function to the one or more values of the first portion of the radar velocity data and to the second portion of the radar velocity data to form a polynomial model that fits the first portion of the radar velocity data and that satisfies the exponential model of the radar velocity data at a transition point between the first portion and the second portion of the radar velocity data; and updating the exponential model by performing the fitting of the exponential function to the second portion of the radar velocity data as additional values in the radar velocity data are received from the radar device.

6. The tracking method of claim 5, wherein, performing a fitting of the polynomial function to the one or more values of the first portion of the radar velocity data and to the second portion of the radar velocity data includes: iteratively including more values of the one or more values of the second portion of the radar velocity data until a threshold level of continuity between the polynomial model and the exponential model is reached at the transition point.

7. The tracking method of claim 5, wherein, fitting the exponential function to the initial values of the radar velocity data includes using random sampling to exclude outliers in the radar velocity data, and updating the exponential model includes using the exponential model to exclude one or more of the additional values from being included in the second portion of the radar velocity data.

8. The tracking method of claim 1, wherein, modeling the two-dimensional trajectory of the golf ball in flight using the two- dimensional image data includes: deriving an initial version of the two-dimensional trajectory of the golf ball in flight; receiving an initial version of the three-dimensional trajectory of the golf ball in flight from the combination; extending the initial version of the three-dimensional trajectory in three-dimensional space according to physical world conditions associated with the golf ball in flight to derive at least one three-dimensional position beyond the initial version of the three-dimensional trajectory; projecting the at least one three-dimensional position into a two-dimensional image plane of the camera to locate a two-dimensional region; and processing the two-dimensional region in the two-dimensional image data to extend the two-dimensional trajectory of the golf ball in flight. extending the initial version of the three-dimensional trajectory in three-dimensional space includes:

9. The tracking method of claim 8, wherein, modifying the physical world conditions associated with the golf ball in flight to form two or more sets of physical world conditions; modeling the flight of the golf ball in three-dimensional space according to the two or more sets of physical world conditions to generate two or more ball flights in three-dimensional space; projecting each of the two or more ball flights in three-dimensional space into the two- dimensional image plane of the camera to form two or more two-dimensional paths for the golf ball in flight; comparing the two or more two-dimensional paths to at least a portion of the two- dimensional image data corresponding to the initial version of the two-dimensional trajectory; and ​ ​ Based on the comparison, one of the two or more groups of physical world conditions is selected for use in extending the initial version of the three-dimensional trajectory in three-dimensional space.

10. The tracking method of claim 8, wherein, Modeling the two-dimensional trajectory of the golf ball in flight using the two-dimensional image data includes: Determining a size of the two-dimensional region based on an estimate of error for the initial version of the three-dimensional trajectory in three-dimensional space.

11. The tracking method of claim 1, including: obtaining a set of two-dimensional image data from the camera; modeling a two-dimensional trajectory of an object in flight using the set of two-dimensional image data; obtaining a set of radar velocity data from the radar device associated with the camera; modeling a velocity of the object in flight using the set of radar velocity data; combining the radar model of the velocity of the object in flight with the optical model of the two-dimensional trajectory of the object in flight to form a three-dimensional trajectory of the object in flight; comparing an initial portion of the three-dimensional trajectory of the object in flight to data representing a typical golf ball launch in three dimensions; and when the initial portion of the three-dimensional trajectory of the object in flight differs from the data representing the typical golf ball launch in the three dimensions by a threshold amount, identifying the three-dimensional trajectory of the object in flight as not being a golf ball.

12. A tracking system, including: a camera; a radar device; and a computer including a hardware processor and a memory coupled with the hardware processor, the memory encoding instructions configured to cause the hardware processor to perform operations of the tracking method of any of claims 1-11.

13. The tracking system of claim 12, wherein the radar device is a single-antenna Doppler radar device designed to provide velocity data rather than range data.

14. The tracking system of claim 12, wherein the camera and the radar device are aligned with each other and integrated into a shared sensor housing.

15. The tracking system of claim 12, including a broadcast camera.

16. A non-transitory computer-readable medium encoding instructions that cause a data processing apparatus associated with a camera and a radar device to perform operations of the tracking method of any of claims 1-11. ​

Images