3D Object Tracking Using Unverified Detections Registered by One or More Sensors
Patent Information
- Application Number
- JP2023526086
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2020-11-03
- Filing Date
- 2021-10-28
- Publication Date
- 2024-11-06
- Estimated Expiration
- 2041-10-28
AI Technical Summary
Existing systems for tracking moving objects, such as golf balls in flight, suffer from high false positive rates, leading to overloading and inefficiencies in 3D tracking systems due to the use of unverified object detections, which results in a significant portion of input data being inaccurate.
A method and system that utilizes data from multiple sensors, including cameras and radar devices, to tolerate a high number of false positives while minimizing false negatives by applying filters and motion models to form hypotheses of object movement in 3D space, using techniques like stereo pairing and triangulation to accurately track objects.
The system effectively minimizes false negatives and maintains robust tracking performance despite high false positive rates, ensuring reliable detection and accurate 3D tracking of moving objects by leveraging multiple sensors and advanced filtering techniques.
Abstract
Description
[Technical Field]
[0001] This specification relates to tracking a moving object, such as a golf ball in flight, using data obtained from different sensors, which may employ different sensor technologies. [Background technology]
[0002] Systems and methods for tracking the flight of a golf shot using sensors include launch monitors, full-flight two-dimensional (2D) tracking, and full-flight three-dimensional (3D) tracking. Commonly used sensor types are cameras, Doppler radar, and phased array radar. Launch monitor methods are based on measuring a set of parameters that can be observed during the swing of a golf club and during the first few inches of ball flight after the club strikes the ball. The measured parameters are then used to estimate the expected ball flight using mathematical and physical modeling.
[0003] In contrast, full-flight 3D tracking systems are characterized by their design, which attempts to track the entire flight of a golf shot rather than extrapolating from launch parameters. Furthermore, full-flight 2D tracking systems track the shape of a golf shot from a particular angle, but do not generate 3D information and generally cannot be used to determine important parameters such as the distance traveled by the ball. Full-flight 3D tracking using a combination of camera and Doppler radar data is described in U.S. Pat. No. 10,596,416. Finally, full-flight 3D tracking using stereo cameras with synchronized image frame capture has been described as potentially usable in some situations for 3D tracking of objects. Summary of the Invention
[0004] This specification describes techniques related to tracking a moving object in three-dimensional (3D) space, such as a golf ball in flight, using data obtained from two or more sensors, such as cameras, radar devices, or a combination thereof. 3D object tracking uses unverified object detection registered by two or more sensors. By allowing more false positives of detected objects to be used as input to 3D tracking in 3D space, the number of false negatives of detected objects can be minimized, thereby ensuring that small objects (e.g., golf balls) are reliably detected even when their distance from (and / or velocity relative to) the sensor makes it difficult for the sensor to acquire enough data to accurately detect the object.
[0005] However, this high number of false positives means that a large proportion (e.g., at least 95%, at least 96%, at least 97%, at least 98%, or at least 99%) of the 3D data input to the 3D tracking system are not actually accurate detections of objects, which could overload the 3D tracking system's ability to find actual objects moving in 3D space in real time without the systems and techniques described herein for performing motion analysis on the 3D cloud of unverified detections of tracked objects. Note that false positives can include erroneous sensor detections that are not actually tracked objects (e.g., more than 90% of the data from each camera identifying a potential golf ball is not actually a golf ball, e.g., simply noise in the signal or other objects that are not golf balls), as well as erroneous combinations of data that are not actually a single object (e.g., more than 50% of the pairing of object detections from different cameras, or the pairing of object detections from a camera sensor and a radar device or other sensor, is incorrect, e.g., only the data from one of the sensors is actually a golf ball, or the data from both sensors is a golf ball, but not the same golf ball). Nevertheless, the systems and techniques described herein can accurately identify objects moving in 3D space even though the input 3D data contains many false detections of objects and many mispairings of data from the sensors used to detect the objects.
[0006] In general, one or more aspects of the subject matter described herein include one or more methods (and one or more non-transitory computer-readable media tangibly encoding instructions for causing a data processing device associated with an object detection system comprising two or more sensors to perform operations) that obtain three-dimensional positions of registered objects of interest by the object detection system configured to tolerate more false positives and minimize false negatives for the registered objects of interest, and form hypotheses of objects moving in three-dimensional space using a filter applied to the three-dimensional positions of the registered objects of interest, wherein the filter determines a hypothesis of an object moving in three-dimensional space if an estimated three-dimensional velocity vector for the particular object of interest approximately corresponds to an object moving in three-dimensional space over time. The method may be embodied in a method including: enabling associations between certain of the objects of interest; eliminating, during the formation process, a suitable subset of hypotheses that are not further expanded by the associations made, using at least one additional object of interest registered by the object detection system; specifying at least one three-dimensional track of at least one ball moving in three-dimensional space by applying a complete three-dimensional physical model to data about the three-dimensional positions of the registered objects of interest used in forming at least one hypothesis that remains after the elimination; and outputting for display at least one three-dimensional track of at least one ball moving in three-dimensional space.
[0007] Other embodiments of this aspect include corresponding systems, devices, and computer program products recorded on one or more computer storage devices, each configured to perform the operations of the method. Thus, one or more aspects of the subject matter described herein may be embodied in one or more systems including two or more sensors (e.g., a hybrid camera / radar sensor, at least one camera and at least one radar device, a stereo camera, multiple cameras that can form various stereo camera pairs, etc.), one or more first computers of an object detection system configured to register objects of interest and tolerate more false positives and minimize false negatives for the registered objects of interest, and one or more second computers configured to perform operations according to one or more methods. The foregoing and other embodiments may optionally include one or more of the following features, alone or in combination.
[0008] Forming hypotheses of an object moving in three-dimensional space using a filter may include using a first motion model in three-dimensional space, the first model having a first complexity level, if the number of registered detections for the given hypothesis is below a threshold, and using a second motion model in three-dimensional space, the second model having a second complexity level higher than the first complexity level of the first model, if the number of registered detections for the given hypothesis is equal to or greater than the threshold. The first motion model in three-dimensional space may be a linear model that associates the first three-dimensional position of the registered object of interest with a given hypothesis if the first three-dimensional position extends through the given hypothesis in approximately the same direction as predicted by the given hypothesis, and the second motion model in three-dimensional space may be a recurrent neural network model trained to perform short-term prediction of motion in three-dimensional space.
[0009] Forming hypotheses of an object moving in three-dimensional space may include using the first model or the second model to predict a next three-dimensional position of a registered detection for a given hypothesis according to the number of registered detections for the given hypothesis and a threshold value; searching a spatial data structure including all of the three-dimensional positions of the object of interest registered by the object detection system for the next time slice to find a set of three-dimensional positions within a defined distance of the predicted next three-dimensional position; if the set of three-dimensional positions is an empty set, using the predicted next three-dimensional position for the given hypothesis; if the set of three-dimensional positions includes only one three-dimensional position, using the one three-dimensional position for the given hypothesis; if the set of three-dimensional positions includes two or more three-dimensional positions, sorting the two or more three-dimensional positions based on their proximity to the predicted next three-dimensional position to form a sorted set; removing any less-close three-dimensional positions in the sorted set that exceed a defined threshold number greater than two; and branching the given hypothesis into two or more hypotheses in the group of hypotheses using the two or more three-dimensional positions remaining in the sorted set.
[0010] Forming hypotheses for an object moving in three-dimensional space may include identifying a data point in a given hypothesis having a minimal vertical position if the number of registered detections for the given hypothesis is greater than or equal to a threshold, checking each vertical component of an estimated three-dimensional velocity vector before and after the data point in the given hypothesis, and designating the data point as a ground collision if a first vertical component of the respective vertical components is negative and a second vertical component of the respective vertical components is positive.
[0011] Each vertical component may be obtained from an average of estimated three-dimensional velocity vectors before and after a data point in a given hypothesis, and the method may include selecting at least one time window based on a noise level associated with the estimated three-dimensional velocity vector, a shortest expected time of flight on one or both sides of the data point, or both the noise level and the shortest expected time of flight, and calculating an average of the estimated three-dimensional velocity vectors before and after the data point that falls within the at least one time window.
[0012] Forming hypotheses of the object moving in three-dimensional space may include dividing the given hypothesis into separate segments including a flight segment, one or more bound segments, and a roll segment if the number of registered detections for the given hypothesis is equal to or greater than a threshold, the dividing including identifying, checking, and designating; classifying a first segment of the given hypothesis before a first designated ground impact as a flight segment; classifying a next segment of the given hypothesis after each next designated ground impact as one of the one or more bound segments if an angle between a first estimated velocity vector before the next designated ground impact and a second estimated velocity vector after the next designated ground impact is greater than a threshold angle; and classifying a next segment of the given hypothesis after the next designated ground impact as a roll segment if an angle between a first estimated velocity vector before the next designated ground impact and a second estimated velocity vector after the next designated ground impact is less than or equal to a threshold angle.
[0013] The given hypothesis may be at least one hypothesis that remains after the elimination, and specifying at least one three-dimensional track of the at least one ball moving in three-dimensional space may include generating data about the three-dimensional positions of registered objects used in forming the at least one hypothesis by triangulating a more accurate 3D path using sensor observations registered by the object detection system in at least the flight segment, and confirming the at least one ball moving by applying a complete three-dimensional physics model to the generated data for the at least the flight segment.
[0014] The acquiring may include receiving or generating three-dimensional positions of the registered objects of interest, where a majority of the registered objects of interest are false positives, including erroneous detections by individual sensors and erroneous combinations of detections from each of the individual sensors. The individual sensors may include three or more sensors, and the acquiring may include generating a three-dimensional position by creating a respective combination of detections from each pair of the three or more sensors for the current time slice. The three or more sensors may include two cameras and a radar device, and the creating may include combining detections of the single object of interest by the two cameras using stereo pairing of the two cameras to create a first three-dimensional position of the single object of interest, and combining detections of the single object of interest by the radar device and at least one of the two cameras to create a second three-dimensional position of the single object of interest. Furthermore, the three or more sensors may include an additional camera, and the creating may include combining detections of the single object of interest by the additional camera and the two cameras using stereo pairing of the additional camera with each of the two cameras to create a third three-dimensional position of the single object of interest and a fourth three-dimensional position of the single object of interest.
[0015] The details of one or more embodiments of the subject matter described herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the invention will become apparent from the description, drawings, and claims. [Brief explanation of the drawings]
[0016] [Figure 1A] An example system is presented that performs motion-based pre-processing of two-dimensional (2D) image data and then performs three-dimensional (3D) object tracking of an object moving through three-dimensional space. [Figure 1B] 1B shows examples of different sensor and computer configurations that may be used in the system of FIG. 1A. [Figure 1C] 1B shows examples of different sensor and computer configurations that may be used in the system of FIG. 1A. [Figure 1D] 1B shows examples of different sensor and computer configurations that may be used in the system of FIG. 1A. [Figure 2] 1 is a schematic diagram of a data processing system including a data processing device; [Figure 3] An example of a process performed on different computers to detect objects, track objects in 2D, generate virtual 2D positions for time synchronization, and build a 3D track of a moving object is shown. [Figure 4A] An example process for finding a sequence of object positions that meets a motion criterion is shown. [Figure 4B] An example process for finding a sequence of object positions that meets a motion criterion is shown. [Figure 4C] An example process for finding a sequence of object positions that meets a motion criterion is shown. [Figure 4D] An example process for finding a sequence of object positions that meets a motion criterion is shown. [Figure 5] 1 illustrates an example process for interpolating between specified 2D locations within particular image frames acquired from a rolling shutter camera. [Figure 6A]1 shows an example of a process for constructing a 3D track for a moving object (e.g., a ball). [Figure 6B] 10 illustrates an example process for filtering out at least one 2D position obtained from a second camera. [Figure 7] We present an example process for 3D object tracking of moving objects using unverified detections registered by two or more sensors. [Figure 8] 8 illustrates an example of a process for hypothesis formation that may be used in the 3D object tracking process of FIG. 7. [Figure 9] 8 illustrates an example process for dividing hypotheses into separate segments during hypothesis formation that may be used in the 3D object tracking process of FIG. 7. [Figure 10A] 8 illustrates an example process for specifying a 3D track for a moving ball that may be used in the 3D object tracking process of FIG. 7. [Figure 10B] 8 illustrates an example process for specifying a 3D track for a moving ball that may be used in the 3D object tracking process of FIG. 7. DETAILED DESCRIPTION OF THE INVENTION
[0017] Like reference numbers and designations in the various drawings indicate like elements.
[0018] 1A illustrates an example of a system 100 that performs motion-based preprocessing of two-dimensional (2D) image data and then performs three-dimensional (3D) object tracking of an object moving through a 3D space 110. The tracked object may be a golf ball or another type of object that is hit, kicked, or thrown (e.g., a baseball, soccer ball, or football / rugby ball). In some implementations, the 3D space 110 is a driving range, a grass field, or another open area where objects may be launched. For example, the 3D space 110 may be part of a golf entertainment facility that includes one or more targets 114, a building that includes golf bays each including at least one teeing area 112 (or, more generally, a launching area 112), and potentially other entertainment and dining options.
[0019] In some implementations, the 3D space 110 is a playing area for a sport such as a golf course, the launch area 112 may be a golf tee for a particular hole on the golf course or an intermediate landing spot for a golf ball in play on the course, and the target 114 may be a cup at the end of a particular hole on the golf course or an intermediate landing spot for a golf ball in play on the course. Other implementations are possible, such as the launch area 112 being one of multiple designated tee areas along a tee line from which a golfer can hit a golf ball into an open field 110, or one of multiple designated tee areas in the stands of a sports stadium from which a golfer can hit a golf ball across and over the playing field 110 of the sports stadium.
[0020] System 100 includes two or more sensors 130, including at least one camera 120 and its associated computer 125. One or more of sensors 130 (including at least one camera 120 and its associated computer 125) may, but need not, be located near launch area 112 for tracking objects. In some implementations, one or more sensors 130 (including cameras 120 and computers 125) may be located along one or both sides of 3D space 110 and / or on the other side of 3D space 110 opposite launch area 112. For example, in a golf tournament, camera 120 and computer 125 may be located behind the green, facing toward the golfer, assuming that shots are being hit toward the green. Thus, in various implementations, the sensor 130 can observe and track objects moving away from the sensor 130, towards the sensor 130, and / or through the field of view of the sensor 130 (note that each set of ellipses in the figure indicates that one or more additional instances of sensors, computers, communication channels, etc. may also be included).
[0021] Sensors 130 may include cameras (e.g., stereo camera pairs), radar devices (e.g., single-antenna Doppler radar devices), or combinations thereof, potentially including hybrid camera / radar sensor units, as described in U.S. Patent No. 10,596,416. However, at least one of sensors 130 is a camera 120 and its associated computer 125, which are connected by a communications channel. Figures 1B-1D show examples of different sensor and computer configurations that may be used in the system of Figure 1A.
[0022] 1B shows an example of a pair of cameras 152, 156 connected to a first computer 150 via first communication channels 154, 158 having a first data bandwidth greater than the bandwidth of at least one other communication channel used in the system. For example, the first communication channels 154, 158 may employ one or more high-bandwidth, short-range data communication technologies, such as Universal Serial Bus (USB) 3.0, Mobile Industry Processor Interface (MIPI), or Peripheral Component Interconnect eXtended (PCIx). As described in further detail below, preprocessing of image data from the cameras 152, 156 may be performed near the cameras in one or more of the first computers 150, and once the preprocessing in the first computer(s) 150 reduces the data bandwidth, the output of this preprocessing may be transmitted via a second communication channel 162 having a second data bandwidth less than the first data bandwidth. Thus, the second communication channel 162 may employ one or more lower bandwidth, longer distance data communication technologies, such as copper Ethernet or a wireless data connection (e.g., using WiFi and / or one or more mobile phone communication technologies).
[0023] This is important because it allows the system to be implemented with higher resolution cameras 120, 152, 156 and computers 125, 150 that operate on raw image (uncompressed) data from these cameras 120, 152, 156. Note that using higher resolution cameras with higher frame rates, whether using stereo camera tracking or hybrid camera / radar tracking, allows for higher quality 3D tracking, but only if the data can be processed efficiently and effectively. Furthermore, if object tracking is intended to work for very small objects (e.g., if the object only appears at a single pixel even in high-resolution camera images), object detection may need access to the raw image (uncompressed) data because using conventional lossy video compression techniques (e.g., MPEG) can remove valuable information about the small object from the image.
[0024] To address these issues, the first computer(s) 150 may perform pre-processing (including object detection and optionally 2D tracking) on image data close to the cameras 152, 156 to reduce the bandwidth requirements for transmitting the sensor data to one or more second computers 160 over a second communication channel 162. In addition, pre-processing (as described herein) enables downstream (post-image capture) virtual time synchronization of measured object positions in time and space, allowing 3D tracking to be performed at the second computer(s) 160 using data received over the one or more second communication channels 162. This allows downstream processing to be easily performed on a remote server, as after pre-processing, the data bandwidth is so narrow that transmitting data over long distances is trivial.
[0025] It should be noted that this can provide significant advantages when setting up system 100 because of the flexibility it offers. For example, in the case of a golf competition television (TV) broadcast, where system 100 can be used to track a golf ball through the 3D space of a golf course and overlay the golf ball's trace onto a generated TV signal for live transmission or recording, sensor 130 may be deployed a mile or more away from the TV production facility (where 3D tracking computer 160 may be positioned). It should be noted that the transformation of the ball position (identified during 3D tracking) to a corresponding position in the video data acquired by the TV camera (which allows for the trace overlay of a graphical representation of the ball's flight path onto the video data) can be performed using known homography techniques. As another example, in the case of a golf entertainment facility, the 3D tracking computer (e.g., server computers 140, 160) does not need to be located at the same facility, and the 3D tracking performed by this computer (e.g., to augment other data or media, such as showing the path of the golf ball in a computer representation of the physical environment in which the golfer is located or in a virtual environment that exists only on the computer) can easily be transferred to another computer (e.g., failover processing).
[0026] Various sensor and computer configurations are possible. Figure 1C shows an example in which each camera 152, 156 has a dedicated first computer 150A, 150B, which communicates its respective preprocessed data to second computer(s) 160 via separate second communication channels 162, 164. Thus, the cameras (or other sensor technologies) may or may not share first computer resources. Furthermore, preprocessing can be split and performed on different computers.
[0027] 1D shows an example in which a camera 152 is coupled to a computer 150 via a first communication channel 154 having a first data bandwidth, the first computer 150 is coupled to a third computer 166 via a third communication channel 168 having a third data bandwidth, and the third computer 166 is coupled to a second computer 160 via a second communication channel 162 having a second data bandwidth, where the second data bandwidth is narrower than the first data bandwidth and the third data bandwidth is narrower than the first data bandwidth but wider than the second data bandwidth. The first computer 150 performs object detection, the third computer 166 performs 2D tracking of the object, and the second computer 160 performs virtual time synchronization and 3D tracking of the object. Furthermore, in some implementations, the first computer 150 performs object detection and pre-tracking in 2D (using very simple / loose constraints), the third computer 166 performs more thorough 2D tracking, and the second computer 160 performs virtual time synchronization and 3D tracking of the objects.
[0028] Other sensor and computer configurations are possible consistent with the disclosure herein. For example, rather than using an intermediate third computer 166 to perform 2D tracking of objects, the first computer 150 can perform object detection (pre-tracking in 2D (using very simple / loose constraints) or no 2D tracking of objects), while the same second computer 160 can perform 2D tracking of objects (more thorough 2D tracking after pre-tracking in 2D or full 2D tracking), virtual time synchronization, and 3D tracking of objects. Conversely, in some implementations, one or more additional intermediate computers can be used. For example, a system can use four separate computers to perform each of the four operations: object detection, 2D tracking, virtual time synchronization, and 3D tracking. As another example, a system can use five separate computers to perform each of the five operations: object detection, pre-tracking in 2D (using very simple / loose constraints), more thorough 2D tracking, virtual time synchronization, and 3D tracking. Other configurations are possible, provided that at least one of the operations is performed on a first computer communicatively coupled to the at least one camera via a first communication channel, and at least one other of the operations is performed on a second computer communicatively coupled to the first computer via a second communication channel having a data bandwidth narrower than the data bandwidth of the first communication channel.
[0029] Various types of computers can be used in the system. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. As used herein, a "computer" may include a server computer, a client computer, a personal computer, an embedded programmable circuit, or a special-purpose logic circuit. Figure 2 is a schematic diagram of a data processing system including a data processing device 200 representing one implementation of the first computer 150, the second computer 160, or the third computer 166. The data processing device 200 may be connected to one or more computers 290 via a network 280.
[0030] The data processing device 200 may include various software modules that may be distributed between the application layer and the operating system. These may include executable and / or interpretable software programs or libraries, including programs 230 that operate as an object detection program (e.g., in the first computer 150), a 2D tracking program (e.g., in the first computer 150 and / or the third computer 166), a virtual time synchronization program (e.g., in the second computer 160), and / or a 3D tracking program (e.g., in the second computer 160), as described herein. The number of software modules used may vary from implementation to implementation. Also, in some cases, for example, the 2D tracking program 230 may be implemented in embedded firmware, while in other cases, for example, the time synchronization and 3D tracking program 230 may be implemented as software modules distributed on one or more data processing devices connected by one or more computer networks or other suitable communication networks.
[0031] The data processing apparatus 200 may include hardware or firmware devices, including one or more hardware processors 212, one or more additional devices 214, a non-transitory computer-readable medium 216, a communication interface 218, and one or more user interface devices 220. The processor 212 may process instructions for execution within the data processing apparatus 200, such as instructions stored on the non-transitory computer-readable medium 216, which may include a storage device, such as one of the additional devices 214. In some implementations, the processor 212 is a single-core or multi-core processor, or two or more central processing units (CPUs). The data processing apparatus 200 communicates with one or more computers 290 using its communication interface 218, for example, via a network 280. Thus, in various implementations, the processes described may be executed in parallel or serially on single-core or multi-core computing machines, and / or on a computer cluster / cloud, etc.
[0032] Examples of user interface devices 220 include a display, a touchscreen display, a speaker, a microphone, a haptic feedback device, a keyboard, and a mouse. Furthermore, the user interface device(s) need not be local device(s) 220 but may be remote from the data processing apparatus 200, e.g., user interface device(s) 290 accessible via one or more communication network(s) 280. The data processing apparatus 200 may store instructions implementing the operations described herein on a non-transitory computer-readable medium 216, which may include, e.g., one or more of the following additional devices 214: a floppy disk device, a hard disk device, an optical disk device, a tape device, and a solid-state memory device (e.g., a RAM drive). Furthermore, instructions for performing the operations described herein may be downloaded to the non-transitory computer-readable medium 216 from one or more computers 290 over the network 280 (e.g., from the cloud); in some implementations, the RAM drive is a volatile memory device to which instructions are downloaded each time the computer is powered on.
[0033] Additionally, the 3D object tracking of the present disclosure may employ the sensor data preprocessing, 2D object tracking, and virtual time synchronization described herein (and in U.S. Provisional Patent Application No. 63 / 065,872, filed August 14, 2020, and U.S. Patent Application No. 17 / 404,953, filed August 17, 2021, both entitled "Motion-Based Pre-Processing of Two-Dimensional Image Data Prior to Three-Dimensional Object Tracking With Virtual Time Synchronization"), and thus the 3D object tracking systems and techniques of the present disclosure, when implemented together, may obtain the attendant benefits of the data preprocessing, 2D object tracking, and virtual time synchronization systems and techniques, although in some implementations, these systems and techniques need not be used to obtain the three-dimensional position of an object of interest registered by an object detection system. One or more other object detection systems may be used with the 3D object tracking, which will be described in further detail in connection with FIGS. 7-10B.
[0034] Figure 3 illustrates an example process performed on different computers to detect objects, track them in 2D, generate virtual 2D positions for time synchronization, and construct a 3D track of a moving object. The process of Figure 3 includes pre-processing operations 310-330 performed on one or more first computers (e.g., computers 125, 150, 166 of Figures 1A-1D) and additional processing operations 360-375 performed on one or more second computers (e.g., computers 140, 160 of Figures 1A-1D). The pre-processing operations may include object detection and 2D tracking that effectively compress ball position data (to reduce bandwidth requirements for transmitting data used in 3D tracking) to enable virtual time synchronization of measured object positions during additional processing on the second computer(s).
[0035] Thus, image frames 300 are received (310) from the camera (e.g., by computer 125, 150) via a first communication channel 305 that couples the camera with a first computer(s). In this case, the first communication channel 305 over which image frames 300 are received has a first data bandwidth. For example, first communication channel 305 may be a USB 3.0, MIPI, or PC1x communication channel, e.g., communication channel(s) 154, 158. It should be noted that the bandwidth requirements between the camera and the computer can easily exceed 1 gigabit per second (Gbps); for example, a 12 megapixel (MP) camera operating at 60 frames per second (FPS) and 12 bits per pixel requires a bandwidth of over 8 Gbps.
[0036] Furthermore, multiple such cameras used in combination may require a total bandwidth of 10-100 Gbps, which would place a serious strain on even Ethernet communication hardware. Furthermore, stereo equipment (e.g., stereo cameras 152, 156 in FIG. 1B ) may require significant distances between cameras or between cameras and computer infrastructure, such as a server room or cloud-based computing, making high-bandwidth communication even more challenging if long cables and / or communication over the Internet are required. As discussed above, traditional video compression techniques, such as MPEG technology, may not be an adequate method for reducing bandwidth, especially when small objects (e.g., a distant golf ball) are tracked, as there is a risk that the tracked object will be removed by traditional video compression. Therefore, a high-bandwidth communication channel 305 (for video frames from one or more cameras) is used to enable high-resolution, high-bit-depth, and / or uncompressed image data to be received (310) as input to the object detection process.
[0037] Locations of interest are identified (315) in the received image frames (e.g., by computers 125, 150). For example, this may involve using image differencing techniques to identify locations in the image frames that have one or more image data values that change from the previous image frame by more than a threshold amount. Additionally, other approaches are possible. For example, the process may look for groups of pixels with a particular brightness or color (e.g., white for a golf ball), look for a shape that matches the shape of the tracked object (e.g., a round or at least oval shape to find a round golf ball), and / or use template matching to locate the object (e.g., a golf ball) in the image.
[0038] Furthermore, locating locations having one or more image data values that change by more than a threshold amount from one image frame to another may include applying image differencing to find pixels or groups of pixels that change by more than a threshold amount. For example, image differencing can be applied to find pixels in each image that change by more than a certain threshold, and groups of such changing pixels that are adjacent to each other can be found using, for example, known connected component labeling (CCL) and / or connected component analysis (CCA) techniques. Such groups of pixels (and potentially single pixels) that meet the object detection criteria may be referred to as "blobs," and the location and size of each such blob may be stored in a list, and the list of all blobs in each image may be sent to the 2D tracking component. Converting images to a list of object locations (or blobs) has a bandwidth reduction effect. In some cases, the bandwidth reduction of this operation may be 10:1 or more. However, as described herein, further bandwidth reduction can be achieved, which may be of great benefit when small objects are being tracked.
[0039] For small object tracking, false positives are a significant problem due to the difficulty of identifying small objects (potentially even a single pixel in an image) based on their features. Therefore, identifying (315) (to detect an object of interest at a particular location in a camera image) can be implemented using a low threshold to favor zero false negatives while tolerating a large number of false positives. It should be appreciated that this approach is generally counterintuitive in that false positives in object tracking are often unfavorable, thus setting up a competition between minimizing false positives and minimizing false negatives. However, our approach to object tracking easily handles false positives because downstream processing is designed to handle a large number of false positives. However, because object detection is designed to tolerate many false positives, more objects are identified (315) in each image frame 300, including many “objects” that are simply noise in the image data, thus partially offsetting the bandwidth-reducing effect of converting an image to a list of objects.
[0040] The sequence of identified locations in the image frames is found (320) (e.g., by computers 125, 150, 160, 166). Note that while the process shown in Figure 3 (and other figures) is shown as sequential operations for ease of understanding, in practice operations may occur in parallel or simultaneously, for example, using hardware and / or operating system-based multitasking and / or using pipelining techniques. Pipelining may be used for concurrent processing; for example, object identification (315) may begin processing frame n+1, if available, immediately after handing off frame n to 2D tracking (320) without having to wait for downstream components to finish first. Therefore, except where the processes performed on each computer are described as sequential processes, i.e., the object identification process, the 2D tracking process(es), and the virtual time synchronization and 3D tracking process(es) occur sequentially, the disclosure presented herein in connection with the drawings is not limited to performing the operations sequentially as shown in the drawings, as each object identification and 2D tracking processing step reduces the bandwidth of the data transmitted to downstream components.
[0041] Each of the discovered (320) sequences meets a motion criterion for the identified location in at least three image frames from the camera. In some implementations, the criterion is measured for four or more frames, and / or one or more criteria are used (e.g., the tree initiation criterion described below). In general, 2D tracking aims to find a sequence of objects (or blobs) over three or more frames that exhibit object movement consistent with the movement of the object in Newtonian motion, unaffected by forces other than gravity, bouncing, wind, air resistance, or friction.
[0042] This object movement criteria may be defined to include displacement, velocity, and / or acceleration in each dimension (x and y in the image) within a predefined range of values. This range of values is set so that the 2D motion and acceleration of a moving object (e.g., a golf ball in flight) as depicted by a 2D camera is well within the specified boundaries, while more jerky motion is rejected (unless there is a known object that may bounce the tracked object). Furthermore, larger systems employ secondary tracking steps in downstream processing that can filter at a finer granularity what constitutes an actual tracked object (e.g., a golf shot), so the finding (320) need not be a perfect (or near-perfect) filter that only allows in actual object motion, such as the motion of a golf ball after being struck from the teeing area 112.
[0043] Rather, the filtering performed in finding 320 may be intentionally imperfect so that objects other than the moving object are included in the discovered sequence (including potential noisy sequences that may have been incorrectly identified 315 as an object of interest and then incorrectly discovered 320 to form the sequence). In other words, finding 320 may implement a loose filter that increases false positives and minimizes false negatives, e.g., all or nearly all moving golf balls are accepted as forming valid sequences in finding 320.
[0044] This looser (benefit from a doubt) approach means that a much simpler tracking algorithm can be used in 320, with the understanding that it need not be perfect in distinguishing desired objects (e.g., golf balls) from unwanted objects (e.g., non-golf balls). The set of rules defining the tracking can be reduced to a minimum, and any errors made by 2D tracking (such as passing non-golf balls) can be removed by downstream components and processing. Instead of emitting entire trajectory paths, each with one start and one end, the discovered 320 sequence can be represented as a “rooted tree,” where each vertex (node in the tree) is an observation blob (x, y, and time t) and each edge is a possible transition between positions of the object whose motion is being tracked. As will be described in more detail in connection with FIG. 4A , each branch may further have some metadata, such as the total depth of the tree.
[0045] However, even with this looser (below-doubt / low-threshold) approach, missed object detections can still occur. Therefore, dummy observations can be used to account for objects that should be in the image data but have not been identified. In some implementations, if a sufficiently good blob cannot be found to which a path can be extended, the 2D tracker can add dummy observations at predicted locations. Dummy observations can be implemented with significant penalty scores, and in some implementations, dummy observations will not be allowed unless the graph is already at a certain depth. Because there is a limit on how much penalty a branch can have, there is in effect a limit on how many dummy observations a path can have.
[0046] As described above, finding (320) can involve forming a rooted tree from the positions of interest, where each rooted tree is a connected acyclic graph with a root node that is the root of the tree, and all edges of the connected acyclic graph stem directly or indirectly from the root. Figure 4A shows an example of a process for finding a sequence of object positions that meets a motion criterion by forming a rooted tree. At (400), the next set of positions identified for an image frame is obtained for processing, and this process continues while the position of interest remains in the set for the current frame (405). For example, when a frame of blobs is processed, every blob in the new frame can be matched to every tree node added during processing of the previous frame to see if the blob could be a possible continuation of its path, depending on how much the points in this branch resemble the desired motion, as defined by the motion criterion.
[0047] The next focus location is retrieved from the set (410), and a check is made to determine whether this focus location meets the tree initiation criteria (415). If the focus location meets the tree initiation criteria, a root node of a new tree is established using this focus location (420). For example, if the image data value at the focus location is larger than the minimum object size, this can be used to indicate that the ball is close to the camera and a new tree should be established. Figure 4B shows a visual example of this, where six blobs 460 are observed, but only four of these blobs 460 are large enough to be used to establish a new root node 465. Note that one blob observation can be added to several trees, and every observed blob could theoretically be the start of a new object observation. This can lead to combinatorial explosion in noisy environments; therefore, in some implementations, some additional constraints (such as a minimum blob size constraint) are imposed before establishing a new tree.
[0048] In this example, all blobs larger than a certain minimum size are promoted to a new tree of depth 0, where the minimum size limit may be set based on the camera resolution and noise level of the incoming video. For example, a criterion may be that a blob must be at least two pixels (not just a single pixel) to establish a new tree. Note that truly random noise affects pixels individually and is very unlikely to generate larger clusters in an image. Other approaches to avoiding combinatorial explosion are possible. For example, the noise threshold in the blob generation (i.e., location identification) process can be adaptively adjusted.
[0049] Furthermore, the minimum tree depth required for a path to be exported can be increased, ensuring that the amount of data exported is limited since it is rare for random noise sequences to construct longer sequences by chance. Because the constraints are very loose, many graphs with depth 0 and depth 1 will be generated, but because of better predictions, most of these graphs will never reach depth 2, and even fewer will reach depth 3. Thus, statistics are valid for the benefit of the systems and techniques of this disclosure, as trees or branches are discarded when there are no more nodes that can be added.
[0050] Furthermore, when the camera is positioned near the target facing the object's launch area, i.e., to detect incoming objects, different size thresholds can be applied to different portions of the image. For example, a minimum size of 1 pixel can be used in portions (e.g., rectangles) where the departing ball is found, and a minimum size of 2 pixels can be used in other portions. Thus, the criteria used to form the tree of possible object paths can be determined based on the position of the camera relative to the launch area.
[0051] For example, if a reasonably flat ground can be assumed and the camera position and aim are known, all ball detections below the ground "horizon" can be "sanity" checked against an estimate of the maximum distance (and therefore minimum size) the ball can be at. The ray from the camera to the ball detection intersects the ground at some distance D. The ball must be located at or nearer distance D; otherwise, it is underground. For random ball detections above the "horizon," there is no simple distance heuristic, since there is no ground intersection. However, if there is knowledge of the limits of the 3D space (e.g., a playing field), some general and / or angle-dependent maximum distance / minimum object size constraints can be used, taking into account that objects must be within the 3D region of interest.
[0052] Returning to FIG. 4A , a check (425) is performed to determine whether the current location of interest is within the root node's established distance threshold for a location identified in the previous image frame. If the location of interest is within the distance threshold, the location of interest is added (430) as a first depth subnode under the root node of this tree. Note that if the tree consists of only one vertex (depth=0), there is still no way to estimate the speed of its path, so any blob within a reasonable distance from the vertex needs to be matched. FIG. 4C shows a visual example of this, continuing from FIG. 4B . Only two of the four observed blobs 470 are within the distance limit 472 of the root node 474 from the previous frame. Therefore, only two of the blobs 470 are added to the rooted tree as the first depth subnode 476. In some implementations, the distance limit depends on the resolution and frame rate of the incoming sensor and the maximum expected speed (perpendicular to the camera's aiming direction) of the tracked object. If the size of the object being tracked is known in advance, as in the case of a golf ball, this can be taken into account when determining the distance limit / threshold 472. Furthermore, larger blobs can be estimated to be closer to the camera than smaller blobs, meaning that more movement between frames can be tolerated if larger blobs are present (either in the previous frame or the current frame, depending on the placement of the camera relative to the expected direction of the object being tracked).
[0053] However, if a blob is matched to a vertex at depth one or more, it is possible to predict where the next point in the path should be. In some implementations, the expected area of the next blob is calculated by assuming that the velocity will be the same since the last frame. FIG. 4D shows a visual example of this. A predicted position 484 is calculated based on the location of the root node 480 and the location of the first depth subnode 482, as well as the known frame rate of the camera. A search area 486 (or region) around the predicted position 484 is then determined for use in searching for blobs to add to the tree. Similar to the details provided above for determining the distance limit / threshold 472, the size of the area / region 486 around the predicted position 484 may be determined based on the camera's resolution and the maximum expected velocity (perpendicular to the camera's aiming direction) of the object being tracked, which may be adjusted according to the known size of the object and the size of the blob in different image frames. Additionally, in some implementations, for example, if the degree of the graph is sufficiently large, a Kalman filter can be used in generating the prediction.
[0054] Any blob that is sufficiently close to the predicted location can be converted into a vertex and added to the tree. Thus, as shown in FIG. 4D , one blob is within region 486 of predicted location 484, and therefore this blob is added as a new subnode 488. Additionally, in some implementations, penalties can be accumulated on tree branches, and the penalties are added to the score used to rank the branches, which can help in determining the best branch to export. Furthermore, in some implementations, the further a blob's location deviates from the expected coordinates, the higher the penalty assigned to this branch.
[0055] In some implementations, the penalty accumulated on a tree branch can be used to limit the magnitude of the allowable penalty and to determine when to discard the branch. In the former case, if a branch already has a high penalty, adding a new node that would cause the extended branch to exceed the limit is not allowed. If the penalty is calculated from the discrepancy between the predicted and actual positions, this is essentially a way to ensure that the acceleration is within limits; otherwise, the penalty will be too high. In the latter case, if there are several branches that share the last three blobs, the deepest branch with the smallest penalty can be kept, and the other branches can be discarded. Note that this type of penalty-based hypothesis tracking can be applied to 3D tracking (e.g., as described in further detail in connection with Figures 7-10B) in addition to 2D tracking.
[0056] Returning to FIG. 4A, a check (435) is performed to determine whether the current position of interest is within a region determined using the estimated velocity of the position used for the parent subnode of the established tree. If the current position of interest is within the region, the position of interest is added to this tree as a second or deeper subnode under the parent subnode (440), as described above in connection with FIG. 4D. Additionally, in some implementations, as the tree depth increases, more advanced prediction algorithms can be used to obtain better predictions. By fitting the nearest nodes in a branch to a second- or third-order polynomial function, a good prediction of where the next point will be can be obtained, and paths with clearly unreasonable changes in acceleration can be discarded. Furthermore, in some implementations, observations can be provided to a Kalman filter, and a model from the Kalman filter can be used to generate a new prediction.
[0057] Also, note that the same location of interest in a single image frame can be added as a subnode (at a first depth or at a greater depth) to two or more rooted trees that are keeping track of potential object paths through the image frame, as well as potentially being used to establish new root nodes for new trees. Furthermore, the same location of interest in a single image frame can be added as a subnode in a tree with two or more linkages back to respective parent nodes in the tree, and the tree will contain the same subnode at different depths depending on which linkage goes back to the root node.
[0058] Thus, many erroneous paths may be generated, and even if downstream processing can filter out these erroneous paths, there may still be limitations on when potential paths are output. In some implementations, a sequence is not considered for output until the rooted tree representing the sequence exceeds a predetermined tree depth (445). Thus, a sequence of identified locations is not output until the sequence is confirmed for output (450) based on whether the tree depth of the sequence exceeds a predetermined tree depth (445), as specified for a given implementation or determined on the fly based on one or more factors.
[0059] Note that only a portion of the rooted tree having a predetermined tree depth needs to be verified for output as a sequence of identified locations (450). This helps prevent noise that would generate erroneous paths in the rooted tree from propagating from the 2D tracking in the pre-processing stage on the first computer to the post-processing stage on the second computer. Also, the minimum tree depth threshold generally depends on the environment and the tolerable delay for generating 3D tracks of objects. In some implementations, the rooted tree of a sequence must have a tree depth greater than 2 before the sequence can be verified for output (450). In some implementations, the tree must have a depth greater than 3, 4, 5, 6, or 7 before the sequence can be verified for output (450).
[0060] For example, in some implementations, a tree vertex (corresponding to a blob / location) is exported only if it is at a certain depth (D) in the tree or has a child node that is exported by a child node with a certain depth (D) in the tree. A vertex with D=5 has five edges between itself and the root. This means that the output may be delayed by the same number of frames D, since it is often not possible to know in advance whether a blob is part of a long enough branch. The described filtering (by limiting the export of tree nodes) can prove to dramatically reduce the bandwidth required to communicate object candidates, since the only blobs that are exported are those that make up possible paths. The table below shows an example of bandwidth reduction for a golf application based on the minimum branch depth required. [Table 1]
[0061] As can be seen from this table, the majority of all blobs / locations of interest are either not connected by any paths or are not connected by paths of sufficient depth, and therefore most blobs / locations of interest are rejected by the effective filtering performed in the pre-processing stage. Returning to Figure 3, at 325, a check is made to determine whether data for a sequence is ready for output, i.e., for any sequence of nodes in the tree that includes at least one node having a tree depth greater than a predetermined tree depth, as described above, to determine whether the nodes in that sequence are ready for output. If ready for output, the output data for one or more sequences of locations is sent (330) (e.g., by computers 125, 150, 166) to one or more second computers (e.g., computers 140, 160). However, in some implementations, the identified (315) locations are transmitted from the first computer (e.g., by computer 125, 150) to a second computer (e.g., computer 140, 160, 166), which discovers (320) the sequence of the identified locations within the image frames and then outputs the sequence data to another process on the same computer or a different computer. In either case, the sequence output data includes, at least, for each location within each sequence, the two-dimensional location of the location within a particular image frame with a timestamp, which is necessary for virtual synchronization at the second computer.
[0062] One of the foundations of 3D tracking is triangulation. However, triangulation of the position of an object seen by two cameras requires that the observations are aligned from the same time instance. A common way to achieve this is to use a common synchronization trigger signal (e.g., transmitted via a cable) for all cameras. However, this method is only effective if the cameras have the same capture frame rate. In some configurations, it is difficult or impossible to ensure that the two cameras are synchronized (triggering image capture in tandem), which in turn makes triangulation of observations from different cameras difficult or impossible. To solve this problem, rather than attempting to actually synchronize the camera images with other sensors at the time of data capture, timestamp information (combined with information about the movement represented by the 2D path) is used for virtual synchronization in a second computer.
[0063] Furthermore, the systems and techniques described herein can be used with both global shutter and rolling shutter cameras. A rolling shutter means that the capture of each row in a camera image is timed differently. Therefore, the object's position can be determined as it is imaged by the sensor, but the measurement time depends on where the object is located in the image. In some implementations, there can be a time difference of approximately 20 milliseconds between the top and bottom image rows. This also creates a triangulation problem, as the requirement for simultaneous measurement in both cameras may not be met. To solve this problem, rolling shutter information is taken into account by considering the relative time of a specific time offset of the "position" within the frame in addition to the frame timestamp, so that the measured object position from the rolling shutter camera can also be used for high-quality triangulation in a second computer during the post-processing stage. For global shutter capture, this offset is always zero.
[0064] As discussed above in connection with Table 1, because several frames of data may be required to make a track conform to a minimum path length (minimum tree depth) requirement, there is a trade-off between introducing a delay for every blob and implementing a more complex protocol that allows already conformed blobs to be communicated as soon as they are recognized. Thus, in some implementations, sending (330) involves delayed dispatching of both frame and blob data, and in some implementations, sending (330) involves incremental dispatching of frame and blob data.
[0065] In the case of delayed dispatch, transmitting (330) involves delaying the output of data for interest locations found in a given image frame and one or more of the sequences until any additional interest locations identified for the given image frame are not included in any of the sequences based on interest locations identified in subsequent image frames. Thus, exporting nodes for a frame can occur when the number of subsequent frames processed excludes a defined tree depth being reached for any more blobs in the current frame. In other words, transmitting the frame is delayed until it is known that any additional blobs identified in the frame are not included in a path in a rooted tree having a required minimum tree depth.
[0066] Furthermore, in some implementations, a dynamic minimum tree depth may be used. The minimum tree depth may be initially set low to minimize latency when there are few paths, and then dynamically increased if the load (total output of bounded paths per time unit) exceeds some maximum load threshold. This may result in a kind of throttling effect that improves performance by adjusting processing in response to current input data.
[0067] Additionally, the frame data structure may include a list of blobs that were detected within this frame and that passed the filter criteria. For example, the frame data structure may include the data fields shown in Table 2 below. [Table 2] Additionally, the blob data structure included in the frame data structure may include the data fields shown in Table 3 below. [Table 3] Note that these data structures can be encoded in a variety of formats, such as JSON format, as follows:
number
[0068] When unfiltered blobs are passed between processing nodes, for example, when blob detection is performed on an edge device and pre-tracking and / or tracking is performed on a different computer, a simplified format can be used since there is no information about the relationship between blobs in different frames. For example, the frame data structure may include the data fields shown in Table 4 below. [Table 4] Additionally, the SimpleBlob data structure contained in the frame data structure may include the data fields shown in Table 5 below. [Table 5]
[0069] In the case of incremental frame dispatch, sending (330) involves outputting the image frame data as the identifying (315) for each image frame is completed, and outputting the data for each interest location (330) only after finding one or more of the sequences containing the interest location to be output (325). As mentioned above, the sequence must have the tree depth required to export the node, and when the tree reaches a threshold depth, all parent nodes (of tree nodes above the threshold depth) that have not previously been exported (by some other branch) are retroactively output (330). This is a more complex approach to transmitting data that can be used when low latency is required. 2D tracking outputs a continuous stream with information about the processed frame and a list of all blobs (i.e., all identified (315) interest locations) that passed ball candidate filtering as soon as the information is available.
[0070] For example, when a new frame is processed, information about the frame may be transmitted using a frame data structure having the data fields shown in Table 6 below. [Table 6] Additionally, the blob data structure may include the data fields shown in Table 7 below. [Table 7] Additionally, each connection between blobs can be represented using a connection data structure that includes the data fields shown in Table 8 below. [Table 8] Records generated using the above data structure can be placed on the stream (from the first computer to the second computer) as soon as the information is established (frame and blob data can be sent 330 during and after identifying 315, and blob connection data can be sent 330 after the discovery 320 sequence formed by these blob connections is confirmed 325 for output). All blobs and frames referenced by a record are guaranteed to be placed on the stream before the record that references it. However, if a consumer starts reading mid-stream, it may miss the original declaration, and this issue should be addressed on the second computer.
[0071] Regardless of the data structure or output timing used, output data 350 is received (360) from one or more first computers (e.g., from computers 125, 150) (e.g., by computers 140, 160) via a second communication channel 355 coupling the first computer(s) to the second computer(s), the second communication channel 355 through which the output data 350 is received having a second data bandwidth that is narrower than the first data bandwidth of the first communication channel 305. For example, second communication channel 355 may be a copper Ethernet or wireless communication channel, e.g., communication channels 162, 164. Furthermore, in implementations in which a portion of the pre-processing (e.g., discovering 320) is performed on the same second computer (e.g., by computers 140, 160) as the additional processing operations 360-375, the received data includes the identified (315) location and associated timestamp information.
[0072] In either case, receiving data continues (360) until checking (365) indicates that the sequence is ready to be processed. Additionally, as described above, operations may be performed in parallel or simultaneously, e.g., during building (375), e.g., using hardware and / or operating system-based multitasking and / or using pipelining techniques. Thus, receiving (360) may continue while processing (370) one or more sequences occurs in parallel or simultaneously, and each component may begin processing frame n+1, if available, immediately after handing off frame n to a downstream component without having to wait for the downstream component to finish first; for example, stereo triangulation may be performed on the next frame even if the 3D tracker has not yet finished processing the focus position.
[0073] The sequence(s) in output data 350 are processed (370) by one or more second computers (e.g., by computers 140, 160) by interpolating between specified 2D positions in particular image frames of the sequence(s) using the timestamps of the particular image frames to generate a virtual 2D position at a given time. The record of each point of interest (e.g., each blob found in each frame) in output data 350 includes both the frame's timestamp and an indication of one or more previous points of interest connected to this point of interest by a 2D tracking component (e.g., each blob is described with pointers to one or more previous blobs to which it belongs). Thus, from this data 350, it is possible to determine the time, location, and direction of movement, as well as the velocity, of each point of interest / blob.
[0074] Thus, the data 350 allows for the use of interpolation to generate a virtual "intermediate" position / blob at any given point along the path, as long as there is at least one position / blob of interest in the tree with an earlier timestamp and at least one position / blob with a later timestamp. Additionally, in some implementations, the predetermined point in time is one of multiple points in time at a predefined constant frame rate of the 3D tracking server. For example, the 3D tracking server computer 140, 160 can operate at a predefined constant frame rate and use interpolation to generate virtual snapshots of the blobs from all cameras at these points in time. Because the blob coordinates all represent the same point in time, triangulation between points is possible even if the original captures were not synchronized. Furthermore, in some implementations, the predetermined point in time is a time specified by another sensor, such as a camera sensor or radar sensor.
[0075] Additionally, as noted above, the camera may be a rolling shutter camera, in which case the output data 350 may include a time offset value for each position of interest included in each sequence. Using this ongoing data, the process (370) also functions for virtual time synchronization with the rolling shutter camera. Figure 5 illustrates an example process for interpolating between specified 2D positions within particular image frames acquired from a rolling shutter camera.
[0076] A first observation time for a first location having a 2D position among the specified 2D positions in a particular image frame is calculated by adding a first time offset value for the first location to a timestamp of a first image frame among the particular image frames (500). For example, a time offset (dt) of a first blob in a first frame (as detailed above in Tables 3 and 7) may be added to a timestamp (t_sync) of that first frame (as detailed above in Tables 2, 3, 6, and 7). A second observation time for a second location having another 2D position among the specified 2D positions in a particular image frame is calculated by adding a second time offset value for the second location to a timestamp of a second image frame among the particular image frames (510). For example, the time offset (dt) of a second blob in a second frame (as detailed above in Tables 3 and 7) may be added to the timestamp (t_sync) of that second frame (as detailed above in Tables 2, 3, 6, and 7). Interpolation is then performed (520) using the first and second observation times calculated from the time offsets and frame timestamps.
[0077] Returning to FIG. 3 , once the virtual 2D position is generated, a 3D track of the object (e.g., ball) moving in 3D space is constructed (375) (e.g., by computer 140, 160) using the generated virtual 2D position and position information obtained from at least one other sensor for a given time. Constructing (375) may be for display (e.g., immediate display) of the object's 3D track, or constructing (375) may generate a 3D track for use as input to further processing before display. For example, the 3D track may be further processed to be effectively displayed by overlaying the golf ball's trace on a TV signal generated for live transmission or recording. As another example, the 3D track may be further processed to be effectively displayed by augmenting other data or media, for example, by showing the path of the golf ball in a computer representation of the physical environment in which the golfer is located, or in a virtual environment that exists only in the computer but is displayed to the user of the system.
[0078] Other types of further processing before display are also possible. For example, the 3D track can be further processed to determine the final resting position of a sports ball, which may be useful for betting applications or general statistics gathering to feed sports websites. In addition to showing the trace, the 3D track can be further processed to measure the speed, spin, carry, and launch angle of the shot. The 3D track can be further processed to determine whether the ball went over the site's net and into an adjacent building. The 3D track can also be further processed to tell the range owner which bays have ongoing activity and to count the number of balls hit from each bay.
[0079] 6A illustrates an example process for constructing a 3D track of a moving object (e.g., a ball). 3D track construction involves combining (e.g., by computers 140, 160) a virtual 2D position with position information obtained from at least one other sensor to form 600 a 3D position of the object of interest. Generally, this involves triangulating observations from different sensors, using object observation data generated using those different sensors, and using calibration data for those different sensors. Note that the generated observation data is available for triangulation because it has been virtual time-synchronized by generating the virtual 2D position.
[0080] For example, the other sensor may be a second camera used as a stereo pair with the first camera from which a virtual 2D position is generated. The object detection, 2D tracking, and virtual time synchronization techniques described above may be used with the second camera. In light of this, the output data for the second camera may generate multiple detected objects (with multiple corresponding virtual 2D positions) at the same common point in time at which the virtual 2D position is generated for the first camera. Thus, the position information for the second camera may be two or more 2D positions obtained from the second camera, and combining (600) may include determining which of the two or more 2D positions from the second camera should be matched with the virtual 2D position generated for the data from the first camera.
[0081] Furthermore, combining (600) may involve excluding (602) at least one, but not all, of two or more 2D positions acquired from the second camera as being unable to form a 3D point with the virtual 2D position acquired from the first camera. Figure 6B illustrates an example process for excluding at least one 2D position acquired from the second camera to perform epipolar line filtering prior to triangulation (604). Using the virtual 2D position generated for the first camera, the optical center of the first camera, the optical center of the second camera, a baseline between the first camera and the second camera, and external calibration data for the first camera and the second camera, a region around at least a portion of the epipolar line in the image plane of the second camera is determined (602A). In some implementations, the region is determined (602A) based on an error in the data being used.
[0082] In some implementations, the baseline between the first camera 120 and the second camera 130 of a stereo camera pair is 30 meters or less, which provides reasonable stereo accuracy combined with the ability to detect the shot early enough to estimate its trajectory back to where it landed. Using a lens with a wider field of view allows for earlier observation of the shot, but at the cost of lower accuracy if the ball is farther away. A higher-resolution camera can alleviate this problem to some extent. A shorter baseline allows for earlier observation of the shot, at the expense of lower depth accuracy. Note that using the virtual time synchronization described in this disclosure, data from different pairs of cameras 120, 130 can be dynamically combined as needed, for example, in real time. Thus, different pairs of cameras 120, 130 in a system can form stereo pairs, and therefore different stereo camera pairs can have different baselines and different depth accuracies.
[0083] Additionally, in some implementations, the determined (602A) region is reduced (or further reduced) based on a portion of the epipolar line that cannot be used due to known limitations on the distance to the object, e.g., a known distance between the camera and the teeing area 112 and / or the target 114 (602A). Additionally, other criteria can be used when matching observations of a potential ball from different cameras used as one or more stereo pairs. For example, an additional criterion can be that the blob contrast of each blob observed by the two cameras (used as a stereo pair) is similar.
[0084] Next, pairings of the virtual 2D position acquired from the first camera with each of the two or more 2D positions acquired from the second camera are rejected (602B) in response to each of the two or more 2D positions being outside a region around at least a portion of an epipolar line in the image plane of the second camera. In essence, a line in 3D space seen by the first camera as a single point in the 2D plane of the first camera (because this line is directly coincident with the optical center of the first camera) is seen by the second camera as a line in the image plane of the second camera (known as the epipolar line). Given the known 2D position of the object observed by the first camera, only objects observed by the second camera that fall along this epipolar line (within a certain tolerance value) can be the same object observed by both cameras. Additionally, if the system is designed and configured with known limits on what distances to objects should be considered (e.g., distance based on distance to tee area 112, distance to target 114 such as distance to the green when tracking a ball onto the green, and / or distance(s) to the currently tracked object), the system can place hard stops (boundaries) on the epipolar line for use in rejecting (602B). Thus, objects that are clearly outside the boundaries (e.g., airplanes, birds, lights from distant traffic, etc.) can be easily ignored by the system.
[0085] Other approaches to filtering out 2D positions obtained from the second camera are also possible. For example, as described above, the output data may include an indication of the previous position in each sequence for each position after the initial position in each sequence. Because this data is included (e.g., because transmitting (330) transmits not only the position of each blob but also the positions of previously observed blobs connected by 2D tracking in the path sequence), the possible 2D motion direction and velocity of each object of interest (blob) can be estimated, and this estimation can be used for motion filtering before triangulation (604). For example, a stereo triangulation algorithm can use this information to reject blob pairings with incompatible parameters, for example, by using velocity vectors of left and right stereo points to filter out false positives. For example, if the position of an object in the next frame is predicted (based on its current velocity) in both cameras, the predicted position (when triangulated) will also be close to the next actual observation, when the blob is actually an observation of the same object from two different angles.
[0086] Thus, the eliminating (602) may involve estimating a 2D velocity for the object at a virtual 2D position based on specified 2D positions in a particular image frame, at least one of the specified 2D positions being identified in the output data using a representation of a previous position in at least one of the sequences; estimating a 2D velocity for the object at two or more 2D positions acquired from the second camera; and rejecting pairings of the virtual 2D position acquired from the first camera with each of the two or more 2D positions acquired from the second camera based on the object at the virtual 2D position and the estimated 2D velocities for the object at the two or more 2D positions, as well as the internal calibration data and the external calibration.
[0087] Note that, as will be understood by those skilled in the art, the coordinates of the positions / blobs need to be undistorted using internal calibration data and further transformed into a compatible coordinate system using external calibration data. Internal calibration can be performed to find the optical distortion and true focal length of each camera. External calibration can be performed to find the orientation of the cameras relative to each other. Internal and external calibrations, together referred to as "calibration" or "calibration data," can be used to triangulate the position of objects seen by both cameras.
[0088] Additionally, other sensors may include radar devices rather than a second camera. For example, one or more radar devices may be combined with a single camera, a stereo camera pair, or three or more cameras to form one or more hybrid camera / radar sensors to provide at least a portion of the sensors of an object detection system that registers objects of interest for 3D object tracking, according to the systems and techniques described in U.S. Patent No. 10,596,416. Readings from the radar(s), which may be detections of moving objects (e.g., golf balls), may be added to a 3D point cloud of the three-dimensional position of the object of interest, which may then add to the potential robustness and coverage area of the combined sensors.
[0089] Readings from the radar can be converted into 3D points using the following approach. The distance measurements from the radar can be combined with all ball candidates detected by one of the cameras at the same time. This can include combining the radar with ball candidates detected by only one camera of the stereo camera pair and / or with each ball candidate detected by both cameras of the stereo camera pair. Note that one stereo camera and one radar device can operate as three separate sensors (stereo camera sensor, radar + left camera sensor, and radar + right camera sensor) that can generate 3D points for the ball candidates. The angles reported by the cameras to each ball candidate and the distance reported by the radar determine its 3D location in space. This is done for each ball candidate seen by the camera, generating an array of 3D points to add to the point cloud. Only one of these points will be the correct association between the radar data and the ball observation (i.e., they both come from the same object—a true positive). The remaining 3D points will be incorrect associations (false positives). However, the 3D point tracking algorithm is selected to be robust against a large proportion of 3D points that are false positives. Additionally, in some implementations, one or more radar sensors that can determine both the range and angle to each observed object may be used in the system.
[0090] However, in some implementations, regardless of whether or how position data from a second sensor is acquired and / or excluded, the 3D position of the object of interest may be triangulated using the virtual 2D position acquired from the first camera, at least one 2D position of the two or more 2D positions acquired from the second camera, internal calibration data for the first and second cameras (the determined optical distortion and true focal length of each camera), and external calibration data for the first and second cameras (the determined orientations of the cameras relative to each other) (604). Note that because the coordinates of the positions / blobs all represent the same point in time, triangulation between points (604) is possible even if the original captures were not synchronized.
[0091] The 3D position of the object of interest may then be added to other 3D positions of the object of interest in the cloud of 3D positions of the object of interest for a given time point (610). Note that various types of data structures can be used. In some implementations, the cloud of 3D positions is stored as an octree; this type of data structure allows any point in 3D space to be represented (up to the precision limit of computer representation of scalar numbers). A motion analysis is performed across multiple clouds of 3D positions (620) to construct a 3D track of the ball moving in 3D space, where each of the multiple clouds is for a single time point. This motion analysis may be performed during hypothesis formation, hypothesis expansion, hypothesis branching, and / or as a final "golf ball" classification to check whether the trajectory hypotheses sufficiently correspond to the expected motion of the golf ball in flight in 3D space.
[0092] It should be noted that various types of motion analysis can be used. However, it should be understood that even if the number of false positives contained in the data 350 is higher than that of traditional object tracking techniques, these false positives tend to be spread across a 3D space that is large compared to the 2D image plane of the camera in which the false positives occurred. Thus, once the 3D tracking system identifies an object undergoing 3D movement, it is straightforward for the 3D tracking system to easily discard any false positives in the 3D space that do not match the ongoing 3D track being constructed. Thus, 3D tracking is robust enough to handle false positives that end up in the 3D cloud. This is because (1) the majority of false positives are randomly spread in 3D space, making it difficult for them to create coherent trajectories (according to motion analysis); (2) if a false positive is close to a hypothesis of an actual object in flight and is added to that hypothesis, 3D tracking can branch (split) the hypothesis when it receives one or more subsequent true observations of that object in flight that do not match the false positive, and since branches containing true observations are much more likely to continue than branches containing false positives, this leads to branches containing false positives being discarded in most (if not all) cases; and (3) in some implementations, the tracking positions are smoothed (e.g., after 3D tracking and / or using a Kalman filter or low-pass filter) to increase accuracy with respect to the actual object in motion, but this also has the effect of effectively removing false positives added during hypothesis formation because they tend to differ by a random amount compared to the actual observations and thus cancel out during smoothing. Additionally, note that hypotheses that contain false positives but have coherent motion (typically the trajectories of objects other than a golf ball) can still be rejected by the "golf ball" classification (either before or after any smoothing), which checks whether the hypotheses sufficiently correspond to the expected motion of a golf ball in flight in 3D space.
[0093] Further, the process may include outputting 630 the 3D track for generation of a representation of the 3D track, or generating and displaying 630 a representation of the 3D track in 3D space. As noted above, this may involve further processing the 3D track to effectively display the 3D track by overlaying the trace of the golf ball on a TV signal generated for live transmission or recording, or it may involve further processing the 3D track to effectively display the 3D track by augmenting other data or media, for example, by showing the path of the golf ball in a computer representation of the physical environment in which the golfer is located, or in a virtual environment that exists only on the computer but is displayed to users of the system, such as a virtual environment shared among participants in a multiplayer game (either locally in the same area or scattered around the world).
[0094] FIG. 7 illustrates an example process for 3D object tracking of a moving object using unverified detections registered by two or more sensors. A three-dimensional (3D) position of an object of interest registered by an object detection system is acquired (700). In some implementations, acquiring (700) involves generating the 3D position of the object of interest using the object detection systems and techniques described above in connection with FIGS. 3-6B. In some implementations, acquiring (700) involves one or more other techniques for generating the 3D position of the object of interest. In some implementations, acquiring (700) involves receiving the 3D position of the registered object of interest from the object detection systems and techniques described above and / or other object detection systems and techniques.
[0095] In either case, the object detection system is configured to postpone strict classification of detections into “true positives” and “false positives” until a later stage in the tracking algorithm, thus allowing a significant amount of false positives for registered objects of interest to minimize false negatives. False positives can include both erroneous detections from individual sensors (e.g., detections from a single sensor when the 3D location of the detection can be determined alone, such as a phased array radar with both angle and distance measurement capabilities) and incorrect combinations of detections from sensors, which may include two or more cameras, camera and radar devices, or stereo camera and radar devices. In some implementations, the majority of registered objects of interest are false positives. In some implementations, the majority of false positives can be 60% or more, 70% or more, 80% or more, 90% or more, 95% or more, or 99% or more of the registered objects of interest (generated by the object detection system) are false positives.
[0096] Additionally, multiple sensors 130 may be used to improve the robustness of the system for detecting a moving ball without overloading the 3D tracking algorithm. In some implementations, acquiring (700) involves generating a 3D position of the registered object of interest by creating different combinations of data from three or more sensors 130. The three or more sensors 130 may include three cameras 130, and creating respective combinations of detections from each pair of the three or more sensors for the current time slice may include using two or three stereo pairings of the three cameras to combine detections of a single object of interest by the three cameras to generate two or three respective three-dimensional positions of the single object of interest. Thus, any combination of two of the three cameras can be used to form a stereo pair, and if one of the three cameras fails to detect the object in a particular time slice, the 3D tracking still has a sensed 3D position to work with, and if all three cameras detect the object in a particular time slice, three separate 3D positions can be added to the 3D point cloud of the object in that time slice, in which case one of these three is likely to be better than the other two. This can therefore add redundancy to the data acquisition and improve 3D tracking.
[0097] Additionally, the three or more sensors 130 may include two cameras 130 and a radar device 130, and creating a respective combination of detections from each pair of the three or more sensors for the current time slice may include using stereo pairing of the two cameras to combine detections of a single object of interest by the two cameras to generate a first three-dimensional position of the single object of interest, and combining detections of a single object of interest by the radar device and one of the two cameras to generate a second three-dimensional position of the single object of interest. As with the three cameras, this also adds redundancy and can improve 3D tracking. In some implementations, data from the radar device is combined separately with data from each of the two cameras to generate three distinct 3D positions in 3D space. In some implementations, more sensors (e.g., cameras and radar devices) can be added to the system to further increase redundancy without overloading the 3D tracking. Furthermore, 3D tracking can be improved by this additional redundancy, in that noise in the input 3D position data can be at least partially counteracted by the system's ability to select the best available observations to extend each hypothesis.
[0098] In either case, once the 3D positions of the objects of interest are obtained (700) (e.g., in real time, in batches, etc.), working hypotheses are formed (710) from the 3D positions, with each hypothesis representing an object moving in 3D space. Forming hypotheses of the objects moving in 3D space (710) uses a slow filter applied to the 3D positions of the registered objects of interest. If the estimated 3D velocity vectors for a particular object of interest approximately correspond to the object moving in three-dimensional space over time, the slow filter enables association between particular objects of interest among the registered objects of interest. Note that each velocity vector is an estimate of the direction in 3D space and the velocity (in that direction) for the object of interest (which may or may not be a ball, since the object has not been verified at the time of hypothesis formation (710)). Furthermore, the hypotheses may include at least some hypotheses that each branch into two or more hypotheses of the object moving in 3D space, as described in more detail below.
[0099] The "roughness" in this case is similar to that described above for 2D tracking, in that the 3D tracker attempts to find a sequence of objects of interest in 3D space that exhibits object movements consistent with those of objects in Newtonian motion, without first considering other forces such as wind, air resistance, or friction, and possibly even ignoring gravity, i.e., without assessing whether the object's acceleration fully matches the physics of a golf ball in flight at this stage. Thus, verification that the apparent forces acting on the object are consistent with a golf ball in flight is deferred until a later stage in 3D tracking.
[0100] Furthermore, in some implementations, the 3D tracking component does more than simply have a cooperating 3D point cloud. For example, as described above, pre-tracking in 2D may be performed on data from one or more camera sensors 130; e.g., a golf ball may be tracked independently in 2D data from each camera to the extent possible to generate several 2D tracks of the ball; thus, the data provided to the 3D tracking process may include partial segments of the object trajectory in addition to a single object detection. When attempting to match detections between different sensors, partial trajectories (formed by pre-tracking) may be matched against each other (whether the partial trajectories completely overlap or not), and the 3D tracking process may compare and connect partial segments of the trajectory in 3D instead of (or in addition to) points in 3D as part of forming a hypothesis (710) of an object moving in 3D space. Note that this approach may reduce the number of false ball detections, which may simplify one or more aspects of 3D tracking, but it also requires a more complex image tracking process, which may be undesirable in some cases.
[0101] In some implementations, forming (710) involves checking (712) whether the number of object detections for a given hypothesis is greater than a threshold. If the number of registered detections for a given hypothesis is less than or equal to the threshold (or simply less than the threshold, which may be equal depending on the threshold used), a first motion model in 3D space is used (714), the first model having a first complexity level. Also, if the number of registered detections for a given hypothesis is greater than the threshold (or greater than or equal to the threshold, which may be equal depending on the threshold used), a second motion model in 3D space is used (716), the second model having a second complexity level higher than the first complexity level of the first model.
[0102] The first model forms the first stage of processing, using a very loose filter for what constitutes an object in Newtonian motion. This allows for very rapid, but also extensive, hypothesis generation, as random false positives can easily form initial hypotheses in this first stage. However, because of this (combined with the number of false positives fed into the hypothesis generation pipeline), it is nearly impossible to miss any actual object in 3D space that exhibits the desired motion. The second model is more complex than the first model in that some knowledge of 3D physics (not necessarily a complete 3D physics model) is used in the second stage to prevent the continued formation of erroneous hypotheses.
[0103] Note that this approach is in contrast to traditional 3D tracking, which uses very stringent filtering early in the tracking pipeline to avoid burdening, e.g., overloading, the processing power of systems further downstream in the tracking pipeline with false positives. In contrast to this traditional approach, the systems and techniques of the present invention delay / defer filtering as much as possible until later stages in the tracking pipeline, when more information about object motion becomes available for filtering hypotheses that should never have been formed in the first place—for example, when combining information about movement and position in 3D space with other information, such as whether object size information matches the triangulated distance from the sensor(s). That is, when filtering can be deferred to later parts of the tracking pipeline, as described herein, more advanced filtering techniques can be used efficiently while minimizing the risk of missing actual objects in flight.
[0104] This approach may also be of particular value when used with a sensor or sensors positioned near the expected target of an object in flight, because over time, the object slows down and its relative size increases within the field of view of such a sensor or sensors. As described, a system could potentially employ both a camera and a radar sensor, including a stereo camera, which is the only sensor in the system, or a combination of data from two separate (and selectable) cameras in the system. In the case of a golf shot, when one or more sensors are positioned on the target (e.g., the green of a golf course), data from the radar sensor is most useful for tracking the ball in flight from the impact location toward the target, while data from the camera sensor is less useful for detecting this initial portion of the ball trajectory when the ball is far from the sensor, depending on the lens and baseline selection used for tracking with the stereo vision. Thus, when two cameras are not combined with a stereo vision system with a wide baseline, radar may provide better tracking data for the flight segment, and a stereo camera system may provide better tracking data for the bounce and roll segments.
[0105] Similarly, if one or more sensors are placed at the launch location (e.g., a tee), data from both radar and stereo camera systems can be useful for tracking the ball from launch, even if the stereo system's baseline is short, but the accuracy of the stereo vision system degrades more quickly than the radar system. If the baseline is short enough, the stereo vision system is likely to lose track of the ball's 3D track when the 3D accuracy is so low that the ball cannot be detected in 3D space. However, if the baseline is long enough, this can be avoided, and radar data may not be necessary for 3D object tracking.
[0106] It should be noted that some system deployments may use sensors at the target (e.g., green), the ball launch location (e.g., tee), and one or more locations between them (e.g., along the fairway), and the type of sensor (e.g., radar vs. camera) and sensor configuration (e.g., baseline for a two-camera stereo vision system) selected for use at a particular location may be determined according to the expected nature of golf shots in the vicinity of each sensor. In cases where the bounce and roll segments of the ball's path in 3D space may be far away from the sensors, additional cameras may be used with different optics (able to zoom in more, have a longer focal length, etc.) to get a better view of the bounce and roll.
[0107] Therefore, the specific set of sensor types, configurations, and placements used can be tailored to the specific deployment of the system, such as two or more cameras with at least four feet (approximately 121.91 cm) between each pair of cameras, such as two cameras mounted on the horizontal beam of a single tripod, or a hybrid camera / radar sensor unit near the tee (to provide high-quality tracking data for the flight segment) combined with a stereo camera unit near the target (to provide high-quality tracking data for the bounce and roll segments), each with wide-angle optics providing a field of view of over 90 degrees, allowing coverage of the entire golf green even in tight spaces. With a four-foot baseline, measurement error can be + / - 3 inches for a typical golf green. Accuracy can be further improved by adding one or more layers of computer vision when the ball comes to a complete stop. This triangulates the ball's resting position based on the center of the ball observed by both cameras, increasing accuracy.
[0108] Furthermore, in some implementations, the first motion model in 3D space is a linear model that associates the first 3D position of the registered object of interest with a given hypothesis when the first 3D position extends through the given hypothesis in approximately the same direction as predicted by the given hypothesis. For example, the first model can be a simple dot product check on a vector defined by three 3D points P1, P2, and P3 representing the registered object of interest (potentially, one or both of P1 and P2 are 3D positions predicted by the active hypothesis). Vector i→(i+1) (P2→P3) and vector (i-1)→i (P1 → P2) (or potentially the vector (i-1)→i (P1→P2) and vector (i-1)→(i+1) If the dot product (with P1 → P3) is positive, i.e., the angle between the previous and new velocity vectors is less than 90 degrees, this can be considered sufficiently close to Newtonian motion in 3D space to connect the specific 3D points P1, P2, and P3 in the formed hypothesis. Also, in some implementations, the threshold angle between the previous and new velocity vectors is a programmable parameter that can be experimentally tuned for a given implementation after deployment.
[0109] While other types of first models are possible, each such first model should be very simple, as this facilitates rapid initial hypothesis formation and minimizes the risk of new objects flying in 3D space going undetected. Note that the system may be designed so that any hypotheses formed using only the first model are never sent as output, thus requiring some degree of expansion of the hypotheses with a second model before they can be output for validation of the flying object. Furthermore, forming hypotheses (710) may include dividing the hypotheses into distinct segments (718), and in some implementations, dividing (718) is performed only for hypotheses that have been expanded at least once using a second model of movement in 3D space.
[0110] In some implementations, the second motion model in 3D space is a recurrent neural network (RNN) model trained to perform short-term predictions of motion in 3D space. In some implementations, the second motion model in 3D space is a numerical physics model that employs numerical integration combined with numerical optimization. Other types of second models are also possible, such as another machine learning model that directly maps observations to future positions, or a set of differential equations (more complex than a linear model) to describe how the ball will fly in the future given current observations. These differential equations can be solved analytically (analytical physics model) or using numerical methods, potentially also using one or more Kalman filters (considered low-pass filtering) to remove noise in the data in support of numerical optimization. Different models have different tradeoffs in terms of performance, accuracy, and noise sensitivity. Thus, the particular type of second model used may depend in part on the particular deployment of the system, for example, on the nature of the sensor(s) 130 used in the system and whether those sensor(s) 130 are located on one, two, three, or all sides of the 3D space 110.
[0111] FIG. 8 illustrates an example of a hypothesis formation process that may be used in the 3D object tracking process of FIG. 7. The 3D positions of an object of interest registered by the object detection system for the next time slice are received (800). The 3D positions may be stored in an octree or another data structure. An octree data structure allows for rapid answers to queries such as, "Which point is x meters from point y?" In general, the received 3D position data may be understood as a point cloud representing the unidentified object in 3D space at a given time slice. Furthermore, the time slice may be one of multiple time slices at a predefined time increment, e.g., a predefined constant frame rate of the 3D tracking server referenced above.
[0112] A check (805) is performed to determine whether any active hypotheses remain to be processed using the 3D position data of the current time slice. Each hypothesis may include models of the hypothesis' past and future trajectories. For each active hypothesis remaining to be processed, a prediction may be made as to where in 3D space the next sensor registration of the potential moving object represented by the given hypothesis will be found according to the number of registered detections for the given hypothesis and a threshold value. Thus, a check (810) may be performed as to whether the number of object detections for the given hypothesis is greater than a threshold value (similar to check 712 in FIG. 7). If the number of registered detections for the given hypothesis is less than or equal to the threshold value (or may simply be less than the threshold value, and may be equal, depending on the threshold value used), a first model is used to predict the object's next 3D position for the given hypothesis (815). If the number of registered detections for the given hypothesis is greater than or equal to the threshold value (or may be greater than or equal to the threshold value, and may be equal, depending on the threshold value used), a second model is used to predict the object's next 3D position for the given hypothesis (820).
[0113] For example, for the first few observations of a hypothesis, a simple linear model may be used as a model of the ball's flight. After a predefined number of observations have been made, a recurrent neural network (RNN) (or a suitable motion of second-order complexity) may instead be used as a model of the ball's flight. In some implementations, the predefined number of observations is 10, although other numbers may be used. Generally, the predefined number should be set to a value less than the number of observations typically required for a golf ball to begin to exhibit curved rather than straight motion, e.g., the number of sensor observations in the system corresponding to approximately 0.3 seconds into the ball's flight, near the edge before the ball begins to exhibit curved motion.
[0114] In some implementations, the RNN only needs to make short-term predictions, which allows for the use of less complex deep learning-based models, e.g., less wide and / or deep model architectures, leading to faster predictions. For RNN implementations, predictions can be made from the last state of the model and can predict several steps ahead, even if not necessary. These predictions do not require updating the model state. To enable faster predictions, predictions for all hypotheses can be made in batches. In each batch, predictions for multiple hypotheses can be made simultaneously using vector instructions, single instruction multiple data (SIMD).
[0115] In some implementations, rather than using an RNN as the second model, a more advanced physics model may be used after, for example, 10 observations. This physics model may be a traditional numerical physics model that uses numerical integration of measured aerodynamic properties such as the ball's drag, lift, and spin damping. This physics model may be a full physics model or a reduced physics model that simplifies one or more aspects of the ball's flight to reduce processing time and, therefore, speed up the prediction of the object's next 3D position for the hypothesis. In some implementations, the state, velocity, and spin of the physics model are continuously updated during the hypothesis. This may be performed by first selecting a subset of the observed 3D points toward the end of the hypothesis. These 3D points may be low-pass filtered, and 3D velocity and acceleration vectors may be calculated from the filtered data. An optimization may then be formulated using the selected physics model to solve for the ball's 3D spin vector. Numerical integration may then be applied to the estimated 3D velocity and spin vector to arrive at the object's predicted position for the current time slice.
[0116] Using the predicted positions allows for more rapid searching of the 3D position data for the current time slice to find any registered objects of interest that can be used to extend a given hypothesis. For example, an octree data structure containing all of the 3D positions of objects of interest registered by the object detection system for this next time slice can be searched (825) to find the set of 3D positions that are within a predefined distance from the predicted next three-dimensional position. The octree can be queried for points that are close to the predicted position, and each 3D point can be determined to be “close” when it is within a predefined distance of the predicted position. Note that using a data structure like an octree, which allows for efficient (faster) searching among points in the 3D point cloud for the current time slice (so that not every point in the 3D cloud needs to be checked), is an important part of achieving real-time performance even when the 3D point cloud contains many false positives. Note also that spatial data structures other than octrees, such as KD trees, can be used to enable fast searching by these types of queries for points that are close to the predicted points for the current hypothesis.
[0117] Furthermore, in some implementations, each 3D point is determined to be "close" only if it (1) is within a predefined distance of the predicted position and (2) exhibits movement in the same general direction as the hypothesis. For example, each hypothesis may include an estimate of the current direction the ball is moving (estimated velocity vector), and a new direction vector may be calculated based on the same anchor points used to calculate the new potential 3D point and ball direction. The 3D point is then accepted only if the angle between the current estimated direction and the new direction is 90 degrees or less (i.e., a positive dot product). Note that even if a second model is used to predict the next 3D position for a given hypothesis, a simple linear motion model may be used to limit the 3D points considered for addition to the given hypothesis, as this provides a quick way to eliminate bad 3D points from consideration for extending the current hypothesis.
[0118] If the search (825) returns zero (0) hits (i.e., the set is an empty set), the predicted 3D position for the given hypothesis can be used (830) to extend the given hypothesis through the current time slice, and the process then returns to checking for the next active hypothesis to be processed (805). If the search (825) returns one (1) hit or a number (#) of hits that is less than or equal to a predefined threshold number controlling the number of branches allowed for each hypothesis in each time slice (e.g., a branching threshold number greater than 2 if the branching number is programmable), one or more discovered 3D points are used to extend the given hypothesis (branch the hypothesis if necessary) (835), and the process then returns to checking for the next active hypothesis to be processed (805).
[0119] If the set of 3D points returned by the search (825) contains more than a branching threshold number (#), the discovered 3D points can be sorted (840) based on their proximity to the predicted next 3D location to form a sorted set. For example, the 3D points can be scored and then ranked (840) during the search (825) according to, for example, the squared distance L2 to the predicted location. 3D points in the sorted set that are less close than a predefined branching threshold number can be removed (845). A given hypothesis is then branched into two or more hypotheses in a group of hypotheses using two or more 3D points remaining in the sorted set (850). The process then returns to checking for the next active hypothesis to be processed (805). Note also that although the set of hypotheses that branch from the initial hypothesis can be stored as a group, each hypothesis branch can be tracked individually and returned to the branching point when that hypothesis dies.
[0120] The process repeats while hypotheses remain to be checked (805). However, while the flowchart depicts a sequential process for each hypothesis, it should be appreciated that in some implementations, hypotheses are checked in parallel. Thus, predictions may be made for all active hypotheses (815, 820) before the octree is searched for 3D points close to the predicted location (825). In any case, all 3D positions of objects of interest registered by the object detection system for the current time slice that are not added to an existing hypothesis may be added to a list of unmatched 3D points for further processing. Additionally, in some implementations, if a hypothesis rapidly dies (e.g., before the hypothesis grows to three or more, four or more, or five or more object detections), the 3D points (corresponding to sensor impressions) used in the defunct hypothesis may be added to a list of unmatched points to reduce the risk of missing a new ball launch due to the sensor impression being "stealthed" by a stagnant hypothesis.
[0121] Additionally, regardless of the nature of the second model used (e.g., an RNN model or a numerically integrated physics model), note that updates to each hypothesis can be delayed. Once work on the hypothesis for the current time slice is complete, the second model is updated. If a new point is added, that point is used to update the model. Otherwise, the model's position prediction is used instead. In either case, updates can be "delayed." When a new point is added, it does not need to be used to update the model immediately. This is because raw 3D observations are somewhat noisy and can confuse the model. This delay can be done as follows: The position used to update the model is delayed by a number of frames from the last added position. When a point is used to update the model, it can be smoothed by positions both before and after the point. In some implementations, a window size of 9 is used, meaning that four points behind and four points before are required before the model is updated. In such implementations, special processing is performed for the first part of the trajectory.
[0122] 3D positions that do not match existing hypotheses are used to detect potential new balls and new hypothetical launches (860). This may involve using the octree(s) for the current time slice and stored additional octrees for previous time slices. To reduce the number of potential launches (particularly for launches in the same vicinity), points closer than ½ meter to each other may be clustered, and points with the lowest measurement error, such as reprojection error for a stereo camera sensor, may be retained in the cluster. The remaining points are added to the octree. In some implementations, octrees for two previous time slices are also retained.
[0123] In some implementations, clustering involves finding points that are less than a threshold distance (e.g., 2 meters or 1 meter) apart (for each point). These points are considered clusters, and if a point belongs to two or more clusters, the two clusters may be merged into one. A cluster is thus a collection of points that are close together, and therefore may correspond to the same object. To avoid generating more than one launch within the system for the same object, the collection of points in each cluster is reduced to one point; the metric used to do this may be reprojection error, with the point with the lowest reprojection error selected to be the point that represents the cluster. In some implementations, a new point cloud is created using the individual points representing the identified clusters; this new point cloud (1) should contain fewer points than the original point cloud for this time slice, and (2) may be used to detect launches (and only launches). Note that the new point cloud may be stored in a spatial data structure, such as an octree, to enable faster searching and facilitate real-time launch detection.
[0124] Launch detection (860) may be designed to find movement that is approximately linear (i.e., in a straight line). In some implementations, this is done by iterating through all remaining 3D points and searching the octree for a nearby 3D point (e.g., within 5 meters) from two time slices prior. A line is then calculated between these two points. This line may then be verified by predicting the ball's position in the time slice between these two points and searching the octree in the previous time slice for a 3D point near this predicted position. If all of these criteria are met, a new hypothesis is created (860) that includes these three points.
[0125] Further, as described above, forming hypotheses (710) may include dividing hypotheses into distinct segments (718). Throughout the duration of a ball's movement in 3D space, e.g., the duration of a golf shot, the movement enters different phases or segments, such as flight, first bounce, second bounce, and roll. Depending on which segment the shot is in, the physics model may be different. That is, the physics model / parameters used for the flight segment may be different from the physics model / parameters used for the roll segment, which may be different from the physics model / parameters used for the skid segment. Additionally, there may be practical advantages to switching the physics / prediction model and / or parameter(s) used depending on which segment the shot is in, e.g., not using the same algorithm for both bounce and flight, since bounce typically has higher noise but a much less noticeable Magnus effect.
[0126] The process of dividing object motion into such segments (718) may be performed after all extensions have been made to the currently active hypothesis for the current time slice, or after each hypothesis has been extended. For example, each time a new point is added to a hypothesis, a determination may be made as to whether the point starts a new segment. Figure 9 illustrates an example of the process of dividing hypotheses into separate segments during hypothesis formation (718) that may be used in the 3D object tracking process of Figure 7. A data point within a given hypothesis is identified (900), the data point having a minimum vertical position.
[0127] For example, the new 3D position Pt x,y,z is added to the hypothesis for time slice t, the two previous 3D positions P(t-1) x,y,z and P(t-2) x,y,z For Y component P(t-1) y <P(t-2) y and Y component P(t-1) y <Pt yNote that the Y component here refers to the vertical position of the 3D point in the 3D coordinate system, and the "Y" position refers to the coordinate axis that defines the height above / below the ground surface. Thus, a local minimum in the Y position in 3D space indicates a possible collision with the ground. While other techniques for finding the minimum vertical position can be used, this technique has the advantage of being very quick and easy to determine. However, due to noise in the system, the number of false positives, and the risk that the ball may not be detected by any of the sensors in a given time slice (note that the 3D data point identified in the hypothesis as the minimum vertical position may be a predicted 3D point rather than a registered detection if no detection was registered in this time slice), another check is required to confirm that this represents a ground collision.
[0128] The vertical component of each estimated 3D velocity vector before and after the data point is checked 930 for a given hypothesis. For example, the vector P(t-2) x,y,z →P(t-1) x,y,z Y (vertical) component of the vector P(t-1) x,y,z →Pt x,y,z Check the Y (vertical) component of the data point P(t-1) to see if the first component is negative (or positive in the inverted coordinate system). x,y,z (sign indicating that the object is moving towards the ground after P(t-1)), and the second component is positive (or negative in the inverted coordinate system, in either case, at data point P(t-1) x,y,z followed by a symbol indicating that the object is moving away from the ground. If so, the data point is designated as a ground impact event (942); if not, the data point is not designated as a ground impact event (944).
[0129] In some implementations, two or more estimated velocity vectors are checked to overcome risks caused by noise in the system. Two or more estimated velocities both before and after the minimum vertical position data point may be averaged, and then the vertical (e.g., Y) component of each velocity vector average may be checked (930). While this may delay the identification of a ground collision by a number of time slices equal to one less than the number of velocity vectors averaged after a potential ground collision, this delay provides the corresponding advantage of more robust ground collision detection using noisy data. This may facilitate the use of less sophisticated sensor technology in the system, thus reducing the overall system cost.
[0130] Further, in some implementations, one or more time windows may be selected based on a noise level associated with the estimated 3D velocity vector, a shortest expected flight time on one or both sides of the data point, or both the noise level and the shortest expected flight time (910). An average of the estimated 3D velocity vectors before and after the data point may then be calculated using velocity vectors that fall within at least one time window on either side of the data point that is a potential ground collision. In some cases, a single averaging window is used, and in some cases, two averaging windows are used, with the averaging window used before the data point being a different length than the averaging window used after the data point.
[0131] Additionally, although shown in the figure as being included in the segmentation process itself, rather than dynamically (e.g., in real time) selecting (910) the time window(s) while hypotheses are being formed, the selecting (910) may be done periodically, e.g., outside of a processing loop, or potentially only once for a given system configuration having a particular set of sensors 130 positioned relative to a given 3D space 110, e.g., the window(s) may be selected (910) based on the nature of the sensor(s) 130 used in the system and whether those sensor(s) 130 are located on one, two, three, or all sides of the 3D space 110. Furthermore, if more advanced sensor technology, e.g., more accurate sensors 110 and / or higher data capture rates, is used in the system, additional checks of the vertical component of velocity may be used, such as checking whether the Y component of one or more velocity vectors is within a small range around 0 to help confirm that the ball is in a roll segment, or whether the rate of change of the Y component of the rolling average of the velocity vectors is within a small range around the Earth's gravitational constant G to confirm that the ball is in a flight or bounce segment.
[0132] In any case, once a ground impact is specified (942), a new segment is assigned to the hypothesis, and a determination is made as to whether this new segment is a bounce segment or a roll segment with respect to the previous segment. If the specified ground impact is the first ground impact specified for a given hypothesis (950), this ground impact divides the hypothesis into two segments, and the segment before the ground impact is classified (952) as a flight segment (e.g., corresponding to the initial portion of a golf shot). Next, a check (960) may be performed to determine the nature of the segment after the ground impact. In some implementations, the check (960) includes determining whether the angle between a first estimated velocity vector (e.g., average velocity vector) before the specified ground impact and a second estimated velocity vector (e.g., average velocity vector) after the specified ground impact is greater than a threshold angle (e.g., 20 degrees). If so, the segment after the ground impact is classified as a bounce segment (962). If not, the segment after the ground impact is classified as a roll segment (964).
[0133] In some implementations, the threshold angle between estimated velocity vectors is a programmable parameter that can be experimentally tuned for a given implementation after deployment. In some implementations, if the previous segment was a roll segment, the next segment cannot be a bounce segment. This logic would identify a "small" bounce as a roll, but the purpose is not to propagate this information to consumers of tracking data, but rather to modify tracking behavior, as described in more detail below. In some implementations, a roll segment can be a true roll or a skid, depending on the rotational component of the ball as it moves along the ground. Finally, note that a skid is typically followed by a roll, and the system can track the ball until it stops (or at least until its velocity falls below a threshold) and discard the detection of the stopped ball.
[0134] Returning to FIG. 7 , hypotheses that are not further expanded by associations made with at least one additional object of interest registered by the object detection system during forming (710) are eliminated (720). For example, a hypothesis may be marked as dead if the potential ball has been missed for more than x time slices, where x is set to different values depending on how many observations have been made. In some implementations, x is set to 3 for hypotheses with fewer than 10 observations, and 20 otherwise. Other values for x for hypotheses with different numbers of observations can be used, depending on the system and the frequency of sensor observations. Also, if all hypotheses in a group are dead, the entire group is marked as dead. In any case, over time, false detections and / or pairings (e.g., 3D positions generated from data from two sensors, either camera or radar, that are determined to be detections of the same object) are randomly distributed across a relatively large 3D space, so that incorrect hypotheses are eliminated relatively quickly, and the size of the volume of space makes it very unlikely to find a pattern of false detections and / or pairings that matches the movement of a golf ball in space.
[0135] Some hypotheses will remain after the elimination (720), and thus at least one 3D track of at least one ball moving in three-dimensional space is designated (730) by applying a complete three-dimensional physical model to data about the 3D position of the registered object of interest used in forming (710) the at least one hypothesis that remains after the elimination (720). In some implementations, at each time slice, the best hypothesis (or a set of active hypotheses that meet predefined criteria) is selected based on a scoring function that looks at hypothesis size and one or more characteristics of the sensor data (e.g., blob contrast), and these one or more hypotheses are then used to designate (730) at least one 3D track of the moving ball after determining a more accurate 3D path according to segmentation performed during forming (710), for example, based on a local minimum in vertical position and an expected average vertical component of an estimated 3D velocity vector before and after the local minimum vertical position.
[0136] FIG. 10A illustrates an example of a process for specifying a 3D track of a moving ball that may be used in the 3D object tracking process of FIG. 7. Data about the three-dimensional position of a registered object used in forming at least one hypothesis may be generated by processing sensor data to generate a more accurate 3D path using sensor observations registered by an object detection system in at least a flight segment (1000). This may include processing the sensor data to remove outliers and smooth the data, followed by recombining the processed sensor data to form 3D positions within the 3D path that have much less noise than the raw 3D combination (used during hypothesis formation) would produce. For example, in the case of two cameras combined into a stereo pair, this may involve performing triangulation again based on the smoothed 2D observations.
[0137] The 3D path is then analyzed to make a final determination of whether this is a golf ball. A full three-dimensional physics model may be applied 1005 to the generated data for at least the flight segment. The full 3D physics model may estimate the ball's spin, and balls with too much spin may be discarded. If the sensor observations (after processing 1000) are found 1010 to not match the ball in flight according to the applied 1005 full physics model, the data is rejected 1015 for output. If the sensor observations (after processing 1000) are found 1010 to match the ball in flight according to the applied 1005 full physics model, the data is confirmed 1020 for output.
[0138] Furthermore, in some implementations, only paths that move smoothly (small angular changes between each point) are allowed. This additional constraint is necessary when there is low spin (low acceleration) but the object does not move smoothly (the way a golf ball moves). Furthermore, rather than allowing or rejecting individual segments within a path, the entire path can be allowed or rejected. Individual points may still be rejected for being outliers. Furthermore, in some implementations, portions of a path that have an excessive number of outlier data points may be identified as bad and discarded.
[0139] FIG. 10B shows a detailed example 1050 of a process for specifying a 3D track of a moving ball that can be used in the 3D object tracking process of FIG. 7. This example corresponds to the case of two cameras combined in a stereo pair, but the process can be more generally applied to combining data from any two suitable sensors. Tracked points 1055 are received by a post-hypothesis process 1060, which recombines the processed sensor data to form 3D positions within a 3D path that are much less noisy than the raw 3D combination used during hypothesis formation. 2D outliers in the data (for the ball track) from the left camera (in the stereo pair) are removed (1062A), and 2D outliers in the data (for the ball track) from the right camera (in the stereo pair) are removed (1062B). The remaining 2D positions in the data (for the ball track) from the left camera are smoothed (1064A), and the remaining 2D positions in the data (for the ball track) from the right camera are smoothed (1064B). Note that smoothing may involve low-pass filtering as described above.
[0140] Non-overlapping data points are removed (1066). Outliers are removed independently for the two cameras (1062A, 1062B), so that only one of the two camera observations of the ball may be considered an outlier, leading to one of the cameras having a detection and the other not. Therefore, those time points with only one detection are removed. Next, the remaining data points (for the ball track) from the left and right cameras are triangulated (1068). This can generate a new set of 3D points, which can then be used to form a final 3D trajectory that is a more accurate representation of the object's actual path in 3D space (1070).
[0141] As noted above, the process of FIG. 10B may be applied more generally to combining data from any two suitable sensors. In general, once the tracking phase for a given ball is completed (1055), all detections for that ball can be used to refine the tracking trajectory (1060). Thus, position data coming later in the trajectory can be used to refine position data coming earlier in the trajectory. In the case of data from a camera plus radar, the camera data can be processed as described above, and the radar data can be processed using low-pass filtering and outlier detection to enhance the quality of the radar data.
[0142] Additionally, the final golf shot classification operation can calculate the spin and average angular change for at least a flight segment of the object's 3D path to confirm that the object is a golf ball (1075). The characteristic that determines a golf ball's trajectory is the effect that spin has on a golf ball in free flight. Therefore, estimating the ball's spin rate from the ball's observed flight segments helps determine whether the object's trajectory is a golf ball, as non-golf ball objects may generate unreasonably high spin rates. Reasonable values for golf ball spin range from 0 to 16,000 revolutions per minute (RPM), with some slack for imprecision in the estimate.
[0143] Furthermore, if the estimated spin for an object that is not a golf ball is very low, this can be detected with an additional test of the average angle change (related to smoothness of movement). Golf balls should have smooth movement through the air, and therefore objects that do not have this smooth movement during the flight segment of their trajectory should be discarded. For each point in the 3D path, a direction vector between that point and the previous point can be calculated. This direction vector can then be compared to the direction vector of the previous point in the 3D path, and the angle between the two vectors can be calculated. An arithmetic mean (average) over two or more of these angles (e.g., all angles within the flight segment) can be calculated (1075), with a large average value (e.g., greater than 10 degrees) indicating the object has jagged (rapidly changing) movement that is not the expected movement of a golf ball.
[0144] In such a case, the tracked object is rejected as not being a golf ball. If the object is accepted as a golf ball (or another type of object being tracked in a given implementation), the ball's 3D path is output (1080) for processing by another system / process and / or for display (e.g., for a video game or TV broadcast). Note that the 10-degree threshold for the average of the direction vector between points is only an example, and different implementations may use different thresholds. Generally, the threshold depends on the capture frequency of the sensor(s), as higher data capture rates can reduce the threshold because potential objects in free-flight motion travel shorter distances between each detection point. In some implementations, the threshold for the average direction vector is a programmable parameter that can be experimentally tuned for a given implementation after deployment.
[0145] Returning to FIG. 10A , in some implementations, processing is performed separately on each separate segment (1025). This processing (1025) may be part of the processing performed to generate data for ball confirmation (1000). For example, based on the path segmentation performed during 3D tracking, the trajectory may be segmented, and in each segment, observations may be smoothed and outliers may be removed. Combination (e.g., triangulation) may then be performed again based on the smoothed observations to generate a 3D path with less noise in the data. Note that the observations (e.g., 2D observations from paired cameras) are smoothed within the segment as opposed to smoothing across the entire trajectory. Otherwise, bounding points would be "smoothed" and their sharp edges softened. Note that by performing smoothing separately on the before- and after-bounce pieces, the sharp edges of the bounce points are preserved, and in some implementations, a more accurate 3D location for each bounce point can be determined based on the 3D path on either side of the identified bounce point, e.g., by extrapolating the two smoothed segments. Note that the finite sampling rate of the sensor means that the exact moment of the ball's bounce may not be captured, so looking for the intersection in space between the (extrapolated) path segments before and after the bounce may be the only way to estimate the actual bounce location.
[0146] As another example, processing 1025 may be performed separately on each of the distinct segments after the data has been validated for output 1020. This processing 1025 may use different techniques on each segment to validate key points in the 3D path, such as the precise 3D location of each bounce point and the precise 3D location of the ball's final rest point. The different techniques used on each segment may include different low-pass filtering techniques, different data smoothing techniques, and / or different ball motion modeling techniques.
[0147] In either case, the separate segments remain associated with each other as part of the same ball trajectory, and the final path data utilized for output may include data for each of the different identified segments, only portions of the identified segments (e.g., only the flight segments), and / or key identified points of the identified segments (e.g., the 3D position of each bounce point and the 3D position of the ball's final rest point).
[0148] Once the 3D path meets all criteria necessary to confirm that it is a real object in motion, e.g., an actual golf ball shot, the final path data is transmitted as an output. Referring again to FIG. 7 , the detected 3D track of the ball is output (740) for generation of a representation of the 3D track, or for generation and display of a representation of the 3D track in 3D space (740). This may involve further processing the 3D track to effectively display the 3D track by overlaying the golf ball trace on a TV signal generated for live transmission or recording, or it may involve further processing the 3D track to effectively display the 3D track by augmenting other data or media, for example, by showing the golf ball's path in a computer representation of the physical environment in which the golfer is located, or in a virtual environment that exists only on the computer but is displayed to users of the system, such as a virtual environment shared among participants in a multiplayer game (either locally in the same area or scattered around the world). 7-10B are shown as sequential operations for ease of understanding, it should be noted that in practice the operations may be performed in parallel or simultaneously, for example, using hardware and / or operating system-based multitasking and / or using pipelining techniques. Accordingly, the disclosure presented herein with respect to the figures is not limited to performing the operations sequentially as shown in the figures, except where the processes being performed require sequential operations with respect to particular inputs and outputs.
[0149] Embodiments of the subject matter and functional operations described herein may be implemented in digital electronic circuitry, or computer software, firmware, or hardware, including the structures disclosed herein and their structural equivalents, or one or more combinations thereof. Embodiments of the subject matter described herein may be implemented using one or more modules of computer program instructions encoded on a non-transitory computer-readable medium for execution by or to control the operation of a data processing apparatus. The non-transitory computer-readable medium may be a hard drive in a computer system, an optical disk sold through retail channels, or an article of manufacture such as an embedded system. The non-transitory computer-readable medium may be obtained separately and later encoded with one or more modules of computer program instructions, such as by distribution of one or more modules of computer program instructions over a wired or wireless network. The non-transitory computer-readable medium may be a machine-readable storage device, a machine-readable storage substrate, a memory device, or a combination of one or more of these.
[0150] The term "data processing apparatus" encompasses all apparatuses, devices, and machines for processing data, including, by way of example, a programmable processor, a computer, or multiple processors or computers. In addition to hardware, an apparatus may include code that creates an execution environment for the computer program, such as code comprising processor firmware, a protocol stack, a database management system, an operating system, a runtime environment, or one or more combinations thereof. Furthermore, an apparatus may employ a variety of different computing model infrastructures, such as web services, distributed computing infrastructures, and grid computing infrastructures.
[0151] A computer program (also known as a program, software, software application, script, or code) may be written in any suitable form of programming language, including compiled or interpreted, declarative or procedural, and may be deployed in any suitable form, for example, as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program may be stored as part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program, or in multiple cooperating files (e.g., files storing one or more modules, subprograms, or portions of code). A computer program may be deployed to be executed on one computer or on multiple computers at one location or distributed across multiple locations and interconnected by a communications network.
[0152] The processes and logic flows described herein may be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. Further, the processes and logic flows may be performed by, and an apparatus may be implemented as, special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).
[0153] Processors suitable for executing a computer program include, by way of example, both general-purpose and special-purpose microprocessors, and one or more processors of any kind of digital computer. Generally, a processor receives instructions and data from a read-only memory or a random-access memory, or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer may further include one or more mass storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data, or may be operatively coupled to a mass storage device for receiving data from or transferring data to the mass storage device, or both. However, a computer need not have such devices. Furthermore, a computer may be incorporated into another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name a few. Devices suitable for storing computer program instructions and data include, by way of example, all forms of non-volatile memory, media, and memory devices, including semiconductor memory devices (e.g., EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, CD-ROM and DVD-ROM disks, network-attached storage, and various forms of cloud storage. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.
[0154] To provide for user interaction, embodiments of the subject matter described herein may be implemented on a computer having a display device (e.g., an LCD (liquid crystal display), OLED (organic light emitting diode), or other monitor) for displaying information to the user, and a keyboard and pointing device (e.g., a mouse or trackball) by which the user provides input to the computer. Other types of devices may also be used to provide for user interaction. For example, feedback to the user may be sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback, and input from the user may be received in various forms, including acoustic, speech, or tactile input.
[0155] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communications network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. Embodiments of the subject matter described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), middleware components (e.g., an application server), or front-end components (e.g., a client computer having a graphical user interface or web browser through which a user can interact with an implementation of the subject matter described herein), or any suitable combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any suitable form or medium of digital data communication, e.g., a communications network. Examples of communications networks include local area networks (LANs) and wide area networks (WANs), internetworks (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks).
[0156] While this specification contains many implementation details, these should not be construed as limiting the scope of the invention or what may be claimed, but rather as descriptions of features specific to particular embodiments of the invention. Certain features described herein in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, while features may be described above as acting in a particular combination and initially claimed as such, one or more features from a claimed combination may, in some cases, be deleted from the combination, and the claimed combination may be directed to a subcombination or a variation of the subcombination. Thus, unless expressly specified otherwise or unless the knowledge of one of ordinary skill in the art clearly indicates otherwise, any feature of the above-described embodiments can be combined with any other feature of the above-described embodiments.
[0157] Similarly, while operations are shown in the figures in a particular order, it should not be understood that such operations need to be performed in the particular order or sequential order shown, or that all of the illustrated operations need to be performed, to achieve desired results. In certain situations, multitasking and / or parallel processing may be advantageous. Furthermore, the separation of various system components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the program components and systems described may generally be integrated together in a single software product or packaged in multiple software products.
[0158] Certain embodiments of the present invention have been described above. Other embodiments are within the scope of the following claims. For example, while the above description focuses on tracking the movement of a golf ball, the described systems and techniques are applicable to tracking the movement of other types of objects, such as baseball or skeet shooting, as well as non-sports applications. Furthermore, tracking a "moving" object, in some implementations, may include tracking the object as it bounces off the ground and / or rolls along the ground.
Claims
1. 1. A method comprising: Obtaining a three-dimensional position of the registered object of interest by an object detection system configured to tolerate more false positives and minimize false negatives for the registered object of interest; forming hypotheses of objects moving in the three-dimensional space using a filter applied to the three-dimensional positions of the registered objects of interest, the filter enabling association between particular objects of interest among the registered objects of interest when an estimated three-dimensional velocity vector for the particular object of interest approximately corresponds to an object moving in the three-dimensional space over time; eliminating a proper subset of hypotheses that are not further expanded by associations made during said forming with at least one additional object of interest registered by said object detection system; specifying at least one three-dimensional track of at least one ball moving in three-dimensional space by applying a complete three-dimensional physical model to the data about the three-dimensional positions of the registered objects of interest used in forming at least one hypothesis remaining after the elimination; outputting for display at least one three-dimensional track of at least one ball moving within said three-dimensional space; A method comprising: forming hypotheses about objects moving in three-dimensional space using said filters; if the number of registered detections for a given hypothesis is less than a threshold, using a first motion model in three-dimensional space, the first motion model having a first complexity level; using a second motion model in three-dimensional space if the number of registered detections for the given hypothesis is equal to or greater than the threshold, the second model having a second complexity level higher than the first complexity level of the first model; A method comprising:
2. 2. The method of claim 1, wherein the first motion model in the three-dimensional space is a linear model that associates the first three-dimensional position of the registered object of interest to a given hypothesis if the first three-dimensional position extends the given hypothesis in approximately the same direction as predicted by the given hypothesis, and the second motion model in the three-dimensional space is a recurrent neural network model trained to perform short-term prediction of motion in the three-dimensional space.
3. Forming a hypothesis about an object moving in three-dimensional space predicting a next three-dimensional position of a registered detection for the given hypothesis using the first model or the second model according to the number of registered detections for the given hypothesis and a threshold value; searching a spatial data structure including all of the 3D positions of objects of interest registered by the object detection system for a next time slice to find a set of 3D positions within a predefined distance of the predicted next 3D position; if the set of 3D locations is an empty set, using a predicted next 3D location for the given hypothesis; if the set of 3D locations includes only one 3D location, using the one 3D location for the given hypothesis; If the set of three-dimensional positions includes two or more three-dimensional positions, sorting the two or more three-dimensional locations based on their proximity to the predicted next three-dimensional location to form a sorted set; removing any less-proximate three-dimensional locations in the sorted set that exceed a predefined threshold number greater than two; and branching the given hypothesis into two or more hypotheses in a group of hypotheses using two or more of the three-dimensional locations remaining in the sorted set; The method of claim 1 , comprising:
4. forming a hypothesis of an object moving in three-dimensional space, if the number of registered detections for the given hypothesis is equal to or greater than the threshold, Identifying a data point in a given hypothesis that has a minimum vertical position; checking the vertical components of each of the estimated three-dimensional velocity vectors before and after the data point in the given hypothesis; designating the data point as a ground hit if a first one of the respective vertical components is negative and a second one of the respective vertical components is positive; The method of claim 1 , comprising:
5. Each said vertical component is obtained from an average of estimated three-dimensional velocity vectors before and after a data point in the given hypothesis, and the method comprises: selecting at least one time window based on a noise level associated with the estimated three-dimensional velocity vector, a minimum expected time of flight on one or both sides of the data point, or both the noise level and the minimum expected time of flight; calculating an average of estimated three-dimensional velocity vectors before and after a data point that falls within said at least one time window; The method of claim 4 , comprising:
6. Forming a hypothesis of an object moving in three-dimensional space includes dividing the given hypothesis into separate segments including a flight segment, one or more bound segments, and a roll segment if the number of registered detections for the given hypothesis is equal to or greater than the threshold, the dividing step being repeated from the identifying, checking, and designating steps. classifying a first segment of a given hypothesis prior to a first specified ground impact as said flight segment; classifying a next segment of a given hypothesis after each next designated ground impact as one of the one or more bounding segments if an angle between a first estimated velocity vector before the next designated ground impact and a second estimated velocity vector after the next designated ground impact is greater than a threshold angle; classifying a next segment of a given hypothesis after the next specified ground impact as the roll segment if an angle between a first estimated velocity vector before the next specified ground impact and a second estimated velocity vector after the next specified ground impact is less than or equal to the threshold angle; The method of claim 4 , comprising:
7. The given hypothesis is at least one hypothesis remaining after the eliminating, and specifying at least one three-dimensional track of at least one ball moving in the three-dimensional space includes: generating data about three-dimensional positions of the registered objects for use in forming the at least one hypothesis by triangulating a more accurate 3D path using sensor observations registered by the object detection system during at least the flight segment; identifying at least one ball in motion by applying the full three-dimensional physics model to the generated data for at least the flight segment; The method of claim 6, comprising:
8. 2. The method of claim 1 , wherein the obtaining includes receiving or generating three-dimensional positions of the registered objects of interest, and a majority of the registered objects of interest are false positives including erroneous detections by individual sensors and erroneous combinations of detections from each of the individual sensors.
9. 9. The method of claim 8, wherein the individual sensors include three or more sensors, and the acquiring includes generating the three-dimensional position by creating a respective combination of detections from each pair of the three or more sensors for a current time slice.
10. The three or more sensors include two cameras and a radar device, and the creating includes: combining the detections of a single object of interest by the two cameras using stereo pairing of the two cameras to generate a first three-dimensional position of the single object of interest; combining detections of the single object of interest by the radar device and / or the two cameras to generate a second three-dimensional position of the single object of interest; 10. The method of claim 9, comprising:
11. 11. The method of claim 10, wherein the three or more sensors include an additional camera, and wherein the creating includes combining detections of the single object of interest by the additional camera and the two cameras using stereo pairing of the additional camera with each of the two cameras to generate a third three-dimensional position of the single object of interest and a fourth three-dimensional position of the single object of interest.
12. 1. A system comprising: two or more sensors and one or more first computers of an object detection system configured to register an object of interest and tolerate more false positives and minimize false negatives for the registered object of interest; One or more second computers configured to perform operations, the operations including: Obtaining a three-dimensional position of an object of interest registered by the object detection system; forming hypotheses of objects moving in the three-dimensional space using a filter applied to the three-dimensional positions of the registered objects of interest, the filter enabling association between particular objects of interest among the registered objects of interest when an estimated three-dimensional velocity vector for the particular object of interest approximately corresponds to an object moving in the three-dimensional space over time; eliminating a proper subset of hypotheses that are not further expanded by associations made during said forming with at least one additional object of interest registered by said object detection system; specifying at least one three-dimensional track of at least one ball moving in three-dimensional space by applying a complete three-dimensional physical model to the data about the three-dimensional positions of the registered objects of interest used in forming at least one hypothesis remaining after the elimination; outputting for display at least one three-dimensional track of at least one ball moving within said three-dimensional space; Including, forming hypotheses about objects moving in three-dimensional space using said filters; if the number of registered detections for a given hypothesis is less than a threshold, using a first motion model in three-dimensional space, the first motion model having a first complexity level; using a second motion model in three-dimensional space if the number of registered detections for the given hypothesis is equal to or greater than the threshold, the second model having a second complexity level higher than the first complexity level of the first model; a second computer including: A system comprising:
13. The system of claim 12 , wherein the two or more sensors comprise a camera and at least one radar device.
14. The system of claim 12 , wherein the two or more sensors include a hybrid camera / radar sensor.
15. The system described in claim 12, wherein the first motion model in three-dimensional space is a linear model that associates the first three-dimensional position of a registered object of interest to a given hypothesis when the first three-dimensional position extends the given hypothesis in approximately the same direction as predicted by the given hypothesis, and the second motion model in three-dimensional space is a recurrent neural network model trained to perform short-term predictions of motion in three-dimensional space.
16. Forming a hypothesis of an object moving in three-dimensional space comprises: predicting a next three-dimensional position of a registered detection for the given hypothesis using the first model or the second model according to the number of registered detections for the given hypothesis and a threshold value; searching a spatial data structure including all of the 3D positions of objects of interest registered by the object detection system for a next time slice to find a set of 3D positions within a predefined distance of the predicted next 3D position; if the set of 3D locations is an empty set, using a predicted next 3D location for the given hypothesis; if the set of 3D locations includes only one 3D location, using the one 3D location for the given hypothesis; If the set of three-dimensional positions includes two or more three-dimensional positions, sorting the two or more three-dimensional locations based on their proximity to the predicted next three-dimensional location to form a sorted set; removing any less-proximate three-dimensional locations in the sorted set that exceed a predefined threshold number greater than two; and branching the given hypothesis into two or more hypotheses in a group of hypotheses using two or more of the three-dimensional locations remaining in the sorted set; The system of claim 12 , comprising:
17. Forming hypotheses of objects moving in three-dimensional space includes, if a number of registered detections for the given hypothesis is greater than or equal to the threshold, Identifying a data point in a given hypothesis that has a minimum vertical position; checking the vertical components of each of the estimated three-dimensional velocity vectors before and after the data point in the given hypothesis; designating the data point as a ground hit if a first one of the respective vertical components is negative and a second one of the respective vertical components is positive; The system of claim 12 , comprising:
18. The respective vertical components are obtained from an average of estimated three-dimensional velocity vectors before and after a data point in the given hypothesis, and the motion comprises: selecting at least one time window based on a noise level associated with the estimated three-dimensional velocity vector, a minimum expected time of flight on one or both sides of the data point, or both the noise level and the minimum expected time of flight; calculating an average of estimated three-dimensional velocity vectors before and after a data point that falls within said at least one time window; The system of claim 17 , comprising:
19. Forming a hypothesis of an object moving in three-dimensional space includes dividing the given hypothesis into separate segments including a flight segment, one or more bound segments, and a roll segment if the number of registered detections for the given hypothesis is equal to or greater than the threshold, the dividing comprising the steps of identifying, checking, specifying, and classifying a first segment of a given hypothesis prior to a first specified ground impact as said flight segment; classifying a next segment of a given hypothesis after each next designated ground impact as one of the one or more bounding segments if an angle between a first estimated velocity vector before the next designated ground impact and a second estimated velocity vector after the next designated ground impact is greater than a threshold angle; classifying a next segment of a given hypothesis after the next specified ground impact as the roll segment if an angle between a first estimated velocity vector before the next specified ground impact and a second estimated velocity vector after the next specified ground impact is less than or equal to the threshold angle; The system of claim 17 , comprising:
20. The given hypothesis is at least one hypothesis that remains after the removing, and specifying at least one three-dimensional track of at least one ball moving in the three-dimensional space includes: generating data about three-dimensional positions of the registered objects for use in forming the at least one hypothesis by triangulating a more accurate 3D path using sensor observations registered by the object detection system during at least the flight segment; identifying at least one ball in motion by applying the full three-dimensional physics model to the generated data for at least the flight segment; 20. The system of claim 19, comprising:
21. The system of claim 12, wherein the obtaining includes receiving or generating three-dimensional positions of the registered objects of interest, and wherein a majority of the registered objects of interest are false positives including false detections by individual ones of the two or more sensors and false combinations of detections from individual ones of the two or more sensors.
22. The system of claim 21, wherein the two or more sensors include three or more sensors, and the acquiring includes generating the three-dimensional position by creating a respective combination of detections from each pair of the three or more sensors for the current time slice.
23. The three or more sensors include two cameras and a radar device, and the creating includes: combining the detections of a single object of interest by the two cameras using stereo pairing of the two cameras to generate a first three-dimensional position of the single object of interest; combining detections of the single object of interest by the radar device and / or the two cameras to generate a second three-dimensional position of the single object of interest; 23. The system of claim 22, comprising:
24. The system of claim 23, wherein the three or more sensors include an additional camera, and wherein the creating includes combining detections of the single object of interest by the additional camera and the two cameras using stereo pairing of the additional camera with each of the two cameras to generate a third three-dimensional position of the single object of interest and a fourth three-dimensional position of the single object of interest.
25. 1. A non-transitory computer readable medium encoding instructions for causing a data processing device associated with an object detection system comprising two or more sensors to perform operations, the operations including: obtaining a three-dimensional position of an object of interest registered by the object detection system, the object detection system being configured to tolerate more false positives and minimize false negatives for the registered object of interest; forming hypotheses of objects moving in the three-dimensional space using a filter applied to the three-dimensional positions of the registered objects of interest, the filter enabling association between particular objects of interest among the registered objects of interest when an estimated three-dimensional velocity vector for the particular object of interest approximately corresponds to an object moving in the three-dimensional space over time; eliminating a proper subset of hypotheses that are not further expanded by associations made during said forming with at least one additional object of interest registered by said object detection system; specifying at least one three-dimensional track of at least one ball moving in three-dimensional space by applying a complete three-dimensional physical model to the data about the three-dimensional positions of the registered objects of interest used in forming at least one hypothesis remaining after the elimination; outputting for display at least one three-dimensional track of at least one ball moving within said three-dimensional space; Including, forming hypotheses about objects moving in three-dimensional space using said filters; if the number of registered detections for a given hypothesis is less than a threshold, using a first motion model in three-dimensional space, the first motion model having a first complexity level; using a second motion model in three-dimensional space if the number of registered detections for the given hypothesis is equal to or greater than the threshold, the second model having a second complexity level higher than the first complexity level of the first model; A non-transitory computer readable medium comprising:
26. The non-transitory computer-readable medium of claim 25, wherein the first motion model in three-dimensional space is a linear model that associates the first three-dimensional position of a registered object of interest to a given hypothesis if the first three-dimensional position extends the given hypothesis in approximately the same direction as predicted by the given hypothesis, and the second motion model in three-dimensional space is a recurrent neural network model trained to perform short-term predictions of motion in three-dimensional space.
27. Forming a hypothesis of an object moving in three-dimensional space includes: predicting a next three-dimensional position of a registered detection for the given hypothesis using the first model or the second model according to the number of registered detections for the given hypothesis and a threshold value; searching a spatial data structure including all of the 3D positions of objects of interest registered by the object detection system for a next time slice to find a set of 3D positions within a predefined distance of the predicted next 3D position; if the set of 3D locations is an empty set, using a predicted next 3D location for the given hypothesis; if the set of 3D locations includes only one 3D location, using the one 3D location for the given hypothesis; If the set of three-dimensional positions includes two or more three-dimensional positions, sorting the two or more three-dimensional locations based on their proximity to the predicted next three-dimensional location to form a sorted set; removing any less-proximate three-dimensional locations in the sorted set that exceed a predefined threshold number greater than two; and branching the given hypothesis into two or more hypotheses in a group of hypotheses using two or more of the three-dimensional locations remaining in the sorted set; 26. The non-transitory computer readable medium of claim 25, comprising:
28. Forming hypotheses of objects moving in three-dimensional space includes, if the number of registered detections for the given hypothesis is greater than or equal to the threshold, Identifying a data point in a given hypothesis that has a minimum vertical position; checking the vertical components of each of the estimated three-dimensional velocity vectors before and after the data point in the given hypothesis; designating the data point as a ground hit if a first one of the respective vertical components is negative and a second one of the respective vertical components is positive; 26. The non-transitory computer readable medium of claim 25, comprising:
29. The respective vertical components are obtained from an average of estimated three-dimensional velocity vectors before and after a data point in the given hypothesis, and the motion comprises: selecting at least one time window based on a noise level associated with the estimated three-dimensional velocity vector, a minimum expected time of flight on one or both sides of the data point, or both the noise level and the minimum expected time of flight; calculating an average of estimated three-dimensional velocity vectors before and after a data point that falls within said at least one time window; 30. The non-transitory computer readable medium of claim 28, comprising:
30. Forming a hypothesis of an object moving in three-dimensional space includes dividing the given hypothesis into separate segments including a flight segment, one or more bound segments, and a roll segment if the number of registered detections for the given hypothesis is equal to or greater than the threshold, the dividing comprising the steps of identifying, checking, specifying, and classifying a first segment of a given hypothesis prior to a first specified ground impact as said flight segment; classifying a next segment of a given hypothesis after each next designated ground impact as one of the one or more bounding segments if an angle between a first estimated velocity vector before the next designated ground impact and a second estimated velocity vector after the next designated ground impact is greater than a threshold angle; classifying a next segment of a given hypothesis after the next specified ground impact as the roll segment if an angle between a first estimated velocity vector before the next specified ground impact and a second estimated velocity vector after the next specified ground impact is less than or equal to the threshold angle; 30. The non-transitory computer readable medium of claim 28, comprising:
