A system for optical infrared tracking to detect and determine the real-world position of objects

Abstract

A system and method for tracking physical objects using optical infrared imaging. The system includes an array of infrared cameras and an aggregation subsystem that computes real-world object positions without the need for external markers or embedded electronics during runtime. Using automated calibration and infrared reflection detection, the system maintains object identity across multiple views and provides real-time location data to consumer applications such as game engines.

Description

A System for Optical Infrared Tracking to Detect and Determine the Real -World Position of ObjectsCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This Application claims the benefit under 35 U.S.C. § 119(e) of U.S. Provisional Patent Application Ser. No. 63 / 669,949, filed on 07 / 11 / 2024, the contents of which are incorporated herein by reference in their entirety.FIELD OF THE INVENTION

[0002] The present invention relates generally to optical tracking systems and, more particularly, to systems and methods for using infrared imaging to detect and determine the real-world position of physical objects.BACKGROUND

[0003] Various methods for tracking physical objects in space are known, including the use of active RFID tags, ultra-wideband (UWB) systems, LiDAR, depth-sensing cameras, and high-speed video systems. These techniques are commonly used in real-time object tracking within defined environments, such as sports facilities or manufacturing spaces.

[0004] One notable example is the tracking of golfballs, especially in driving ranges. Existing systems, such as those used by Top Golf, provide ball tracking capabilities over long distances but fail at tracking balls over putting greens. Another example is one used by Puttshack to track balls for minigolf games. However, these systems typically rely on marked objects, added tags, specialized equipment, or controlled environments, and are not well-suited for continuous tracking of standard, unmarked golfballs over large, arbitrarily shaped putting greens.

[0005] Tracking technologies for other types of objects also exist, but issues such as equipment cost, setup complexity, scalability, and system limitations often hinder practical implementation.

[0006] Another limitation of conventional systems is the difficulty in maintaining object coherence, or persistent identification of objects as they move across multiple sensor fields. Once an object leaves the field of view, or when collisions occur, identity tracking is often lost.

[0007] Furthermore, many tracking systems do not support real-time output to consumer-level software. They often require post-processing or proprietary tools to visualize results or integrate with interactive applications, such as game engines.

[0008] Accordingly, there exists a need for an optical infrared tracking system that can determine real -world object positions in real time, maintain object identity across multiple cameras,and interface with consumer systems. The system should further include automated calibration capabilities to ensure spatial accuracy and minimize manual configuration, thereby overcoming the limitations of existing technologies.BRIEF DESCRIPTION OF THE DRAWINGS

[0009] These and other features, aspects, and advantages of the invention will be better understood from the following detailed description, appended claims, and accompanying drawings, in which:

[0010] Figure 1 is a diagram of a system for optical infrared tracking, according to one embodiment of the present invention.

[0011] Figure 2 is a diagram showing an example of a real -world calibration grid.

[0012] Figure 3 is a diagram showing an example of a real-world calibration grid as represented in image-space, including typical tangential distortion introduced by physical lensing.

[0013] Figure 4 shows a flowchart of the steps used to generate the lookup tables between camera and real-world coordinates.

[0014] Figure 5 illustrates the runtime use of the lookup tables to determine the real-world coordinates of tracked objects.SUMMARY

[0015] The present disclosure describes a system for determining the real -world position of unmarked physical objects using near-infrared optical tracking. The system comprises a plurality of near-infrared cameras and an infrared illumination source for viewing and illuminating a tracking region. A calibration subsystem maps camera pixel coordinates to real-world coordinates by detecting known calibration markers and generating interpolation models or lookup tables. One or more vision processing units identify objects based on infrared reflectivity and calculate their pixel position. A position aggregation subsystem receives real-world position data from various vision processing units, determines when data from multiple sources corresponds to the same object, and produces a unified, real-time representation of the position and identity of each tracked object. The position aggregation subsystem can accept data from a variable number of concurrently operating cameras, assign persistent unique identifiers to tracked objects, and provide object position and identity data to external systems in real time. Each vision processing unit may classify objects based on geometric features before determining their position. The calibration subsystem can detect infrared-reflective markers arranged in a grid, associate pixel coordinates with known real-world coordinates, and construct a continuousmapping between pixel space and physical space using interpolation techniques, potentially generating two-dimensional lookup tables for coordinate conversion.

[0016] The present invention addresses the limitations of prior art by providing a system for optical infrared tracking that determines the real -world position of physical objects in real time.DETAILED DESCRIPTION OF THE INVENTION

[0017] The present invention provides a system for optical infrared tracking that determines the real-world position of physical objects without requiring external markers or embedded electronics during runtime.

[0018] In one embodiment, the system utilizes an array of two-dimensional infrared cameras to track the location of standard golfballs on a putting green within centimeter precision. The system outputs real -world coordinates of the tracked objects, which are then used by a consumer application such as a game engine to projection-map content onto the physical environment in real time.

[0019] For all embodiments, the camera array may be configured in various sizes and layouts. Object data collected by the cameras is processed by an “aggregator” module that assembles a unified view of all detected objects. This module uses automatically generated calibration data and rectilinear grid interpolation techniques to transform camera-space coordinates into real-world coordinates.

[0020] These coordinates are then fed into a consumer system, such as a game engine, for realtime display or interaction. The resulting system enables dynamic and accurate visual alignment between virtual content and physical objects.

[0021] All dimensions and proportions provided in the figures are for illustrative purposes only and and do not represent the domain of configurations of the invention. Actual system configurations will vary depending on application-specific requirements.

[0022] In the descriptions that follow, reference is made to certain drawings, which serve to illustrate embodiments of the invention but do not represent the full domain of possible configurations. The term “one embodiment” indicates that a particular feature or functionality is included in at least one embodiment. Multiple instances of the term do not necessarily refer to the same embodiment.

[0023] Reference numerals are reused across figures to indicate corresponding components. The leading digit of each numeral corresponds to the figure in which the element is first introduced.

[0024] Unless explicitly stated otherwise, the terms “comprise,” “comprising,” and their variants are used in a non-limiting sense, meaning the presence of stated features or components does not exclude the presence of others.

[0025] Some details may be omitted or shown in simplified form to avoid obscuring the invention. Block diagrams, flowcharts, and other schematic representations are used to illustrate system architecture, logical structure, and operation.

[0026] A storage medium may include any device capable of storing data, such as RAM, ROM, magnetic or optical storage, flash memory, and other non-transitory media.

[0027] The invention may be implemented in hardware, software, firmware, or any combination thereof. When implemented in software or firmware, program code may be stored in a machine- readable medium and executed by one or more processors operating in sequence, parallel, or distributed fashion.

[0028] Terminology used in this disclosure refers to illustrative examples and should not be interpreted as limiting unless stated otherwise.CALIBRATION PROCESS AND COORDINATE MAPPING

[0029] Referring now to Figure 1, there is shown a diagram of a system 100 for optical infrared tracking, according to one embodiment of the present invention.

[0030] The system comprises a plurality of near-infrared cameras 102, 104 and 106 are positioned to view a tracking region, each configured to capture image data in the near-infrared spectrum. An infrared illumination source is configured to illuminate the tracking region. A calibration subsystem 114 is configured to map each of the plurality of near-infrared cameras’ 102, 104 and 106 pixel coordinates to real-world coordinates 200 by executing instructions on one or more than one processor for detecting known calibration markers 202 at predetermined physical locations and generating interpolation models 300 or lookup tables from the detected calibration markers 202 corresponding to the real-world coordinates 200. One or more than one vision processing units 108, 110, 112 is configured to identify objects based on infrared reflectivity and to calculate each object's pixel position.

[0031] The position aggregation subsystem 114 is configured to receive real -world coordinates 200 from one or more than one vision processing units 108, 110, 112. The one or more than one vision processing units 108, 110, 112 comprise instructions executable on the processing units 108, 110, 112 to determine when calibration markers 202 from multiple real-world coordinates 200 corresponds to the same object. Finally, the position aggregation subsystem 114 produces a unified, real-time representation of the position and identity of each tracked object across the tracking region.

[0032]

[0033] Referring now to Figure 3, there is shown a diagram of a real-world calibration grid 200 as represented in an image-space 300, including typical tangential distortion introduced by physical lensing.

[0034] Referring now to Figure 4, there is shown a flowchart 400 of some steps of a method used to generate the lookup tables between camera and real-world coordinates. To accurately determine the real -world coordinates 200 of an object based on one or more than one pixel 302, 304, and 308 coordinates from images from the plurality of near-infrared cameras 102, 104 and 106, the system uses a calibration process. This involves generating a model that maps image-space 300 comprising one or more than one pixel 302, 304, and 308 coordinates to real -world coordinates 200 taken from the real- world calibration grid 200.

[0035] First, the process begins by placing 402 calibration markers 202 at known real-world coordinates 200, in physical locations, across a target area, such as, for example, a putting green. The calibration markers 202 can be physical or digital, and can be manually placed or automatically placed by the system 100. Each calibration markers 202 corresponds to a known (X, Y) location real-world coordinates 200. The calibration markers 202 are placed at regular intervals — e.g., one marker per foot across a 10-foot by 20-foot region. While shown as a regular grid, the system 100 supports irregular configurations. Next, due to lens distortion, angle of view, and other physical factors, inherent in the plurality of near-infrared cameras 102, 104 and 106, each of the camera images detects the grid defined by the calibration markers 202 as warped or non-linear an image-space 300. Then, the calibration process captures 404 the one or more than one pixel 302, 304, and 308 coordinates of each visible calibration markers 202 from each of the plurality of near-infrared cameras’ 102, 104 and 106 perspective using computer vision techniques 406 to determine the one or more than one pixel 302, 304, and 308 pixel position . Next, a center point of each of the one or more than one pixel 302, 304, and 308 pixel for each of the calibration markers 202 is determined 408. Then, a sanity check 408 is applied to each marker to ensure it appears within an expected pixel range. Invalid calibration markers 202 (e.g., from debris or unintended reflections) are excluded. The calibration procedure is automated. Next, each of the one or more than one pixel 302, 304, and 308 in the plurality of near-infrared cameras 102, 104 and 106 is matched to real -world coordinates 200 in the physical space. Then, using one or more than one pixel 302, 304, and 308 and the real-world coordinates 200 pairs, the system 100 generates two lookup tables 412 or datasets per camera: one that maps one or more than one pixel 302, 304, and 308 position to real -world coordinates 200 in the X plane, and another to real -world coordinates 200 in the Y plane. Each lookup tables 412 entry contains a tuple: (pixel_x, pixel_y, real_x) or (pixel x, pixel_y, real_y). Finally, a weighted extrapolation 414 is applied to estimate real-worldcoordinates 200 at the one or more than one pixel 302, 304, and 308 locations not directly corresponding to a calibration markers 202. The end result is a mapping function that enables the transformation of any object’s pixel position into its corresponding real-world coordinates 200.

[0036] Referring now to Figure 5 there is shown a flowchart of some steps of a method for a runtime use of the lookup tables 412 to determine the real-world coordinates 200 of tracked objects. The method comprises: First capturing new frame from camera 502. Then, masking out areas where tracking is not desired 504. Next, utilizing computer vision techniques to determine the XY coordinates 302, 304, and 308 of each object of interest 506. Then, determining the XY pixel center point of each object of interest 508. Next, using center point to find real-world coordinates 200 in the X plane utilizing look-up table 510. Then, using center point to find the real -world coordinates 200 in the Y plane utilizing look-up table 512. Finally, passing XY real-world coordinates 200 to a ball aggregator 514.POSITION AGGREGATION SUBSYSTEM

[0037] The system includes a position aggregation subsystem that receives real-time object location data from one or more one or more than one vision processing units 108, 110, 112. It fuses the incoming data to generate a coherent, global representation of object positions within the monitored space.

[0038] The subsystem identifies when two or more data sources are referencing the same object and assigns a unique, persistent identifier to each object. This identity is preserved even as the object moves across different camera fields of view, or during collisions with other objects.

[0039] When a new object is detected, its movement patterns are tracked and analyzed to distinguish it from existing objects and non-tracked objects such as debris. If no match within the existing objects is found after analysis the object is considered new, or discarded if it was determined to be debris.

[0040] In the case of collisions and other interactions with tracked objects, a predictive model is applied to maintain object identity. Factors considered include but are not limited to object velocity, direction, angle of impact, and physical characteristics of the impacted object. The subsystem uses this information to preserve object identity across object-object or object-environment interactions.

[0041] The aggregated position data is made available to consumer systems via a requestresponse protocol, such as UDP. On receiving a request from a consumer system, the aggregator returns a packet containing all currently tracked objects, each with an identifier and real -world coordinates. This interface is designed to be agnostic to client platforms or use cases.

[0042] Although the present invention has been described with a degree of particularity, it is understood that the present disclosure has been made by way of example and that other versions are possible. As various changes could be made in the above description without departing from the scope of the invention, it is intended that all matter contained in the above description or shown in the accompanying drawings shall be illustrative and not used in a limiting sense. The spirit and scope of the appended claims should not be limited to the description of the preferred versions contained in this disclosure.

[0043] All features disclosed in the specification, including the claims, abstracts, and drawings, and all the steps in any method or process disclosed, may be combined in any combination, except combinations where at least some of such features and / or steps are mutually exclusive. Each feature disclosed in the specification, including the claims, abstract, and drawings, can be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise. Thus, unless expressly stated otherwise, each feature disclosed is one example only of a generic series of equivalent or similar features.

[0044] Any element in a claim that does not explicitly state "means" for performing a specified function or "step" for performing a specified function should not be interpreted as a "means" or "step" clause as specified in 35 U.S.C. § 112.

Claims

CLAIMS1. A system for determining the real-world position of unmarked physical objects using nearinfrared optical tracking, comprising: a. a plurality of near-infrared cameras positioned to view a tracking region, each configured to capture image data in the near-infrared spectrum; b. an infrared illumination source configured to illuminate the tracking region; c. a calibration subsystem configured to map camera pixel coordinates to real-world coordinates by: i. detecting known calibration markers at predetermined physical locations; and ii. generating interpolation models or lookup tables from the detected marker correspondences; d. one or more vision processing units configured to identify objects based on infrared reflectivity and to calculate each object's pixel position; e. a position aggregation subsystem configured to: i. receive real-world position data from an arbitrary number of vision processing units; ii. determine when data from multiple sources corresponds to the same object; and iii. produce a unified, real-time representation of the position and identity of each tracked object across the tracking region.2 The system of claim 1, wherein the position aggregation subsystem is configured to accept data from a variable number of cameras operating concurrently, each producing independent object tracking data within overlapping or adjacent fields of view.3 The system of claim 1, wherein the position aggregation subsystem assigns persistent unique identifiers to tracked objects and maintains continuity of identity across multiple views and object interactions, including collisions.4 The system of claim 1, wherein the position aggregation subsystem is further configured to provide object position and identity data to one or more external systems via a request-response protocol in real time.

5. The system of claim 1, wherein each vision processing unit classifies objects based on geometric features including contour area, perimeter, and shape prior to determining object position.

6. The system of claim 1, wherein the calibration subsystem is further configured to: a. detect infrared-reflective markers arranged in a grid on the tracking surface; b. associate pixel coordinates of each marker with corresponding known real-world coordinates; and c construct a continuous mapping between pixel space and physical space using interpolation techniques.7 The system of claim 6, wherein the calibration grid comprises reflective physical markers, projected infrared patterns, or projected visible patterns with camera sensitivity toggling between visible and infrared modes.8 The system of claim 6, wherein the calibration subsystem generates two-dimensional lookup tables for converting pixel coordinates into real-world X and Y coordinates through weighted extrapolation.

Images