Error compensation for 3D tracking systems

Abstract

A tracking system for tracking one or more reflective markers includes at least two optical sensors configured to obtain image data of an environment including at least one marker. The tracking system obtains the image data from the at least two optical sensors. The tracking system is configured to: extract an optical signature representing a reflection of an optical signal from the at least one marker from the image data, determine an optical centroid of the optical signature of the at least one marker, estimate an initial pose for the at least one marker, determine an offset error vector from the optical centroid of the at least one marker based on the initial pose, determine a corrected optical centroid based on the offset error vector and the optical centroid, and determine a corrected three-dimensional position of the marker in the environment based on the corrected optical centroid of the marker.

Description

[0001] Claiming priority

[0002] This application claims priority under 35 USC §119(e) to U.S. patent application No. 63 / 115,154, filed on November 18, 2020, and the entire contents of the above application are hereby incorporated by reference. Technical Field

[0003] The present invention relates to error compensation for optical tracking systems. Background Art

[0004] Tracking systems typically rely on an object to which one or more markers are attached. The markers attached to the object may be active markers (eg, light emitting diode markers), passive markers, or a combination of active and passive markers.

[0005] Typically, a passive marker can be configured to reflect an optical signal toward a camera. The marker can be configured to reflect the optical signal back to the signal source on a parallel path. In response to detecting the optical signal reflected from the marker by the camera, the tracking system can estimate the position of the marker in the environment.

[0006] In the context of medical applications, a user (e.g., a doctor) touches a surface of interest (e.g., a surface of a patient's body) with the distal end of a tracked object (e.g., a probe or surgical instrument). An object sensing device observes marker(s) attached to the object. Based on the known position of the sensing device and the determined positions and orientations (e.g., poses) of the marker(s) as seen by the sensing device, such a system calculates the three-dimensional coordinates of the object to which the marker(s) are attached. Summary of the invention

[0007] This specification describes a tracking system that is configured to determine the position of a tracked object in an environment by estimating the posture (e.g., position and orientation) of multiple markers attached to the tracked object. At least one marker provides a signal to a sensor of the tracking system. The signal indicates the position and orientation (e.g., posture) of the marker in the environment, from which the posture of the tracked object can be estimated. The tracking system can be an optical tracking system, and the marker can be a passive marker configured to reflect an optical signal to the tracked object. In some implementations, the markers can each include a retroreflective coating configured to reflect the optical signal toward the source of the optical signal. Typically, an optical sensor (e.g., a camera) is located near the source of the optical signal and is configured to detect a reflected optical signal from the marker. Reflection (e.g., flicker) is detected on each marker in the marker. The tracking system is configured to estimate the position of the marker in the environment based on the position where the reflected signal is detected. The posture of the tracked object is then determined based on a predetermined relationship between the posture of each marker in the marker and the posture of the tracked object.

[0008] Markers (e.g., passive markers) typically include geometric shapes and can be covered in a reflective coating. In some implementations, the marker can be a sphere. In some implementations, the marker can be a cylinder. Tracking systems typically include a disk or sphere among the markers for the tracked object because for these shapes, when the marker is viewed from any angle, the optical center of mass of the object in two-dimensional image space (e.g., recorded in the sensor image plane) is in a straight line with the volumetric center of mass of the object in three-dimensional (3D) space. The volumetric center of mass of the marker typically represents the position of the marker in 3D space. Alignment occurs assuming that the marker has perfect or near-perfect retroreflective symmetry of light return. Alignment of the volumetric center of mass and the optical center of mass can enable simple triangulation of the position and orientation of the marker.

[0009] The retroreflective sphere marker has a wide (almost complete) viewing angle range for which it can be used in situations where the optical center of mass of the sphere is not the same as the volumetric center of mass of the sphere, as previously described. However, it can be difficult to create a low level of optical variability on the surface of a passive marker using a reflective coating. The variability can be the result of stretching the retroreflective material on the spherical surface of the passive marker. This variability reduces the overall accuracy of tracking the sphere.

[0010] Retroreflective disk targets are highly uniform in optical performance because it is easier to uniformly coat the disk with reflective material. The low variability in the reflective surface can result in higher accuracy for tracking disk-shaped markers relative to spherical markers. However, because disks have a shallow viewing angle, they offer lower usability than spheres. Beyond a threshold viewing angle, the disk is generally not visible.

[0011] Thus, the use of an alternative geometry for the marker (e.g., a cylindrical marker) can provide a wide viewing angle and also allow the reflective material to be applied uniformly to the marker. However, for a range of viewing angles, the optical center of mass of the marker is not aligned with its volumetric center of mass. Therefore, the tracking system performs compensation to reduce or eliminate errors caused by the misalignment of the optical center of mass in the image plane with the volumetric center of mass representing the position of the marker in 3D space.

[0012] The tracking system is configured to compensate for alternative geometric markers (e.g., cylindrical markers or other ruled surfaces) on the tracked device in order to accurately determine the pose of the tracked object. The tracking system is configured to use an iterative compensation process. Typically, the tracking system is calibrated based on specific parameters of the markers of the tracked object, including their size, shape, orientation, relative position on the tracked object, and their reflective properties. When object tracking begins, the tracking system does not need to know the initial pose of the tracked object. Instead, the initial pose estimate is based on the image of the marker, and the offset error vector model is applied to correct the initial pose estimate. The applied offset is an estimate of the misalignment of the volume centroid and the optical centroid according to the angle of each optical sensor in a plurality of optical sensors, and is based on the pose estimate for each marker. The correction model is typically continuous and takes into account any pose in the environment of the marker. The correction estimate is determined and applied to the initial optical centroid, and the initial optical centroid adjustment supports the calculation of the corrected pose. The tracking system is configured to iterate the estimation process by using the updated pose estimate as the initial pose estimate for the corrected pose. The initial pose estimate becomes increasingly accurate in its representation of the true pose of the marker in the environment.

[0013] Implementations described herein may provide various technical benefits. For example, the techniques described herein enable tracking systems to use cylindrical markers, which provide more accurate tracking due to lower surface variability, as previously described with respect to spheres, while maintaining a wide viewing angle. This may enable a lower cost manufacturing process for the markers and an overall more accurate tracking system.

[0014] In addition, the tracking system described herein is configured to estimate the pose of the tracked object without requiring a known initial pose of the object or an initialization process performed prior to tracking. For example, the tracking system may include a memory that is preloaded with parameters of the tracked object, such as the size, shape, relative position, etc. of markers on the tracked object. The tracking system may immediately start tracking by performing an initial estimate of the offset value for each marker based on the stored parameters.

[0015] The above advantages may be achieved through one or more of the following embodiments.

[0016] In a general aspect, a tracking system for tracking one or more reflective markers. The tracking system includes: at least two optical sensors configured to obtain image data of an environment including at least one marker, wherein at least a portion of the at least one marker is configured to reflect an optical signal; and a processing device configured to obtain the image data from the at least two optical sensors. The processing device is configured to perform operations, the operations including: extracting an optical signature representing a reflection of an optical signal of the at least one marker from the image data; determining an optical centroid of the optical signature in the image data; estimating an initial pose of the at least one marker in the environment based on the optical centroid; determining an offset error vector from the optical centroid of the at least one marker based on the estimate of the initial pose; determining a corrected optical centroid of the at least one marker based on the offset error vector and the optical centroid; and determining a corrected three-dimensional (3D) position of the at least one marker in the environment based on the corrected optical centroid of the at least one marker.

[0017] In some implementations, estimating an initial pose of at least one marker in an environment based on an optical centroid includes obtaining one or more optical sensor configuration parameters, the one or more optical sensor configuration parameters including one or more of a lens size, a focal length, and an optical sensor position in the environment for the at least two optical sensors; and estimating offset error vectors of multiple markers in the environment based on the one or more optical sensor configuration parameters.

[0018] In some embodiments, the at least one marker is a first marker, and wherein estimating an initial pose of the at least one marker in the environment includes receiving spatial data representing a predetermined spatial relationship between the first marker and one or more additional markers of a plurality of markers, the first marker and the one or more additional markers being rigidly fixed relative to each other on the tracked object; extracting an optical signature representing a reflection of an optical signal from the one or more additional markers from the image data; and estimating the initial pose of the first marker in the environment based on the optical signatures of the one or more additional markers and the predetermined spatial relationship between the first marker and the one or more additional markers.

[0019] In some implementations, the tracking system includes an inertial measurement unit (IMU) configured to provide orientation data representing an orientation for at least one marker. In some implementations, estimating an initial pose of the at least one marker is based on the orientation data provided by the IMU.

[0020] In some implementations, determining the offset error vector is also based on one or more parameters of at least one mark, where, for a given mark, the one or more parameters include a size of the given mark, a shape of the given mark, (if there are multiple marks) a relative position of the given mark relative to at least one other mark, and a configuration of a reflective coating on the given mark.

[0021] In some implementations, the operations include using the corrected three-dimensional position as part of an initial pose of at least one marker in the environment; and determining a bias error vector based on the corrected three-dimensional position of the at least one marker through an iterative process.

[0022] In some implementations, the operations include determining a pose of a portion of the tracked object in the environment based on the corrected three-dimensional position of the at least one marker. In some implementations, the at least one marker is fixed to the tracked object at a predetermined position on the tracked object.

[0023] In some implementations, the tracked object is a surgical tool. In some implementations, the portion of the tracked object is a working portion of the tool.

[0024] In some embodiments, at least one mark includes one of a cylinder with only reflective sides, a cylinder with reflective sides and a reflective top, a convex conical mark, a concave conical mark, a concave hemispherical mark, a convex hemispherical mark, a disk mark, any mark with a ruled surface, and a spherical mark.

[0025] In some implementations, at least one mark is one of a plurality of marks that are identical or nearly identical in size or shape to one another.

[0026] In some implementations, at least one marker is one of a plurality of markers that are each unique in size or shape relative to at least one other marker of the plurality of markers.

[0027] In some implementations, the optical signal includes infrared (IR) or near infrared radiation.

[0028] In a general aspect, a process performed by a computing system for tracking one or more reflective markers includes: obtaining image data of an environment including at least one marker from at least two optical sensors, wherein at least a portion of the at least one marker is configured to reflect an optical signal; extracting an optical signature representing a reflection of an optical signal of the at least one marker from the image data; determining an optical center of mass of the optical signature in the image data; estimating an initial pose of the at least one marker in the environment based on the optical center of mass; determining an offset error vector from the optical center of mass of the at least one marker based on the estimate of the initial pose; determining a corrected optical center of mass of the at least one marker based on the offset error vector and the optical center of mass; and determining a corrected three-dimensional (3D) position of the at least one marker in the environment based on the corrected optical center of mass of the at least one marker.

[0029] In some implementations, estimating an initial pose of at least one marker in an environment based on an optical centroid includes obtaining one or more optical sensor configuration parameters, the one or more optical sensor configuration parameters including one or more of lens size, focal length, and optical sensor position in the environment for at least two optical sensors; and estimating offset error vectors of multiple markers in the environment based on the one or more optical sensor configuration parameters.

[0030] In some embodiments, the at least one marker is a first marker, and wherein estimating an initial pose of the at least one marker in the environment includes receiving spatial data representing a predetermined spatial relationship between the first marker and one or more additional markers of a plurality of markers, the first marker and the one or more additional markers being rigidly fixed to each other on the tracked object; and extracting an optical signature representing a reflection of an optical signal from the one or more additional markers from the image data; estimating the initial pose of the first marker in the environment based on the optical signatures of the one or more additional markers and the predetermined spatial relationship between the first marker and the one or more additional markers.

[0031] In some implementations, the process includes providing, by an inertial measurement unit (IMU), orientation data representing an orientation for at least one marker, wherein estimating an initial pose of the at least one marker is based on the orientation data provided by the IMU.

[0032] In some implementations, determining the offset error vector is also based on one or more parameters of at least one mark, where, for a given mark, the one or more parameters include a size of the given mark, a shape of the given mark, (if there are multiple marks) a relative position of the given mark relative to at least one other mark, and a configuration of a reflective coating on the given mark.

[0033] In some implementations, the process includes using the corrected three-dimensional position as part of an initial pose of at least one marker in the environment; and determining a bias error vector based on the corrected three-dimensional position of the at least one marker through an iterative process.

[0034] In some implementations, the process includes determining a pose of a portion of a tracked object in an environment based on a corrected three-dimensional position of at least one marker, wherein the at least one marker is fixed to the tracked object at a predetermined position on the tracked object.

[0035] In some implementations, one or more non-transitory computer-readable media store instructions that, when executed by a computing system, cause the computing system to perform operations, the operations comprising: obtaining image data of an environment including at least one marker through at least two optical sensors, wherein at least a portion of the at least one marker is configured to reflect an optical signal; extracting an optical signature representing a reflection of an optical signal of the at least one marker from the image data; determining an optical center of mass of the optical signature in the image data; estimating an initial pose of the at least one marker in the environment based on the optical center of mass; determining an offset error vector from the optical center of mass of the at least one marker based on the estimate of the initial pose; determining a corrected optical center of mass of the at least one marker based on the offset error vector and the optical center of mass; and determining a corrected three-dimensional (3D) position of the at least one marker in the environment based on the corrected optical center of mass of the at least one marker.

[0036] The details of one or more embodiments are set forth in the accompanying drawings and the description below. Other features and advantages will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 is a block diagram of an example tracking system for locating markers.

[0038] Figure 2 is a perspective diagram of an example tracking system for determining the pose of a tracked object having three markers.

[0039] Figure 3A-3C Each includes an example of a retroreflective marker.

[0040] Figure 4 An example process for performing three-dimensional tracking is shown.

[0041] Figure 5 is a schematic diagram of an example computing system.

[0042] Throughout the various drawings, like reference numbers and designations indicate like elements. DETAILED DESCRIPTION

[0043] This specification describes a tracking system configured to determine the posture (e.g., position and orientation) of a tracked object in an environment. The tracked object typically includes one or more markers configured to provide a signal to the tracking system, and the tracking system includes a sensor configured to measure the signal from the marker. The signal indicates the posture of the tracked object in the environment based on the posture of each marker in the marker. For example, the tracking system can be an optical tracking system, and the marker can be a passive marker configured to reflect the optical signal to the tracked object. The passive markers can each include a retroreflective coating that is configured to reflect the optical signal back to the source of the optical signal along a parallel path. Typically, an optical sensor (e.g., a camera) is located near the source of the optical signal and is configured to detect a reflected optical signal from each marker in the marker. Reflection (e.g., flicker) is detected for each marker. The tracking system is configured to estimate the position of the passive marker in the environment based on the position where the reflected signal is detected.

[0044] The markings typically include a geometric shape and can be covered in a reflective coating. The markings can be, for example, cylindrical. However, other shapes are possible. In some implementations, the markings typically include a shape with a 3D surface around which a flat film can be wrapped without two-dimensional (2D) distortion, stretching, wrinkling, or overlapping. This is to facilitate application of a uniform reflective coating on the marking with as perfect a retroreflective light return as possible.

[0045] The marker reflects a signal (e.g., an infrared signal) to an optical sensor to provide an optical signature in the image data captured by the optical sensor. The marker is associated with an optical centroid, which is the center of the marker's reflected signal in a two-dimensional (2D) plane of the image. The marker is associated with a volume centroid, which represents the position of the marker in a 3D space (e.g., in a tracking environment). For markers other than spheres or disks, at some viewing angles, the optical centroid and the volume centroid are misaligned due to the geometric properties of the marker. In these cases, compensation is performed to correct the optical centroid to a corrected position in the 2D image space so that the precise 3D position of the marker can be estimated from the image data. The corrected optical centroid of each marker represents an estimate of the optical centroid aligned with the volume centroid of the marker. Together, the corrected optical centroids of the markers enable triangulation of the pose of the tracked object including the marker.

[0046] Figure 1A schematic diagram of an example tracking system 100 is shown that includes optical sensors 102a-102b and at least one marker 104 that is associated with (e.g., connected to) a tracked object (e.g., a medical device such as a surgical scalpel). Typically, the tracked object includes a tracked tool to which a marker is attached. As previously stated, this can include a surgical tool. Typically, the passive marker 104 includes a geometric shape having a retroreflective material applied to a portion of a surface. This can include spheres, disks, cylinders, etc., and is further described below. Typically, the optical signature includes a retroreflected signal from the marker 104 on a sensor image plane. Typically, the optical centroid 118a-118b represents the center of mass of the optical signature. The optical centroid 118a-118b is represented by a 2D value recorded on the image plane of the image sensor 102a-102b. The offset error vector 112a-112b includes an adjustment to be made to the 2D position of the optical centroid in the image plane. The offset error vectors 112a, 112b thus comprise 2D vectors on the image plane of the sensors 102a-102b. The corrected optical centroids 120a-120b comprise optical centroids 118a-118b values ​​adjusted by the offset error vectors 112a-112b. The corrected optical centroids 120a-120b are represented by 2D coordinate values ​​in the sensor image plane. The volume centroid 108 is the volume center of the marker 104 in 3D space. This value is represented by the corrected 3D position of the marker 104 based on the corrected optical centroid 120a-120b values. The volume centroid 108 may be calculated by projecting the line of sight from the corrected optical centroids determined for at least two optical sensors 102a-120b. The reflection center 106 of the marker represents an example of the 3D position of the marker as measured by the uncorrected optical centroids 118a-118b.

[0047] The tracking system 100 is configured to determine the pose of the marker 104 based on an optical signal (not shown) emitted from a location near the optical sensors 102a-102b. The marker 104 is configured to reflect the optical signal (eg, infrared light) back to the optical sensors 102a-102b.

[0048] The optical sensors 102a-102b of the system may be implemented in various forms, such as video cameras (e.g., stereo cameras) to generate images of the measurement volume. These images are analyzed by a controller associated with the optical sensors 102a-102b to identify the location of the marker 104 in the images, for which image coordinates (e.g., {U, V}, {row, column}, etc.) are calculated to sub-pixel resolution. These image coordinates (e.g., {U, V} coordinates) from two or more optical sensors 102a-102b are used to calculate the location of the marker 104 in a coordinate system (e.g., in Figure 1 102a-102b).

[0049] The optical sensors 102a-102b of the system can be implemented in various implementations, for example, the optical sensors 102a-102b can be part of a single device (e.g., a stereo camera), a group of devices (e.g., a camera array), etc. In some implementations, the optical sensors 102a-102b are part of an image capture unit that includes one or more marker lighting devices (e.g., electromagnetic wave emission sources). For example, a lighting device can be paired with each optical sensor. For example, in a stereo camera, two lighting devices are included. In this example, the lighting device emits electromagnetic waves (e.g., optical signals such as infrared light or near infrared light, etc.).

[0050] The optical sensors 102a-102b are typically measured by a known baseline value (at Figure 1 The optical sensors 102a-102b each receive an optical signal reflected from the marker 104. In addition to other parameters of the optical sensors 102a-102b, the computing system uses the known baseline value D of at least two optical sensors 102a-102b to calculate the offset values ​​112a-102b.

[0051] Typically, the image planes of the sensors 102a-102b are at some angle to each other. Each sensor 102a-102b typically includes a CCD / CMOS sensor with a lens or a lens stack. Each optical sensor 102a-102b includes a separate lens stack, and the two sensors are separated by a baseline D (e.g., strictly in one unit) to enable estimation of depth / distance based on triangulation between the two sensors. Therefore, in some implementations, there are two sensors rigidly mounted and displaced from each other in one unit. However, this can be extended to multiple cameras cooperating in a shared volume.

[0052] The optical signal is directed to the area including the mark 104. Figure 2 As described and mentioned above, the marker 104 can be attached to an object, such as a tool (e.g., a surgical tool, a medical device for treating a patient, etc.). The marker 104 is configured (e.g., covered in a retroreflective material, has a retroreflective surface feature, etc.) so that an incident electromagnetic wave is reflected in a retroreflective manner (e.g., the reflected wave is directed back to the source with minimal scattering in other directions). The optical sensors 102a-102b capture images of the illuminated markers 104. Due to the highly retroreflective nature of the markers 104, the markers appear as bright spots in the captured images (e.g., compared to other objects within the captured scene), and the tracking system 100 can extract the reflected optical signature of the markers in the image data. The positions of the optical centroids 118a-118b can be determined from the image data, corrected, and 3D position coordinates (e.g., Cartesian coordinates, spherical coordinates, cylindrical coordinates, etc.) can be estimated from the corrected optical centroids 120a-120b for each marker.

[0053] The marker 104 can generally include a cylindrical geometry. Additionally, the marker includes a reflective surface. In some implementations, the cylinder is coated with a reflective material. Due to the shape of the cylinder, it is relatively simple to wrap the reflective material around the circumference of the cylinder and cover any available end circular surface of the cylinder with the reflective material without stretching the material. This results in a surface with low reflection variability and / or high uniformity. The uniformity of the reflective surface of the cylinder results in higher accuracy in optically tracking the cylinder, as previously described. For example, optical signal reflections are not distorted by variability in the reflective surface.

[0054] Other shapes are possible for the marker 104. For example, the shapes may include a cylinder with only reflective sides, a cylinder with reflective sides and a reflective top, a convex conical marker, a concave conical marker, a concave hemispherical marker, a convex hemispherical marker, a disk marker, a spherical marker, or any ruled surface. This list illustrates the types of shapes that may be used for the marker 104, but the list is not exhaustive or limited to the possible shapes of the marker. In the examples, for cases other than a sphere and a disk, compensation is performed for accurate use.

[0055] The use of a retroreflective material (e.g., wrapped around the tag) causes the tag to passively reflect a significant amount of electromagnetic energy (e.g., IR light or near IR light) that is incident on the tag 104. The electromagnetic energy can be reflected so that most of the reflected energy is directed toward the optical sensor. Retroreflective materials can be considered to be materials that reflect electromagnetic waves back to their source with minimal scattering. For example, electromagnetic waves (e.g., light waves) are reflected back along a vector that is substantially parallel to, but in the opposite direction of, the source of the wave's transmission.

[0056] continue Figure 1 , the marker 104 is in an environment at a distance from the optical sensors 102a-102b. The position of the marker 104 can be based on a coordinate system 110. Although a Cartesian coordinate system 110 is shown, any coordinate system can be used, including a polar coordinate system, a cylindrical coordinate system or a spherical coordinate system, a homogeneous coordinate system, etc. The position of the marker 104 in the coordinate system 110 can be based on various quantities, such as a volume centroid 108 of the marker, which is the approximate center of the volume of the marker in 3D space. In this example, the volume centroid 108 is used to represent the position of the marker in the coordinate system 110. The tracking system 100 is configured to determine the position of the marker 104 by determining the coordinate values ​​of the volume centroid 108.

[0057] The tracking system 100 is configured to estimate the position of the volume centroid 108 of the marker 104. Because the volume centroid 108 is within the marker 104, the actual position of the volume centroid is not directly visible to the optical sensors 102a-102b. The tracking system 100 is configured to estimate the position of the volume centroid 108 based on the position of the optical centroid 118a-118b in the image data captured by the sensors 102a-102b. From the perspective of the optical sensors 102a-102b, the optical centroid 118a-118b represents the position of the marker 104 in space. From the perspective of each image sensor 102a-102b, the optical centroid 118a-118b is based on the reflection center 106 of the marker. The reflection center 106 is the 3D point where rays from the optical centroid passing through their respective pinholes approximately intersect. This is where the marker is estimated to be located by the system without compensation. The optical centroid 118a-118b is the central position in the optical signal on the image plane of each of the optical sensors 102a-102b. In fact, the position of each marker in space is the volume centroid 108, and this set of positions for the markers 104 enables the calculation of the pose when tracking the tracked object. When the volume centroid 108 is not at the same point as the reflection center 106, from the perspective of the optical sensors 102a-102b, the optical centroid 118a-118b is slightly offset from the actual position where the optical centroid should appear in the image plane. The corrected optical centroid 120a-120b represents the actual center of the marker as if the reflection center 106 coincides with the volume center 108. In other words, the optical centroid area is corrected to align with the centroid 108 so that the optical centroid and its pinhole are collinear with the centroid 108. In doing so, the reflection center (e.g., the 3D point approximation intersection of its rays) is moved to better approximate the position of the centroid 108.

[0058] For passive tags, the optical signal is generated by reflecting the optical signal emitted from a position near the optical sensor 102a-102b by the tag. The retroreflective coating of the tag 104 is configured to reflect the optical signal back to the source of the optical signal on an approximately parallel path to generate an optical signature (e.g., an optical pattern or an optical shape) representing the tag 104. The light is reflected by the tag and received by the optical sensor 102a-102b, and the optical sensor provides image data including the optical signature of the tag. For example, the optical signature can be extracted from image data captured by a camera. The optical sensors 102a-102b are each configured to capture the environment and capture an image of the illuminated tag 104. The illuminated tag 104 forms a shape in the image that can be extracted from the image using filtering or other image processing. The optical centroid 118a-118b represents the center of the illuminated tag 104 in the image plane of the image sensor 102a-102b.

[0059] like Figure 1 As shown, when the marker is not a sphere or a disk, the reflection center 106 does not necessarily coincide with the volume centroid 108 for the marker 104, resulting in an offset 112a-112b of the optical centroids 118a-118b. For example, for a cylindrical marker 104, when the disk is viewed from a position on the axis of the cylinder (viewing the top circular surface), or when the disk is viewed from a plane perpendicular to the axis and at its midpoint (viewing the rectangular aspect of its side), the reflection center 106 and the volume centroid 108 are aligned. However, when the cylinder is tilted, such as Figure 1 As shown, the viewing angle θ increases. As the angle θ increases, the volume centroid 108 and the reflection center 106 are further apart when viewed from the optical sensors 102a-102b. The distance between the volume centroid 108 and the reflection center 106 results in an offset 112a-112b on the respective optical sensors 102a-120b between the position of the optical centroid 118a-118b and the position of the corrected optical centroid 120a-120b in the 2D image plane if the reflection center 106 coincides with the volume centroid 108 for each marker. The exact offset value 112a-112b may depend on the configuration of the camera, including the position of the pinhole 114a-114b or lens of the camera. The tracking system is configured to determine these offset error vectors 112a-112b, and thus determine the positions of the corrected optical centroids 120a-120b. Similar compensations are performed for other geometric values ​​besides θ, such as range, azimuth, elevation, and roll of the marker 104.

[0060] The distance between the volume centroid 108 and the reflection center 106 (and the resulting error offset values ​​112a-112b) can vary depending on the precise parameters of the marker 104. For example, the size of the marker 104, the shape of the marker, the distance of the marker from the sensor, and the viewing angle of the marker relative to the optical sensors 102a-102b can affect the displacement of the volume centroid 108 from the reflection center 106, and subsequently the error offset vector 112a-112b values. Some of these parameters are at least initially unknown during tracking, including the distance of the marker 104 relative to the optical sensors 102a-102b and the viewing angle θ of the marker relative to the sensor. In one embodiment, the tracking system 100 can determine these values ​​during tracking by approximating the pose of the tracked object based on the reflection center 106 and the tracking system parameters.

[0061] See also Figure 1-Figure 2 ,The general tracking process is now described. Figure 2An example tracking system 200 is shown including a tracked object 216, markers 202, 204, and 206, a camera 208, an optical signal source 218, and a computing system 220. Tracking system 200 is configured to determine a position of tracked object 216 in an environment by determining a pose of each marker 202, 204, and 206.

[0062] The sensors 102a-102b may be part of a camera system and include a sensing device such as a CCD or CMOS optical sensor, which may capture reflected electromagnetic waves (e.g., optical signals) from a working volume including the tracked object. The image sensors 102a-102b each generate an image of the working volume, and in particular, an image of markers (such as marker 202, marker 204, and marker 206) on the tracked object 216. The optical signal may be emitted by one or more signal generators 218 (e.g., light emitting diodes (LEDs), LED arrays, or any other such IR sources) placed near the sensor of the camera 208.

[0063] The image obtained by the optical sensor generally includes the optical signatures of the marker 202, the marker 204, and the marker 206. The image captured by the camera 208 can be analyzed by a processing device of the computing system 220 containing appropriate image analysis software. Due to the highly retroreflective nature of the markers 202, the markers 204, and the markers 206, under electromagnetic waves of a certain wavelength, the markers appear as bright spots (optical signatures) in the captured image. This is because the markers 202, the markers 204, and the markers 206 are all coated in a reflective material that is illuminated in response to the optical signal emitted by the source 218. The optical signature appears as a shape in the image and can be extracted from the image using image processing.

[0064] Tracking system 200 determines the optical centroid 118a-118b for each of the bright spots in the image. Known parameters of image sensors 102a-102b (such as baseline, lens parameters, etc.) are combined with the optical centroid 118a-118b data to generate an initial pose estimate for marker 202, marker 204, and marker 206.

[0065] Typically, each marker is associated with at least two images, and the image sensor captures at least two markers in the tracking area. Predetermined relationships of the markers relative to each other (such as distance L1 and distance L2) are used to identify the optical centroid associated with each marker, and the reflection center is calculated from the optical centroid 118a-118b as an initial pose estimate for the pose 210, 212, and 214 of each respective marker 202, 204, and 206. These pose estimates can be represented in 3D space.

[0066] The computing system 220 estimates an error offset vector 112a-112b for the optical centroid 118a-118b for each marker 202, 204, and 206. The error offset vector 112a-112b for each marker 202, 204, and 206 indicates the difference between the observed optical centroid value 118a-118b and the true optical centroid value 120a-120b for each marker represented by the reflection center 106. The true optical centroid value 120a-120b is the location in the 2D image plane of each image sensor 102a-102b where the optical centroid would be if the reflection center of the marker coincided with the volumetric centroid of the marker.

[0067] To compensate for the error pattern of the cylindrical marker, the following input parameters can be used. The physical cognition of the tracking system 200 is known before tracking, including the camera 208 parameters. For example, the relative six degrees of freedom (6DOF) geometry between each of the optical sensors (e.g., two optical sensors) is known. The lens parameters (including focal length, center position on the sensor, distortion parameters, etc.) are known and saved in a configuration file. The relative geometry between the camera's illumination source 281 and the pinhole 114 is known. Typically, the camera 208 includes at least two optical sensors. In this example, in addition to the camera's parameters and the measured optical centroids 118a-118b, the initial poses of the markers 202, 204, and 206 from the camera can be determined based on the markers 202, 204, and 206 between two images captured by the two sensors in the sensor.

[0068] In addition to the parameters of camera 208 and light source 218, computing system 220 may be loaded with configuration data including a physical description of each of retroreflective markers 202, 204, and 206. For example, when the markers are cylinders, the computing device is loaded with the diameter of each cylinder, the height of each cylinder, the coverage configuration of the retroreflective material on each marker (e.g., coverage included only on the sides or the top). The retroreflective intensity according to the angular parameters (e.g., the angle of incidence) may be known. The configuration data may include data describing whether markers 202, 204, and 206 are the same, unique, or vary in some way and which marker is associated with which marker parameter. The configuration data may include an initial estimate of the position and orientation of the retroreflective markers relative to each other. Typically, this information is obtained from initial measurements or CAD data.

[0069] The parameters of the camera system and the parameters of the markers constitute an error model. In other words, the error model is based on known physical parameters of the tracking system. For example, optical sensor configuration parameters are included in the model. These parameters include one or more of the following: lens size, focal length, and optical sensor position in the environment for at least two optical sensors. The tracking system estimates the offset error vectors 112a-112b of the respective markers 202, 204, and 206 based on some method of estimating the initial marker pose, and based on these optical sensor configuration parameters and the physical parameters of the markers.

[0070] In some implementations, the computing system 220 obtains data specifying the initial position of one or more of the markers to assist in determining the position of the markers 202, 204, and 206 in the environment. If the data is available, the computing system 220 estimates the offset error vectors 210, 212, and 214 based on the data specifying the initial pose of at least one marker. For example, the tracking system 200 may include an inertial measurement unit (IMU) that provides orientation data representing the orientation of a marker in the plurality of markers. The computing system 220 is configured to estimate the initial orientation of the markers 202, 204, and / or 206 based on the orientation data provided by the IMU.

[0071] In some implementations, estimating the initial pose of the markers 202, 204, 206 in the environment includes knowledge of a predetermined spatial relationship between the markers 202, 204, and 206. In this example, the markers 202, 204, and 206 are rigidly fixed relative to each other. The computing system 220 is configured to extract optical signatures representing reflections of optical signals from the markers 202, 204, and 206 from the image data. Based on these optical signatures and their positions in the image, the computing system can estimate the initial poses of the markers 202, 204, and 206 in the environment.

[0072] The error model also includes data describing the marker 202, the marker 204, the marker 206. Determining the offset error vector by the computing system 220 includes analyzing one or more parameters of the marker 202, the marker 204, and / or the marker 206. The parameters may include the size of each of the markers 202, the marker 204, the marker 206, the shape of each marker, the relative position of the markers with respect to each other, and the configuration of the reflective coating on at least one marker. The computing system 220 uses the error model to determine the offset error vector 112a-112b for each marker 202, 204, and 206 on the tracked object. The data input to the error model is an initial estimate of the marker's orientation and position - the latter supports the marker's range, orientation, elevation, pitch, and roll.

[0073] The computing system 220 determines a corrected optical centroid 120a-120b for each marker based on the calculated offset error vector 112a-112b for each marker. The corrected optical centroid 120a-120b represents an offset in the 2D image plane from the measured optical centroid 118a-118b, which is the offset error vector 112a-112b.

[0074] The computing system 220 determines a corrected pose value for each of the markers 202, 204, and 206. The computing system 220 uses known parameters of the camera, known parameters of each of the markers 202, 204, and 206, and known parameters of the markers together in the system to calculate the poses 210, 212, and 214 for each respective marker 202, 204, and 206.

[0075] Given the position of camera 208 (e.g., a stereo camera or a camera with at least two optical sensors) and the positions of retroreflective markers 202, 204, and 206, computing system 220 computes the pose of tracked device 216. In some implementations, the position representing tracked object 216 may represent the tool tip 216a of the tracked object. In some implementations, based on configuration data preloaded into computing system 220, the position representing tracked object 216 in space may represent the center, edge, or any other portion of the tracked object. In other words, although 216a is shown at a particular location, once the positions of markers 202, 204, and 206 are determined, any portion of tracked object 216 may be discerned from the positions of markers 202, 204, and 206. For example, the point or center of a spherical stylus may be represented by point 216a.

[0076] In one aspect, the tracked object 216 can include any object to be tracked by the tracking system 200, but typically includes a rigid or semi-rigid object so that the markers 202, 204, and 206 do not move relative to each other in space during tracking. The tracked object 216 is typically configured to support multiple markers 202, 204, 206. The tracked object 216 is typically coupled to multiple markers (e.g., two markers, three markers, etc.). Additional markers may be added. During tracking, multiple markers are typically visible to the camera 208. In some implementations, the tracked device 216 includes a tool for performing a surgical task. In some implementations, the tracked device 216 includes an object to be presented in a virtual reality or augmented reality environment.

[0077] In the depicted example, the computing system 220 is connected to the camera 208 (or an array of two or more cameras) via at least one communication link. In other examples, the computing system 220 is located within the camera mounting unit. The computing system 220 includes a processing device. The computing system 220 may include one or more of various forms of digital computers (including, for example, laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers). The computing device is intended to represent various forms of mobile devices, including, for example, personal digital assistants, tablet computing devices, cellular phones, smart phones, and other similar computing devices. The components shown here, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the techniques described and / or claimed in this document.

[0078] Figure 3A-3C Examples of markers 302, 304, and 306 are shown for use in tracking systems 100, 200. For example, Figure 1 and Figure 2 The mark 104, mark 202, mark 204 and mark 206 may include Figure 3A-3C Any one of the markers 302, 304, or 306 in the example. Figure 3A In FIG. 3 , the indicia 302 comprises a cylinder having retroreflective material in a continuous manner on both the top and the sides. Figure 3B In FIG. 3 , the indicia 304 comprises a cylinder having retroreflective material in a discontinuous configuration on both the top and the sides. Figure 3C Indicia 306 is shown which comprises a cylinder having retroreflective material only on one side of the indicia.

[0079] Various techniques for providing retroreflective markings are possible. The marking may be covered with a coating or reflective material for retroreflection. The coating may include a reflective film or surface. In another example, the coating may include reflective objects having different geometries, shapes, etc. For example, retroreflection may be induced using microprisms, microspheres, or any other technology that supports retroreflection. In some implementations, the optical signal may not reside in the visible spectrum, but may be located in the infrared spectrum (or other regions of the electromagnetic spectrum). For example, the signal may include a wavelength between approximately 700-1000 nanometers.

[0080] In some implementations, a material in the light path through the marking can provide one or more absorption peaks for one or more wavelengths. The material can be, for example, an outer coating on the retroreflective layer, the material of the retroreflective layer itself, such as the material of the glass microspheres or the base material of the microprisms, etc. In some embodiments, the material is doped with a dopant to provide one or more absorption peaks for one or more wavelengths.

[0081] The markings can be made of various materials, shapes, sizes or coatings to achieve retroreflectivity. In some implementations, the coating or reflective material includes particles of solid glass, and the shape of the particles is approximately spherical. The solid glass beads can have a diameter of about 20-200 microns (μm). In some implementations, the particles include a rear surface optically associated with a specular reflector. The reflector can be a metal coating (e.g., a copper-based coating) or a non-metallic thin film stack designed to reflect electromagnetic waves. Typically, the reflective material is uniformly applied to the rear surface of the particles coated with the marking.

[0082] Figure 3A-3C The cylinder in is a non-limiting example of a possible shape for a marker. The marker may include a variety of regular and irregular shapes (e.g., disks, cubes, and spheres). Other shapes may include cones, cylinders, conic sections, or any object with a ruled surface.

[0083] Figure 4 Shown is a representation for use by a tracking system (e.g., Figure 2 Flowchart of an example process 400 for locating a marker using a tracking system 200 of the present invention. The process 400 includes illuminating (402) a marker of the tracking system and obtaining image data including the illuminated marker. The tracking system is configured to determine (404) an optical centroid of the marker from the image data. For each marker, the optical centroid may be represented by {U, V} coordinates. The tracking system is configured to estimate (406) an initial pose in 3D (e.g., Cartesian) coordinates based on parameters from a camera of the optical centroid determined for each of the images obtained by the image sensor.

[0084] The tracking system is configured to use an iterative compensation process. Typically, the tracking system is calibrated based on specific parameters of the marker of the tracked object, including the size, shape, orientation, relative position on the tracked object, and reflective properties of the marker. When object tracking begins, the tracking system does not require knowledge of the initial pose of the marker. Instead, the initial pose estimate is based on the image of the marker, the position of the marker relative to each other, and the parameters of the image sensor. In some implementations, the tracking system (408) uses data from multiple markers or data from another sensor such as an IMU to generate an initial pose estimate.

[0085] The tracking system determines (410) the offset error magnitude and direction of the optical center of mass for each marker in {U, V} space. As previously described, the configuration data and initial pose are used to estimate an error offset vector for the optical center of mass in the image data for each marker.

[0086] The tracking system is configured to perform (412) offset compensation on each of the optical centroids of the marker to determine a corrected optical centroid in the image plane of each image. This includes applying the determined error magnitude and direction to the optical centroid. Typically, the offset is performed in {U, V} coordinate space to adjust the optical centroid to represent the corrected optical centroid of the marker in two-dimensional image space, as previously described. The corrected optical centroid is used to determine the pose of the marker in 3D space.

[0087] The tracking system applies (414) the parameters of the optical sensor (as previously described) to the corrected optical centroid to determine a corrected pose for each marker. The corrected pose for each marker in the markers is represented in 3D space. The specific values ​​of the coordinates are based on the optical sensor parameters. In some implementations, the optical sensor may represent an origin for a 3D coordinate system, or some other reference point of a coordinate system in which the pose of each marker in the markers is determined. The tracking system determines the pose of the tracked object from the corrected pose of each marker in the markers because the relative positions and orientations of the markers on the tracked object are known. From this information, the position, orientation, or both of a specific point of the tracked object (e.g., the working tip of a surgical tool) can be determined and presented to the user on a user interface of the tracking system or an associated computing system in communication with the tracking system.

[0088] The tracking system is configured to iterate the estimation process by using the corrected pose estimate as an initial pose estimate for pose compensation of the same image data in subsequent iterations. As the offset error vector calculation for the optical centroid improves in each iteration, the corrected pose estimate becomes increasingly accurate. This results in a more accurate determination of the pose of the tracked object to which the marker is attached.

[0089] A number of implementations of the present invention have been described. However, it will be appreciated that various modifications may be made without departing from the spirit and scope of the present invention. For example, the positions of the marker and the tracked object may change over time. The computing device may be configured to automatically detect which bright spots in the image identified as marker reflections at a first time and a first position correspond to marker reflections identified at a second time and a second position. Therefore, other implementations are within the scope of the following claims.

[0090] Some implementations of the subject matter and operations described in this specification may be implemented in digital electronic circuitry, or in computer software, firmware, or hardware (including the structures disclosed in this specification and their structural equivalents), or in a combination of one or more of them. For example, in some implementations, tracking system 100 and tracking system 200 and computing system 220 may be implemented using digital electronic circuitry, or in computer software, firmware, or hardware, or in a combination of one or more of them. In another example, process 400 may be implemented using digital electronic circuitry, or in computer software, firmware, or hardware, or in a combination of one or more of them.

[0091] Some implementations described in this specification may be implemented as one or more groups or modules of the following: digital electronic circuits, computer software, firmware or hardware, or a combination of one or more of them. Although different modules may be used, each module need not be different, and multiple modules may be implemented on the same digital electronic circuit, computer software, firmware or hardware, or a combination thereof.

[0092] Some implementations described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions, which are encoded on a computer storage medium for execution by a data processing device or for controlling the operation of a data processing device. A computer storage medium may be or may be included in a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of them. In addition, although a computer storage medium is not a propagation signal, a computer storage medium may be a source or destination of computer program instructions encoded in an artificially generated propagation signal. A computer storage medium may also be one or more separate physical components or media (e.g., multiple CDs, optical disks, or other storage devices) or be included in one or more separate physical components or media.

[0093] The term "data processing device" includes all types of devices, equipment and machines for processing data, including, for example, a programmable processor, a computer, a system on a chip, or a plurality or combination of the above. In some implementations, the computing system 220 includes a data processing device as described herein. The device may include a dedicated logic circuit, for example, an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the device may also include code that creates an execution environment for the computer program in question, for example, code that constitutes processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination of one or more thereof. The device and the execution environment can implement a variety of different computing model infrastructures, such as web services, distributed computing, and grid computing infrastructures.

[0094] A computer program (also referred to as a program, software, software application, script or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages. A computer program may, but need not, correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., a file that stores a portion of one or more modules, subroutines or codes). A computer program may be deployed for execution on one computer or on multiple computers located at one site or distributed on multiple sites and interconnected by a communication network.

[0095] Some of the processes and logic flows described in this specification may be performed by one or more programmable processors executing one or more computer programs to perform actions by operating on input data and generating output. The processes and logic flows may also be performed by a dedicated logic circuit (e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit)), and the apparatus may be implemented as the dedicated logic circuit (e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit)).

[0096] For example, processors suitable for executing computer programs include both general and special purpose microprocessors and processors of any type of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The computer includes a processor for performing actions according to instructions and one or more memory devices for storing instructions and data. The computer may also include one or more mass storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data, or may be operatively coupled to one or more mass storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data to receive data from or transfer data to, or both. However, the computer need not have such a device. Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM, flash memory devices, and others), magnetic disks (e.g., internal hard disks, removable disks, and others), magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and memory may be supplemented by or incorporated into a dedicated logic circuit.

[0097] To provide interaction with a user, operations may be implemented on a computer having a display device (e.g., a monitor or another type of display device) for displaying information to the user and a keyboard and pointing device (e.g., a mouse, trackball, tablet computer, touch screen, or another type of pointing device) through which the user can provide input to the computer. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including auditory, voice, or tactile input. In addition, the computer may interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's client device in response to a request received from the web browser.

[0098] A computer system may include a single computing device or multiple computers operating in close proximity to each other or generally remote from each other and typically interacting through a communication network. Examples of communication networks include local area networks ("LAN") and wide area networks ("WAN"), interconnected networks (e.g., the Internet), networks including satellite links, and peer-to-peer networks (e.g., ad hoc peer-to-peer networks). The relationship of client and server may arise through computer programs running on the respective computers and having a client-server relationship to each other.

[0099] Figure 5An example computer system 500 is shown, which includes a processor 510, a memory 520, a storage device 530, and an input / output device 540. Each of the components 510, 520, 530, and 540 can be interconnected, for example, by a system bus 550. The processor 510 is capable of processing instructions for execution within the system 500. In some implementations, the processor 510 is a single-threaded processor, a multi-threaded processor, or another type of processor. The processor 510 is capable of processing instructions stored in the memory 520 or on the storage device 530. The memory 520 and the storage device 530 can store information within the system 500.

[0100] Input / output devices 540 provide input / output operations for system 500. In some implementations, input / output devices 540 may include one or more of the following: a network interface device (e.g., an Ethernet card), a serial communication device (e.g., an RS-232 port), and / or a wireless interface device (e.g., an 802.11 card, a 3G wireless modem, a 4G wireless modem, a 5G wireless modem, etc.). In some implementations, input / output devices may include a driver device configured to receive input data and send output data to other input / output devices (e.g., a keyboard, a printer, and a display device 560). In some implementations, mobile computing devices, mobile communication devices, and other devices may be used.

[0101] Although this specification contains many details, these details should not be interpreted as limitations on the scope of what can be claimed, but rather as descriptions of features that are unique to a particular example. Certain features described in this specification in the context of separate implementations may also be combined. Conversely, various features described in the context of a single implementation may also be implemented in multiple embodiments individually or in any suitable sub-combination.

[0102] Thus, specific embodiments of optical tracking systems and retroreflective markers and methods for tracking retroreflective markers using optical tracking systems have been disclosed. However, it should be apparent to those skilled in the art that more modifications are possible in addition to those already described without departing from the inventive concepts herein. Therefore, the inventive subject matter is not limited except in the spirit of the present disclosure. Furthermore, in interpreting the present disclosure, all terms should be interpreted in the broadest possible manner consistent with the context. In particular, the terms "comprises" and "comprising" should be interpreted as referring to elements, components, or steps in a non-exclusive manner, indicating that the mentioned elements, components, or steps may exist, or be utilized, or be combined with other elements, components, or steps not explicitly mentioned.

[0103] Where a range of values ​​is provided, it is understood that each intervening value between the upper and lower limits of the range (to the tenth of the unit of the lower limit unless the context clearly indicates otherwise), as well as any other stated or intervening value within the stated range, is included in the invention. The upper and lower limits of these smaller ranges may independently be included in the smaller ranges of the invention and are also included in the invention, subject to any specifically excluded limitations within the stated ranges. Where a stated range includes one or both of the limits, ranges excluding one or both of those included limits are also included in the invention.

[0104] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the invention belongs. Although any methods and materials similar or equivalent to those described herein can also be used in the practice or testing of the present invention, a limited number of exemplary methods and materials are described herein.

Claims

1. A tracking system for tracking one or more reflective markers, the tracking system comprising: at least two optical sensors configured to obtain image data of an environment including at least one marker, wherein at least a portion of the at least one marker is configured to reflect an optical signal; and a processing device configured to obtain the image data from the at least two optical sensors, the processing device configured to perform operations comprising: extracting an optical signature from the image data, the optical signature representing a reflection of the optical signal of the at least one marker; determining an optical centroid of the optical signature in the image data; estimating an initial pose of at least one marker in the environment based on the optical centroid; determining a bias error vector from the optical centroid of the at least one marker based on the estimate of the initial pose; determining a corrected optical center of mass of the at least one marker based on the offset error vector and the optical center of mass; and A corrected three-dimensional (3D) position of the at least one marker in the environment is determined based on the corrected optical center of mass of the at least one marker.

2. The tracking system according to claim 1, wherein: Estimating an initial pose of the at least one marker in the environment based on the optical centroid comprises: obtaining one or more optical sensor configuration parameters, the one or more optical sensor configuration parameters comprising one or more of: lens size, focal length, and optical sensor position in the environment for the at least two optical sensors; and A plurality of offset error vectors of the marker in the environment are estimated based on the one or more optical sensor configuration parameters.

3. The tracking system according to claim 1, wherein: The at least one marker is a first marker, and wherein estimating an initial pose of the at least one marker in the environment comprises: receiving spatial data representing a predetermined spatial relationship between the first marker and one or more additional markers of a plurality of markers, the first marker and the one or more additional markers being rigidly fixed relative to each other on a tracked object; extracting an optical signature from the image data, the optical signature representing reflections of the optical signal from the one or more additional markers; and An initial pose of the first marker in the environment is estimated based on the optical signature of the one or more further markers and the predetermined spatial relationship between the first marker and the one or more further markers.

4. The tracking system of claim 1 , further comprising an inertial measurement unit (IMU) configured to provide orientation data, the orientation data representing an orientation for the at least one marker; and in, Estimating the initial pose of the at least one marker is based on orientation data provided by the IMU.

5. The tracking system according to claim 1, wherein: Determining the offset error vector is also based on one or more parameters of the at least one mark, and for a given mark, the one or more parameters include the size of the given mark, the shape of the given mark, the relative position of the given mark with respect to at least one other mark when there are multiple marks, and the configuration of the reflective coating on the given mark.

6. The tracking system according to claim 1, wherein: The operations also include: using the corrected three-dimensional position as part of the initial pose of the at least one marker in the environment; and The offset error vector is determined based on the corrected three-dimensional position of the at least one marker by an iterative process.

7. The tracking system according to claim 1, wherein: The operations also include: A pose of a portion of a tracked object in the environment is determined based on the corrected three-dimensional position of the at least one marker, wherein the at least one marker is fixed to the tracked object at a predetermined position on the tracked object.

8. The tracking system according to claim 7, wherein: The tracked object is a surgical tool, and wherein the portion of the tracked object is a working portion of the tool.

9. The tracking system according to claim 1, wherein: The at least one mark includes one of the following: a cylinder with reflective sides only, a cylinder with reflective sides and a reflective top, a convex conical mark, a concave conical mark, a concave hemispherical mark, a convex hemispherical mark, a disk mark, any mark with a straight-grained surface, and a spherical mark.

10. The tracking system according to claim 1, wherein: The at least one mark is one mark among a plurality of marks that are identical or nearly identical to each other in size or shape.

11. The tracking system according to claim 1, wherein: The at least one marking is one of a plurality of markings, each of which is unique in size or shape relative to at least one other marking of the plurality of markings.

12. The tracking system according to claim 1, wherein: The optical signal comprises infrared (IR) or near infrared radiation.

13. A method performed by a computing system for tracking one or more reflective markers, the method comprising: obtaining image data of an environment including at least one marker from at least two optical sensors, wherein at least a portion of the at least one marker is configured to reflect an optical signal; extracting an optical signature from the image data, the optical signature representing a reflection of the optical signal of the at least one marker; determining an optical centroid of the optical signature in the image data; estimating an initial pose of at least one marker in the environment based on the optical centroid; determining a bias error vector from an optical centroid of said at least one marker based on said estimate of said initial pose; determining a corrected optical center of mass of the at least one marker based on the offset error vector and the optical center of mass; and A corrected three-dimensional (3D) position of the at least one marker in the environment is determined based on the corrected optical center of mass of the at least one marker.

14. The method according to claim 13, wherein: Estimating an initial pose of the at least one marker in the environment based on the optical centroid comprises: obtaining one or more optical sensor configuration parameters, the one or more optical sensor configuration parameters comprising one or more of: lens size, focal length, and optical sensor position in the environment for the at least two optical sensors; and A plurality of offset error vectors of the marker in the environment are estimated based on the one or more optical sensor configuration parameters.

15. The method according to claim 13, wherein: The at least one marker is a first marker, and wherein estimating an initial pose of the at least one marker in the environment comprises: receiving spatial data representing a predetermined spatial relationship between the first marker and one or more additional markers of a plurality of markers, the first marker and the one or more additional markers being rigidly fixed relative to each other on a tracked object; and extracting an optical signature from the image data, the optical signature representing reflections of the optical signal from the one or more additional markers; The initial pose of the first marker in the environment is estimated based on the optical signatures of the one or more further markers and the predetermined spatial relationship between the first marker and the one or more further markers.

16. The method according to claim 13, further comprising: providing orientation data by an inertial measurement unit (IMU), the orientation data representing an orientation for the at least one marker, Wherein, estimating the initial pose of the at least one marker is based on orientation data provided by the IMU.

17. The method according to claim 13, wherein: Determining the offset error vector is also based on one or more parameters of the at least one mark, and for a given mark, the one or more parameters include the size of the given mark, the shape of the given mark, the relative position of the given mark with respect to at least one other mark when there are multiple marks, and the configuration of the reflective coating on the given mark.

18. The method according to claim 13, further comprising: using the corrected three-dimensional position as part of the initial pose of the at least one marker in the environment; as well as The offset error vector is determined based on the corrected three-dimensional position of the at least one marker by an iterative process.

19. The method according to claim 13, further comprising: A pose of a portion of a tracked object in the environment is determined based on the corrected three-dimensional position of the at least one marker, wherein the at least one marker is fixed to the tracked object at a predetermined position on the tracked object.

20. One or more non-transitory computer-readable media storing instructions that, when executed by a computing system, cause the computing system to perform operations comprising: obtaining, by at least two optical sensors, image data of an environment including at least one marker, wherein at least a portion of the at least one marker is configured to reflect an optical signal; extracting an optical signature from the image data, the optical signature representing a reflection of the optical signal reflected by the at least one marker; determining an optical centroid of the optical signature in the image data; estimating an initial pose of the at least one marker in the environment based on the optical centroid; determining a bias error vector from the optical centroid associated with the at least one marker based on the estimate of the initial pose; determining a corrected optical center of mass associated with the at least one marker based on the offset error vector and the optical center of mass; and A corrected three-dimensional (3D) position in the environment associated with the at least one marker is determined based on the corrected optical center of mass of the at least one marker.

Images