Camera and position sensor calibration

Abstract

The present disclosure relates to systems, devices, and methods for calibrating an endoscope using a position sensor-to-camera transformation.

Description

[Technical field]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims priority to U.S. Provisional Patent Application No. 63 / 295,512, entitled “CALIBRATION OF CAMERA AND LOCATION SENSOR,” filed December 30, 2021, the disclosure of which is incorporated by reference in its entirety herein. [Background technology]

[0002] Various medical procedures involve the use of one or more devices configured to penetrate the human anatomy to reach a treatment site. Certain operational processes can include locating a medical instrument within a patient and visualizing an area of ​​interest within the patient. To do so, many medical instruments may include sensors to track the instrument's location and may include vision capabilities, such as an embedded camera or compatible use with a visual probe. [Brief description of the drawings]

[0003] Various embodiments are illustrated in the accompanying drawings for illustrative purposes and should not be construed as limiting the scope of the present disclosure in any way. In addition, various features of different disclosed embodiments may be combined to form further embodiments that are part of the present disclosure. Throughout the drawings, reference numbers may be reused to indicate correspondence between referenced elements. [Figure 1] FIG. 1 illustrates an automatic calibration system in accordance with an exemplary embodiment. [Figure 2A] 1A-1C illustrate different exemplary embodiments of a calibration control platform that provides robotic and / or manual control during a calibration procedure. [Figure 2B] 1A-1C illustrate different exemplary embodiments of a calibration control platform that provides robotic and / or manual control during a calibration procedure. [Figure 2C]1A-1C illustrate different exemplary embodiments of a calibration control platform that provides robotic and / or manual control during a calibration procedure. [Diagram 3] FIG. 2 is a block diagram illustrating a method for automatically calibrating an endoscope, for example, such that a transformation between the coordinate frames of a position sensor and a camera is determined, in accordance with one or more embodiments. [Figure 4] 4 is a block diagram illustrating an exemplary embodiment of an automatic calibration system for implementing the method of FIG. 3. [Diagram 5] 1 is a block diagram illustrating image-based visual servoing calibration according to an exemplary embodiment. [Figure 6] 1 is a block diagram illustrating a pose-based visual servoing calibration approach in accordance with an illustrative embodiment; [Figure 7] FIG. 2 is a block diagram illustrating a method for automatically calibrating an endoscope, for example, such that a transformation between the coordinate frames of a position sensor and a camera is determined, in accordance with one or more embodiments. [Figure 8] 1 is a block diagram illustrating an example embodiment of an automatic calibration system implementing a method for position sensor servo control based calibration. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0004] The directions provided herein are for convenience only and do not necessarily affect the scope or meaning of the disclosure. Although certain exemplary embodiments are disclosed below, the subject matter extends beyond the specifically disclosed embodiments to other alternative embodiments and / or uses, as well as to modifications and equivalents thereof. Thus, the scope of claims that may arise from this specification is not limited by any of the specific embodiments described below. For example, in any method or process disclosed herein, the acts or operations of the method or process may be performed in any suitable order and are not necessarily limited to any particular disclosed order. Various operations may be described in sequence as multiple separate operations in a manner that may be helpful in understanding a particular embodiment, however, the order of description should not be construed to imply that these operations are order dependent. Furthermore, the structures, systems, and / or devices described herein may be embodied as integrated or separate components. For purposes of comparing various embodiments, certain aspects and advantages of these embodiments are described. Not necessarily all such aspects or advantages are realized by any particular embodiment. Thus, for example, various embodiments may be performed in a manner that achieves or optimizes one advantage or group of advantages taught herein without necessarily achieving other aspects or advantages that may also be taught or suggested herein.

[0005] overview The present disclosure relates to systems, devices, and methods for calibrating an endoscope. For example, an endoscope can be manufactured with multiple sensors, including robotic encoders, position sensors, and camera sensors. Since readings from different sensors are represented in their respective coordinate frames, the present disclosure contemplates calibration to generate one or more transformations between the coordinate frames of two or more of these sensors, such as position sensors and camera sensors. As used herein, a transform (interchangeably referred to herein as a "transformation") can be data that provides a mapping between positions across multiple coordinate frames. In some cases, the transformations described herein can be unidirectional, such that the transformations only provide a mapping from a position in one coordinate frame to another coordinate frame, and not vice versa. In other cases, the transformations described herein can be bidirectional, such that the transformations allow for mapping of a position from a first coordinate frame to a position in a second coordinate frame, and vice versa (e.g., a position from the second coordinate frame to a position in the first coordinate frame). The transformation maps positions in the position sensor coordinate frame to positions in the camera coordinate frame (and sometimes vice versa) and is referred to as the "position sensor-camera" transformation. The position sensor-camera transformation can be useful in allowing readings (e.g., position and rotation) from one sensor (e.g., position sensor or camera sensor) to be mapped to readings in the other coordinate frame. Furthermore, this calibration to generate the position sensor-camera transformation can be useful in allowing readings from either or both of the sensors to be used in a unified coordinate system, such as a robot coordinate system. Additionally, some embodiments automate at least some aspects of calibrating an endoscope, which can be useful in calibrating robotic endoscopes in industrial environments.

[0006] Automated Endoscope Calibration FIG. 1 illustrates an automatic calibration system 100, according to an exemplary embodiment. The automatic calibration system 100 can include an endoscope 110, a calibration pattern device 120, and a calibration control platform 130. The endoscope 110 can be any suitable medical instrument capable of being coupled to a medical robotic system capable of providing positional and visual data. Thus, the term "endoscope" as used herein can refer to any type of elongated medical instrument having image generating, viewing, and / or capturing capabilities equipped to also include sensors usable to determine the position or shape of the endoscope. The endoscope can also be configured to be introduced into any type of organ, cavity, lumen, chamber, and / or space of the body. For example, references herein to a scope or endoscope may refer to a ureteroscope (e.g., for accessing the urinary tract), a laparoscope, a nephroscope (e.g., for accessing the kidneys), a bronchoscope (e.g., for accessing the airways such as the bronchi), a colonoscope (e.g., for accessing the colon), an arthroscope (e.g., for accessing a joint), a cystoscope (e.g., for accessing the bladder), a borescope, etc.

[0007] 1, endoscope 110 may include a camera 112, a position sensor 114, and a calibration data store 116. Camera 112 may be any suitable device capable of providing an endoscope with a visual capability, such as, by way of example and not limitation, a camera, a fluoroscope, etc. The camera may be integrated into the instrument itself or may be provided as a detachable probe that passes through the working channel of the endoscope.

[0008] The position sensor 114 may be any suitable sensor (or sensors) capable of providing position or movement data regarding the endoscope. Examples of position sensors include, but are not limited to, electromagnetic (EM) sensors, shape sensing fibers, gyroscopes, accelerometers, magnetometers, etc. As with the camera 112, the position sensor 114 may be integrated directly into the endoscope or may be provided as a detachable probe that passes through a working channel of the endoscope.

[0009] As the callouts in FIG. 1 indicate, the camera 112 and the position sensor 114 each have their own respective coordinate frames. For example, the camera 112 has a camera coordinate frame 160, and the position sensor 114 has a position sensor coordinate frame 180. Each of the coordinate frames 160, 180 may be approximately axially aligned with the tip of the endoscope 110, although it should be understood that there may be slight misalignment with each of the coordinate frames 160, 180 due to, for example, manufacturing differences. The embodiments described herein attempt to resolve these differences by generating a position sensor-to-camera transformation to allow a position and orientation in one coordinate frame to be mapped to the other, or possibly to a universal coordinate frame, such as a robot coordinate frame.

[0010] The calibration data store 116 may be a computer-readable medium configured to store, among other things, data indicative of the position sensor-to-camera transformation. Furthermore, in some embodiments, the calibration data store 116 may be configured to interface with a medical robotic system to communicate the position sensor-to-camera transformation to the medical robotic system. The communication between the calibration data store 116 and the medical robotic system may be wireless or wired communication. Examples of the calibration data store 116 include radio frequency identification (RFID), USB, FireWire, Ethernet, ATA / IDE, SCSI, PCI, and any other suitable communication protocol. As shown in FIG. 1, the calibration data store 116 may be coupled to an instrument handle of an endoscope. As used herein, "instrument handle" may refer to a base portion of an endoscope configured to be removably coupled to a robotic medical system. The instrument handle may include input elements for steering and actuating an instrument driven by an output element of the robotic medical system.

[0011] The calibration pattern device 120 may be a device having a known or determinable visual pattern. Although FIG. 1 illustrates the calibration pattern device 120 as having a checkerboard pattern, it should be understood that the disclosure is not so limited. Other embodiments may include other types of visual indicators. For example, the calibration pattern device 120 may include any type of visual indicator with features that are easily identifiable via visual processing algorithms. In some embodiments, the calibration pattern device 120 may further include an EM field generator. This may be useful when the position sensor 114 is an EM sensor, because having the pattern and the EM field generator in the same device aligns both the high and low lines for the camera 112 and the position sensor 114 with the calibration pattern device 120, which may be useful in the calibration steps described in more detail below.

[0012] The calibration control platform 130 may be a platform that facilitates the process of calibrating an endoscope, i.e., generating a position sensor-to-camera transformation and transferring or otherwise storing the position sensor-to-camera transformation to the endoscope. As discussed in more detail, the calibration control platform 130 may provide mechanical features and control circuitry for placing the endoscope 110 (and, in turn, the camera 112 and position sensor 114) and the calibration pattern device 120 in a determinable position relative to one another. The calibration control platform 130 may also include control circuitry for recording position sensor attitude data and camera attitude data and then calculating the position sensor-to-camera transformation therefrom. As used herein, "position sensor attitude" may refer to any element of attitude (e.g., position and / or orientation) within the coordinate frame of the position sensor. The coordinate frame of the position sensor may be determined by the tracking modality utilized by the position sensor 114 of the endoscope 110. For example, in the case of an EM sensor, the coordinate frame may be defined by six degree of freedom (DOF) readings from the sensor in an EM field generated by an EM field generator embodied in the calibration pattern device 120 or a stand-alone device. As another example, in the case of a shape sensing fiber, the coordinate frame may be defined by shape data from the shape sensing fiber relative to a known location, such as the base of the shape sensing fiber, which may be located in the handle of the endoscope or elsewhere.

[0013] In comparison, "camera pose" may refer to some elements of the pose (e.g., position and / or orientation) in the camera's coordinate frame. The camera's coordinate frame may be based on a vision algorithm that operates on an image acquired by the camera. In addition, the camera coordinate frame may also be influenced by a camera calibration that measures characteristics of the camera such as the principal point and focal length and distortion coefficients.

[0014] Although not shown, the calibration control platform 130 may also include control circuitry for transferring data indicative of the position sensor-to-camera transformation to the calibration data store 116 of the endoscope 110.

[0015] 2A and 2B are diagrams illustrating two different exemplary embodiments of a calibration control platform 130 that provides robotic control during a calibration procedure. For example, FIG. 2A illustrates an embodiment in which the calibration control platform 130 includes a robotic arm 220a capable of robotically positioning the endoscope 110 relative to the calibration pattern device 120 at several determinable poses (e.g., 210a, 212a, 214a). The calibration control platform 130 may include control circuitry configured to record the position sensor pose data and the camera pose data at each of the positions to form a data set including pairs of sensor pose data and camera pose data, each pair corresponding to a different pose (e.g., 210a, 212a, 214a). From the data set, the control circuitry of the calibration control platform 130 may calculate a position sensor-to-camera transformation and store data indicative of the position sensor-to-camera transformation on a computer-readable medium configured to be coupled to the endoscope.

[0016] In contrast, FIG. 2B illustrates an embodiment in which the calibration control platform 130 includes a robotic arm 220b that robotically positions the calibration pattern device 120 relative to the endoscope 110 at several determinable poses (e.g., 210b, 212b, 214b). The calibration control platform then operates similarly to the calibration control platform of FIG. 2A. That is, the control circuitry of the calibration control platform 130 may be configured to record the position sensor pose data and the camera pose data at each of the positions to form a data set including pairs of sensor pose data and camera pose data, each pair corresponding to a different pose (e.g., 210b, 212b, 214b). From the data set, the control circuitry of the calibration control platform 130 may calculate a position sensor-to-camera transformation and store data indicative of the position sensor-to-camera transformation on a computer-readable medium configured to be coupled to the endoscope.

[0017] 2C illustrates an exemplary embodiment of a calibration control platform 130c with manual adjustment. For example, the calibration control platform 130c includes a jig 260 having a number of positioning slots 280, 282, 284, 286, a calibration pattern device 120c, and a calibration control circuit 290. The calibration control circuit 290 is configured to generate and store camera pose data and position sensor pose data from the endoscope 110c, generate a camera-to-position sensor transformation from the stored camera pose data and position sensor pose data, and transfer data indicative of the camera-to-position sensor transformation to a calibration data store (e.g., the calibration data store 115 shown in FIG. 1).

[0018] To generate the camera-to-position sensor transformation, the operator employs an approach that iteratively positions the endoscope 110c in several different positions using positioning slots 280, 282, 284, 286 of the jig 260. Each of the positioning slots 280, 282, 284, 286 stabilizes the endoscope in a different configuration relative to the calibration control platform 130c. At each iteration of positioning the endoscope 110c in a different positioning slot, the calibration control circuitry 290 captures and stores the camera pose data and position sensor pose data at that particular positioning slot.

[0019] In some embodiments, the iterations of positioning endoscope 110c in each of slots 280, 282, 284, 286 are performed for one or more sides of the endoscope. For example, with the endoscope in a given orientation, the endoscope is placed in each of slots 280, 282, 284, 286, and with the endoscope in a different orientation, the endoscope is again placed in each of slots 280, 282, 284, 286, and so on for any number of orientations.

[0020] Upon completing iterations of positioning the endoscope 110c within the different slots 280, 282, 284, 286 (possibly for multiple orientations of the endoscope 110c), the calibration control platform 130c generates a camera-to-position sensor transform. Techniques for generating the camera-to-position sensor transform are described in more detail below.

[0021] It should be understood that other embodiments of the calibration control platform are contemplated by this disclosure. For example, an exemplary embodiment consistent with this disclosure includes a calibration control platform that robotically controls both the endoscope and the calibration pattern device to reach determinable poses.

[0022] Calibration Method and Operation Details of the operation of an exemplary auto-calibration system will now be described. The methods and operations disclosed herein are described in conjunction with the auto-calibration system 100 shown in FIG. 1. However, it should be understood that the methods and operations may be performed by any of the components discussed herein, alone or in combination. Additionally, the operation of the auto-calibration system is described in conjunction with a two-phase approach. However, it should be understood that the present disclosure contemplates other embodiments that may use additional phases or any sub-operations or sub-phases disclosed herein, and thus the present disclosure is not limited to a two-phase approach.

[0023] The automatic calibration system may implement a two-phase approach to automatically calibrate an endoscope. In general, the two-phase approach may include a first phase that generates an initial transformation (e.g., a camera-to-robot-based transformation or a position sensor-to-robot-based transformation) and a second phase that uses the initial transformation to servo the endoscope to a determinable pose and further generate a final position sensor-to-camera transformation. An exemplary embodiment of this process will now be described in more detail.

[0024] i. Camera-based servo control approach FIG. 3 is a block diagram illustrating a method 300 for automatically calibrating an endoscope, for example, such that a transformation between the coordinate frames of a position sensor and a camera is determined. As FIG. 3 illustrates, the method 300 may include a first phase 302. During the first phase 302, the automatic calibration system may generate a camera-to-robot base transformation. The first phase 302 may involve several sub-blocks that collectively represent the first phase 302 of a multi-phase approach. For example, the first phase 302 may include sub-blocks 302a, 302b, and 302c. In sub-block 302a, the automatic calibration system may command the robot a first movement between the endoscope and a calibration pattern device to achieve one or more robot poses in the robot coordinate frame. As used herein, a set of one or more robot poses may be commanded and expressed using coordinates of the robot coordinate frame. For example, the robot controller may issue commands to the robot arm to move the endoscope to a given position in the robot coordinate frame (e.g., to a given x, y, z position and / or yaw, pitch, roll orientation of the robot coordinate frame). Similarly, the robot controller may issue commands to the robot arm to move the calibration pattern device to a given position in the robot coordinate frame (e.g., to a given x, y, z position and / or yaw, pitch, roll orientation of the robot coordinate frame). In some embodiments, the robot poses may be selected such that images captured by the camera in each of the robot poses capture the entire pattern displayed by the calibration device.

[0025] In sub-block 302b, the auto-calibration system may record the robot pose data and the corresponding first camera pose data in each of the one or more robot poses achieved in sub-block 302a. As explained above, the camera pose data may be data operable to identify the pose (or at least some aspects thereof) of the camera in the camera coordinate frame. For example, in some embodiments, the auto-calibration system may derive the camera pose data from the image data acquired in one of the robot poses of block 302a and extract the position information therefrom. In an exemplary embodiment, the auto-calibration system may identify the location of specific features of the calibration pattern represented in the image captured in one of the robot poses and determine the camera pose in the camera coordinate frame based on the feature location. In another exemplary embodiment, a perspective-n-point algorithm may be used to obtain the current camera pose of the object pattern in the camera coordinate frame. The input of the perspective-n-point algorithm may be the object point positions defined on the pattern and their corresponding positions detected on the image, along with the camera intrinsic parameters (obtained in the camera calibration).

[0026] Thus, the auto-calibration system may store n pairs of robot pose data and camera pose data, where n is the number of robot poses achieved in sub-block 302a. The robot pose data stored in sub-block 302b may be the robot pose of the command sent to the robot controller. In other embodiments, there may be some sensor feedback that estimates the pose of the robot system, such as an external camera, an EM sensor embedded in the robot, etc.

[0027] The auto-calibration system calculates a camera-to-robot base transformation in sub-block 302c using the pair of robot pose data (or a transformation thereof) and corresponding first camera pose data (or a transformation thereof) as recorded in sub-block 302b. As described in more detail elsewhere in this disclosure, generating the camera-to-robot base transformation may involve collecting paired measurements of the robot transformation and the pattern-to-camera transformation pair, and solving: RX R =X R B

[0028] In this calculation, R represents the relative transformation between the robot poses (e.g., R1, R2) in the first phase 302, and R=R2_inv * R1. The robot pose can be expressed as a transformation from the robot base to the end effector (as can be determined using the robot's forward kinematics). B represents the relative transformation between the camera poses (e.g., B1, B2) in the first phase 302, where B=B2_inv * The camera pose can be defined as R1, B1, B2, and B3. The camera pose can be the transformation from the camera to the visual pattern. Thus, R1 can be the transformation from the robot base to the end effector for the first pose, B1 can be the transformation from the camera to the visual pattern for the first pose, R2 can be the transformation from the robot base to the end effector for the second pose, and B2 can be the transformation from the camera to the second pose pattern for the second pose. (R1, B1) and (R2, B2) are pairs that are meant to be synchronized. There are also probabilistic methods that use batches of data that do not require the R, B pairs to be synchronized. X R represents the camera-to-robot base transformation.

[0029] During the second phase 304, the auto-calibration system may generate a position sensor-to-camera transformation. Block 304 may include several sub-blocks, such as sub-blocks 304a, 304b, and 304c. In sub-block 304a, the auto-calibration system may use the camera-to-robot-based transformation generated during the first phase 302 to robotically command a second movement between the endoscope and the calibration pattern device to achieve one or more camera poses in the camera coordinate frame. As mentioned above, the camera poses may be expressed using coordinates in the camera coordinate frame. For example, the set of one or more camera poses in block 304a may be expressed as feature positions of a visual pattern on the calibration pattern device, or as object point positions defined on the pattern and their corresponding positions detected on the image, together with camera intrinsic parameters (obtained in the camera calibration).

[0030] It should be appreciated that sub-block 304a may operate by commanding robot motion between the calibration pattern device and the endoscope such that a determinable camera pose in the camera coordinate frame is achieved. However, the camera coordinate frame is a different coordinate frame than the robot coordinate frame. To account for this difference, for example, the auto-calibration system may determine that the camera needs to move 50° to the left in the camera coordinate frame to achieve a desired camera pose. If the auto-calibration system commands the robot controller to move 50° to the left, this movement occurs in the robot coordinate frame, and this movement may result in the camera moving to the left at other angles in the camera coordinate frame. Thus, in normal operation, the movement required to move the current camera pose to the desired camera pose does not necessarily translate as the same movement in the robot coordinate frame. The camera-to-robot based transformation facilitates this operation in that the auto-calibration system can map movements in the camera coordinate frame to movements in the robot coordinate frame. This approach is referred to as a visual servoing approach and is described in more detail below.

[0031] In sub-block 304b, the auto-calibration system may record second camera pose data and corresponding position sensor pose data at each of the one or more camera poses. As explained above, the position sensor pose data may be data operable to identify the pose (or at least some aspects thereof) of the position sensor within the position sensor coordinate frame. Thus, the auto-calibration system may store n pairs of camera pose data and position sensor pose data, where n is the number of camera poses arrived at in sub-block 304a.

[0032] In sub-block 304c, the auto-calibration system calculates a camera-to-position sensor transformation using the pair of camera pose data (or a transform thereof) and corresponding position sensor pose data (or a transform thereof) as recorded in sub-block 304b. Generating the camera-to-position sensor transformation may include collecting paired measurements of the electromagnetic field generator-to-EM transformation and the pattern-to-camera transformation pairs, and solving: AX C2LS =X C2LS B

[0033] where A represents the relative transformation between position sensor poses of the second phase 304 (e.g., from the position sensor to the field generator if the position sensor is an EM sensor), B represents the relative transformation between pattern poses of the second phase 304, and X C2LS represents the camera-position sensor transformation.

[0034] In block 306, the automatic calibration system stores data indicative of the camera-to-position sensor transformation in a computer readable medium of the endoscope, such as the calibration data store 116 of Figure 1. As discussed above with respect to Figure 1, the calibration data store may be an RFID tag, but may be any suitable device capable of storing data indicative of the camera-to-position sensor transformation and communicating the camera-to-position sensor transformation to the robotic system when the endoscope is coupled to the robotic system.

[0035] FIG. 4 is a block diagram illustrating an example embodiment of an auto-calibration system 400 for implementing the method 300 of FIG. 3. As FIG. 4 illustrates, similar to the method 300 of FIG. 3, the auto-calibration system 400 operates to perform multiple phases, a first phase 402 and a second phase 404, where the first phase 402 operates to generate a camera-to-robot base transformation 406 and the second phase 404 operates to generate a position sensor-to-camera transformation 408. In the first phase 402, a set of robot poses 412 in the robot coordinate frame can be empirically pre-determined to obtain the camera-to-robot base transformation 406. It is worth noting that in some embodiments, the pattern should be fully visible in the captured image for each robot pose. For example, the robot controller 414 may issue commands to the robot 416 to move the endoscope to a given position in the robot coordinate frame (e.g., to a given x, y, z position and / or yaw, pitch, roll orientation of the robot coordinate frame).

[0036] After executing each robot pose 412, the corresponding robot pose and camera pose data are recorded. For example, after a robot pose is reached, the robot pose data associated with the robot pose (e.g., x, y, z and yaw, pitch, roll coordinates of the robot command) is stored along with the camera pose data obtained based on an analysis of the constituent patterns represented in the image data acquired by the endoscopic camera 418. Using the resulting set of data pairs, the camera-to-robot base transformation 406 is calculated.

[0037] In the second phase 404, the camera servo control based calibration controller 422 performs a camera servo control based calibration using the camera-to-robot base transformation 406. In the camera servo control based calibration, the auto-calibration system may perform a 6-DOF motion of the endoscope or the calibration pattern device to command the robot movement to reach a set of camera poses (e.g., camera poses 420) such that, once the camera poses are reached, the corresponding position sensor poses can be recorded. By way of example and not limitation, this disclosure describes different ways in which the auto-calibration system can ensure that the camera poses are reached. The first approach is called an image-based servo control approach, in which the positions of the features of the visual pattern of the calibration pattern device are utilized. The second approach is a pose-based servo control approach, in which the current camera pose is estimated, for example, from visual processing of the image and the camera intrinsic parameters.

[0038] Both the image-based and position-based servo control approaches aim to reduce the error between an expected state (position or pose) and the current state by providing a feedback loop where the system can compare some attribute associated with the endoscope's current camera pose with the desired camera pose. Furthermore, the camera servo control based calibration controller uses a camera-to-robot base transformation 406 to convert the movements required to achieve the camera pose in the camera coordinate frame into movements in the robot coordinate frame. In this way, the control circuitry (referred to herein as a control law module) implementing the control laws of the camera servo control based calibration controller can generate instructions to the robot controller of one or more robot arms to effect movements in the robot coordinate frame that minimize the difference between the current camera pose in the camera coordinate frame and the desired camera pose.

[0039] The corresponding camera pose data (e.g., via camera 418) and position sensor pose data (e.g., via position sensor 424) are recorded whenever a desired camera pose is reached, and are used to calculate the final position sensor-to-camera transformation 408 by a camera servo control based calibration controller.

[0040] Next, by way of example and not limitation, several different camera servo control based calibration approaches are described. As briefly mentioned above, in a camera servo control based calibration approach, an automatic calibration system adjusts the camera pose (e.g., B i ) is reached, the corresponding position sensor attitude (e.g., A i ) can be recorded by the 6-DOF motion of the endoscope or calibration pattern device.

[0041]

number

[0042] FIG. 5 is a block diagram illustrating image-based visual servoing calibration, according to an example embodiment.

[0043] In block 502, the auto-calibration system selects a target camera pose from a set of camera poses (e.g.,

[0044]

number

[0045] The control law module 506 calculates the target camera pose B t (e.g., the target position of a feature) and the current camera pose, B c Calculate the robot command to achieve the robot movement, B t B. Reducing the difference between the target camera pose and the current camera pose c As shown in Figure 5, the current feature position (e.g., the current camera pose B c ) is provided as part of a feedback path in which the camera sensor 512 captures an image of the calibration pattern device, features are extracted from the image (see, e.g., block 514), and the position of the extracted features from the current image is determined (see, e.g., block 516). The position information derived from the extracted features may be referred to as the current camera pose.

[0046] It should be appreciated that the control law module 506 may transform the desired movements in the camera coordinate frame into desired movements in the robot coordinate frame. Such a transformation may be obtained using a camera-to-robot base transformation, as may be determined during an initial calibration setup step (e.g., the first phase in a two-phase calibration approach). The robot controller 508 transforms the desired movements in the robot coordinate frame into movements of the joints in the robot system. The robot 510 then receives the commands and accomplishes the movements of the robot system according to the robot commands determined by the control law module 506.

[0047] The feedback loop of elements 506, 508, 510, 512, 514, and 516 allows the auto-calibration system to, for example, determine the target camera pose and B t This is repeated until the auto-calibration system determines that the current camera pose and the target camera pose are sufficiently similar (e.g., the target feature positions match the current feature positions), as may be determined by a comparison of the difference in the positions of the image features against a threshold amount. Once the auto-calibration system determines that the target pose has been reached, it then uses the corresponding position sensor data from position sensor 513 while in the target camera pose (e.g., A i ) may be recorded. t In some embodiments, the automatic calibration system includes t The target camera pose may be recorded in a data representation different from the data representation used to represent the target feature positions. For example, in some embodiments, the 6DOF position of the camera in the camera coordinate frame is recorded as B t It is determined from the image data when is achieved. Thus, for a set of camera poses, the system records the corresponding position sensor data (from position sensor 513) for those camera poses.

[0048] FIG. 6 is a block diagram illustrating a pose-based visual servoing calibration approach, in accordance with an example embodiment.

[0049] In block 602, the visual servoing-based calibration module selects a target camera pose from a set of camera poses (e.g.,

[0050]

number

[0051] The control law module 606 determines the target camera pose B t(e.g., a target pose in 6DOF coordinates in the camera coordinate frame) and the current camera pose, B c Calculate the robot command to achieve the robot movement, B t B. Reducing the difference between the target camera pose and the current camera pose c As shown in Figure 6, the current camera pose B c is provided as part of the feedback path where the camera sensor 612 captures an image of the calibration pattern device and the current camera pose B c (e.g., 616) are estimated from the images (see, e.g., block 614) and sent to the control law module 606. In some embodiments, block 614 estimates the camera pose based on, e.g., a perspective-n-point algorithm. The input to the perspective-n-point algorithm can be the object point positions defined on the pattern and their corresponding positions detected on the image, along with camera intrinsic parameters (obtained from a camera calibration step (not shown)).

[0052] It should be appreciated that the control law module 606 may transform the desired movement in the camera coordinate frame into a desired movement in the robot coordinate frame. Such a transformation may be obtained using a camera-to-robot base transformation, as may be determined during an initial calibration setup step (e.g., a first phase in a two-phase calibration approach). The robot controller 608 transforms the desired movement in the robot coordinate frame into movements of the joints in the robot system. The robot 610 then receives the commands and accomplishes the movements of the robot system according to the robot commands determined by the control law module 606.

[0053] The feedback loop of elements 606, 608, 610, 612, 614, and 616 allows the auto-calibration system to, for example, determine the target camera pose and B tThis is repeated until it is determined that the current camera pose and the target camera pose are sufficiently similar (e.g., the target feature positions match the current feature positions), as may be determined by a comparison of the difference in the positions of the image features against a threshold amount. Once the auto-calibration system determines that the target pose has been reached, it then uses the corresponding position sensor data (e.g., A) at the target camera pose to i ) may be recorded. t In some embodiments, the automatic calibration system includes t The target camera pose may be recorded in a data representation different from the data representation used to represent the target feature positions. For example, in some embodiments, the 6DOF position of the camera in the camera coordinate frame is recorded as B t It is determined from the image data when is achieved. Thus, for a set of camera poses, the system records the corresponding position sensor data (from position sensor 613) for those poses.

[0054] ii. Position Sensor Based Servo Control Approach In position sensor servo control based calibration, the automatic calibration system automatically detects the position sensor attitude (e.g., A i ) is reached, the corresponding camera pose (e.g., B i ) to record a set of position sensor poses (

[0055]

number

[0056] FIG. 7 is a block diagram illustrating a method 700 for automatically calibrating an endoscope, such that a transformation between the coordinate frames of a position sensor and a camera is determined. As FIG. 7 illustrates, the method 700 may include a first phase 702. During the first phase 702, the automatic calibration system may generate a position sensor-to-robot base transformation. In embodiments where the position sensor is an EM sensor, the position sensor-to-robot base transformation may be an electromagnetic field generator-to-robot base transformation. The first phase 702 may involve several sub-blocks that collectively represent the first phase 702 of a multi-phase approach. For example, the first phase 702 may include sub-blocks 702a, 702b, and 702c. In sub-block 702a, the automatic calibration system may command the robot a first movement between the endoscope and a calibration pattern device to achieve one or more robot poses in the robot coordinate frame. As used herein, a set of one or more robot poses may be commanded and expressed using coordinates of the robot coordinate frame. For example, the robot controller may issue commands to the robot arm to move the endoscope to a given position in the robot coordinate frame (e.g., to a given x, y, z position and / or yaw, pitch, roll orientation of the robot coordinate frame). Similarly, the robot controller may issue commands to the robot arm to move the calibration pattern device to a given position in the robot coordinate frame (e.g., to a given x, y, z position and / or yaw, pitch, roll orientation of the robot coordinate frame). In some embodiments, the robot pose may be selected such that the position sensor of the endoscope is entirely within the operating portion of the EM field generated by the EM field generator (which may be part of the calibration pattern device 120 of FIG. 1, as described above).

[0057] In sub-block 702b, the auto-calibration system may record the robot posture data and the corresponding first position sensor posture data at each of the one or more robot postures achieved in sub-block 702a. As described above, the position sensor posture data may be data operable to identify a posture (or at least some aspects thereof) of a position sensor within a position sensor coordinate frame. For example, in some embodiments, the auto-calibration system may derive the position sensor posture data from position sensor readings taken at one of the robot postures of block 702a.

[0058] Thus, the auto-calibration system may store n pairs of robot pose data and position sensor pose data, where n is the number of robot poses achieved in sub-block 702a. The robot pose data stored in sub-block 702b may be the robot pose of the command sent to the robot controller. In other embodiments, there may be some sensor feedback that estimates the pose of the robot system, such as an external camera, an EM sensor embedded in the robot, etc.

[0059] The auto-calibration system calculates a position sensor-to-robot base transformation in subblock 702c using the pair of robot pose data (or a transform thereof) and corresponding position sensor pose data (or a transform thereof) as recorded in subblock 702b (e.g., the first position sensor pose data). Generating the position sensor-to-robot base transformation may include collecting paired measurements of the electromagnetic field generator-EM transformation and the robot transformation pair, and solving: RX RL =X RL A

[0060] In this calculation, R represents the relative transformation between two robot poses (e.g., R1, R2), and R = R2_inv * R1. A represents the relative transformation of two position sensor poses (A1, A2), and A = A2_inv *A1 can be defined as the transformation from the robot base to the end effector. As mentioned above, the robot pose can be expressed as a transformation from the robot base to the end effector (as can be determined using the forward kinematics of the robot). The position sensor pose can be a transformation from the measured positions (e.g., position sensor-field generator). Thus, R1 can be the transformation from the robot base to the end effector for the first pose. A1 can be the transformation from the position sensor for the first pose. R2 can be the transformation from the robot base to the end effector for the second pose. A2 can be the transformation from the position sensor to the second pose pattern for the second pose. (R1,A1) and (R2,A2) are pairs that are meant to be synchronized. There are also probabilistic methods that use batches of data that do not require the R,A pairs to be synchronized.

[0061] During the second phase 704, the auto-calibration system may generate a position sensor-to-camera transformation. Block 704 may involve several sub-blocks, such as, for example, sub-blocks 704a, 704b, and 704c. In sub-block 704a, the auto-calibration system may use the position sensor-to-robot-based transformation generated during the first phase 702 to robotically command a second movement between the endoscope and the calibration pattern device to achieve one or more position sensor poses in the position sensor coordinate frame. As previously mentioned, the position sensor poses may be expressed using coordinates of the position sensor coordinate frame. For example, the set of one or more position sensor poses of block 704a may be expressed as readings from a position sensor.

[0062] It should be appreciated that sub-block 704a may operate by commanding robot motion between the calibration pattern device and the endoscope such that a determinable position sensor pose in the position sensor coordinate frame is achieved. However, the position sensor coordinate frame is a different coordinate frame than the robot coordinate frame. To account for this difference, for example, the auto-calibration system may determine that the position sensor needs to move 50° to the left in the position sensor coordinate frame to achieve a desired position sensor pose. If the auto-calibration system commands the robot controller to move 50° to the left, this movement occurs in the robot coordinate frame, and this movement may result in the position sensor moving to the left at other angles in the position sensor coordinate frame. Thus, in normal operation, the movement required to move the current position sensor pose to the desired position sensor pose does not necessarily translate as the same movement in the robot coordinate frame. The position sensor-to-robot based transformation facilitates this operation in that the auto-calibration system can map movements in the position sensor coordinate frame to movements in the robot coordinate frame. This approach is referred to as a position sensor servo control approach and is described in more detail below.

[0063] In sub-block 704b, the auto-calibration system may record second position sensor pose data and corresponding camera pose data at each of the one or more position sensor poses. As explained above, the camera pose data may be data operable to identify the pose (or at least some aspects thereof) of the camera in the camera coordinate frame. Thus, the auto-calibration system may store camera pose data and position sensor pose data for n pairs, where n is the number of position sensor poses arrived at in sub-block 704a.

[0064] In subblock 704c, the auto-calibration system calculates a camera-to-position sensor transformation using the pair of camera pose data (or a transform thereof) and the corresponding position sensor pose data (or a transform thereof) as recorded in subblock 704b. Generating the camera-to-position sensor transformation may include collecting paired measurements of the electromagnetic field generator-to-EM transformation and the pattern-to-camera transformation pairs, and solving: AX C2LS =X C2LS B

[0065] where A represents the relative transformation between position sensor poses of the second phase 704 (e.g., from position sensor to field generator if the position sensor is an EM sensor), B represents the relative transformation between pattern poses of the second phase 704, and X C2LS represents the camera-position sensor transformation.

[0066] At block 706, the automatic calibration system stores data indicative of the camera-to-position sensor transformation in a computer-readable medium of the endoscope, such as the calibration data store 116 of Figure 1. As discussed above with respect to Figure 1, the calibration data store may be an RFID tag, but may be any suitable device capable of storing data indicative of the camera-to-position sensor transformation and communicating the camera-to-position sensor transformation to the robotic system when the endoscope is coupled to the robotic system.

[0067] FIG. 8 is a block diagram illustrating an exemplary embodiment of an auto-calibration system 800 implementing a method for position sensor servo control based calibration. As FIG. 8 illustrates, the auto-calibration system 800 operates to execute multiple phases, namely a first phase 802 and a second phase 804, where the first phase 802 operates to generate a position sensor-to-robot base transformation 846 and the second phase 804 operates to generate a position sensor-to-camera transformation 848. In the first phase 802, a set of robot poses 812 in the robot coordinate frame can be empirically pre-determined to obtain the position sensor-to-robot base transformation 846. It is worth noting that depending on the type of position sensor utilized by the auto-calibration system, it may be beneficial to select a robot pose that allows the position sensor to generate pose data indicative of the sensor's pose. For example, in an embodiment utilizing an EM sensor, the robot pose results in the EM sensor being located within the detectable region of an electromagnetic field generator. The robot controller 814 issues commands to the robot 816 to move the endoscope to a given position in the robot coordinate frame (e.g., to a given x, y, z position and / or yaw, pitch, roll orientation in the robot coordinate frame).

[0068] After executing each robot pose 812, the corresponding robot pose and position sensor pose data are recorded. For example, after reaching a robot pose, the robot pose data associated with the robot pose (e.g., position and / or orientation in 6 DOF) is stored along with the position sensor pose data acquired at the robot pose. Using the acquired pairs of robot pose data and position sensor data, a position sensor-to-robot base transform 846 is calculated.

[0069] In a second phase 804, the auto-calibration system performs a position sensor servo control based calibration using the position sensor-to-robot based transformation 846 generated in the first phase 802 to generate a position sensor-to-camera transformation 848. In the camera servo control based calibration, the auto-calibration system may command the robot movement to perform a 6-DOF movement relative to the position sensor such that the auto-calibration system can record pairs of position sensor pose data and camera pose data. For example, in block 820, the auto-calibration system selects a target position sensor pose from a set of position sensor poses (e.g.,

[0070]

number

[0071] The control law module 822 also receives a current position sensor pose 828 (for ease of explanation, A ) from a feedback path where the current position sensor pose is determined (e.g., in block 826) from the position sensor 818. c is called).

[0072] For example, the current pose A, as may be determined by a comparison of the difference between the current pose and the target pose c and target posture A t It should be appreciated that the feedback loop of elements 822, 814, 816, 818, 826, and 828 is repeated until the auto-calibration system determines that the and are sufficiently similar (e.g., the target pose is reached).

[0073] When the auto-calibration system determines that the target position sensor attitude has been reached, the auto-calibration system may then record the image attitude data (e.g., B i). The next target pose is selected and the process is repeated to record corresponding image pose data until a sufficient number of target position sensor poses have been reached and corresponding camera pose data recorded.

[0074] Thus, for a set of position sensor poses, the system records the corresponding image sensor poses. A position sensor-to-camera transformation is then generated using a method similar to that described above with reference to FIG.

[0075] Mounting system and terminology The embodiments disclosed herein provide systems, methods, and apparatus for calibration of a robotically or user-controlled medical instrument. Various implementations described herein provide improved calibration between a camera and a position sensor embedded within or otherwise configured to be coupled to the medical instrument.

[0076] Automated calibration system 100 may include various other components. For example, automated calibration system 100 may include one or more control electronics / circuitry, power sources, pneumatics, light sources, actuators (e.g., motors for moving a robotic arm), memory, and / or a communication interface (e.g., for communicating with another device). In some embodiments, the memory may store computer-executable instructions that, when executed by the control circuitry, cause the control circuitry to perform any of the operations discussed herein. For example, the memory may store computer-executable instructions that, when executed by the control circuitry, cause the control circuitry to receive inputs and / or control signals related to the operation of the robotic arm and, in response, control the robotic arm to position it in a particular arrangement.

[0077] The various components of the automatic calibration system 100 may be electrically and / or communicatively coupled using certain connection circuits / devices / features, which may or may not be part of the control circuitry. For example, the connection feature(s) may include one or more printed circuit boards configured to facilitate mounting and / or interconnection of at least some of the various components / circuitry of the automatic calibration system 100. In some embodiments, two or more of the control circuitry, data storage / memory, communication interface, power supply unit(s), and / or input / output (I / O) component(s) may be electrically and / or communicatively coupled to one another.

[0078] The term "control circuitry" is used herein according to its broad and ordinary meaning and may refer to any collection of the following: one or more processors, processing circuits, processing modules / units, chips, dies (e.g., semiconductor dies including one or more active and / or passive devices and / or connection circuits), microprocessors, microcontrollers, digital signal processors, microcomputers, central processing units, graphics processing units, field programmable gate arrays, programmable logic circuits, state machines (e.g., hardware state machines), logic circuits, analog circuits, digital circuits, and / or any device that manipulates signals (analog and / or digital) based on hard-coding of circuit and / or operational instructions. The control circuitry may further include one or more storage devices, which may be embodied in a single memory device, multiple memory devices, and / or embedded circuits of the device. Such data storage devices may include read-only memory, random access memory, volatile memory, non-volatile memory, static memory, dynamic memory, flash memory, cache memory, data storage registers, and / or any device that stores digital information. It should be noted that in embodiments where the control circuitry includes a hardware state machine (and / or implements a software state machine) and includes analog, digital, and / or logic circuitry, the data storage device(s) / register(s) storing any associated operational instructions may be embedded within or external to the circuitry including the state machine, analog, digital, and / or logic circuitry.

[0079] The term "memory" is used herein according to its broad and ordinary meaning and may refer to any suitable or desired type of computer-readable medium, including, for example, one or more volatile, non-volatile, removable, and / or non-removable data storage devices implemented using any suitable or desired technology, layout, and / or data structures / protocols that contain any suitable or desired computer-readable instructions, data structures, program modules, or other types of data.

[0080] Computer-readable media that may be implemented according to embodiments of the present disclosure may include, but are not limited to, phase-change memory, static random-access memory (SRAM), dynamic random-access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage device, magnetic cassette, magnetic tape, magnetic disk storage device or other magnetic storage device, or any other non-transitory medium that may be used to store information for access by a computing device. As used in certain contexts herein, computer-readable media may generally not include communication media such as modulated data signals and carrier waves. Thus, computer-readable media should generally be understood to refer to non-transitory media.

[0081] Further embodiments Depending on the embodiment, certain acts, events, or functions of any of the algorithms or processes described herein may be performed in a different order, added, merged, or omitted entirely, and thus, in a particular embodiment, not all of the described acts or events are necessary to the execution of a process.

[0082] In particular, conditional language used herein, such as "can," "could," "might," "may," "eg," and the like, unless specifically stated otherwise or understood otherwise within the context in which it is used, is intended to have its ordinary meaning and is generally intended to convey that certain embodiments include certain features, elements, and / or steps, while other embodiments do not. Thus, such conditional language is not intended to imply that features, elements, and / or steps are generally required in any way for one or more embodiments, or that one or more embodiments necessarily include logic for determining whether those features, elements, and / or steps are included or performed in any particular embodiment, with or without author input or prompting. Terms such as "comprising," "including," "having," and the like, are synonymous and used in their ordinary sense and are used inclusively in a non-limiting manner and do not exclude additional elements, features, acts, operations, etc. Also, when the term "or" is used, for example, to connect a list of elements, the term "or" is used in its inclusive sense (and not its exclusive sense) to mean one, some, or all of the listed elements. Unless specifically stated otherwise, connective language such as the phrase "at least one of X, Y, and Z" is understood in the context as it is commonly used to convey that an item, term, element, etc. can be either X, Y, or Z. Thus, such connective language is not generally intended to imply that a particular embodiment requires that at least one of X, at least one of Y, and at least one of Z, respectively, be present.

[0083] In the above description of the embodiments, it should be understood that various features are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects. However, the method of the disclosure should not be interpreted as reflecting an intention that any claim requires more features than are expressly recited in that claim. Moreover, any component, feature, or step illustrated and / or described in a particular embodiment herein may be applied to or used in conjunction with any other embodiment. Moreover, no component, feature, step, or group of components, features, or steps is necessary or essential for each embodiment. Thus, it is intended that the scope of the present disclosure should not be limited by the particular embodiments described above, but should be determined solely by a fair reading of the following claims.

[0084] It should be understood that certain ordinal terms (e.g., "first" or "second") may be provided for ease of reference and do not necessarily imply physical characteristics or ordering. Thus, as used herein, ordinal terms (e.g., "first," "second," "third," etc.) used to modify an element, such as a structure, component, operation, etc., do not necessarily indicate a priority or order of the element with respect to any other elements, but rather may generally distinguish the element from another element having a similar or identical name (apart from the use of the ordinal term). In addition, as used herein, the indefinite articles ("a" and "an") may indicate "one or more" rather than "one." Furthermore, an operation performed "based on" a condition or event may also be performed based on one or more other conditions or events not expressly recited.

[0085] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the example embodiments belong. It is further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning consistent with their meaning in the context of the relevant art, and should not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0086] Spatially relative terms such as "outer", "inner", "upper", "lower", "below", "upper", "vertical", "horizontal", and similar terms may be used herein for ease of description to describe the relationship between one element or component and another element or component as illustrated in the figures. It should be understood that the spatially relative terms are intended to encompass different orientations of the device in use or operation in addition to the orientation depicted in the figures. For example, if the device shown in the figures is inverted, a device positioned "below" or "under" another device may be placed "above" the other device. Thus, the illustrative term "lower" may include both lower and upper positions. The device may also be oriented in other directions, and thus the spatially relative terms may be interpreted differently depending on the orientation.

[0087] Unless otherwise specified, comparative and / or quantitative terms such as "less," "more," "greater than," and the like are intended to encompass the notion of equality. For example, "less" can mean not only "less than" in the strict mathematical sense, but also "less than or equal to."

[0088] [Embodiment] (1) A method for automating calibration of an endoscope of a robotic medical system, comprising: robotically commanding a first movement between the endoscope and a calibration pattern to achieve one or more robot poses in a robot coordinate frame; recording robot pose data and corresponding first camera pose data at each robot pose from the one or more robot poses; calculating a camera-to-robot base transformation based on the robot pose data recorded at each robot pose and the corresponding first camera pose data; robotically commanding a second movement between the endoscope and the calibration pattern to achieve one or more camera poses in a camera coordinate frame using the camera-to-robot based transformation; recording second camera pose data and corresponding position sensor pose data at each camera pose from the one or more camera poses; calculating a position sensor-to-camera transformation based on the second camera pose data and the corresponding position sensor pose data recorded at each of the camera poses; storing data indicative of the position sensor-to-camera transformation in a computer readable medium configured to be coupled to the endoscope; A method comprising: (2) The method of embodiment 1, wherein robotically commanding the first movement between the endoscope and the calibration pattern includes commanding robotic movement of the calibration pattern. (3) The method of embodiment 1, wherein robotically commanding the first movement between the endoscope and the calibration pattern includes commanding robotic movement of the endoscope. (4) The method of embodiment 1, wherein the endoscope includes a position sensor, the position sensor including at least one of an electromagnetic sensor, a shape sensing fiber, an accelerometer, a gyroscope, and a magnetometer. (5) The method of embodiment 1, wherein the computer-readable medium includes a radio frequency identification (RFID) tag configured to transmit the position sensor-to-camera transformation to an RFID reader disposed on the robotic medical system.

[0089] (6) The method of claim 1, wherein the one or more camera poses each include a desired feature position corresponding to the calibration pattern. (7) The method of embodiment 1, wherein the one or more camera poses each include a derived position and orientation determinable from an object point position in an image and camera intrinsic parameters. (8) robotically commanding the second movement between the endoscope and the calibration pattern to achieve the one or more camera poses in the camera coordinate frame includes: estimating a current camera pose of the endoscope in the camera coordinate frame; determining a first motion for moving the endoscope from the current camera pose toward a first camera pose, the first motion being within the camera coordinate frame; determining a second movement corresponding to the first movement using the camera-to-robot base transformation, the second movement being in the robot coordinate frame; and 2. The method of claim 1, further comprising robotically commanding the endoscope to the one or more camera poses based on the second movement. (9) robotically commanding the second movement between the endoscope and the calibration pattern to achieve the one or more camera poses in the camera coordinate frame includes: estimating a current camera pose of the endoscope in the camera coordinate frame; determining a first motion for moving the calibration pattern from the current camera pose toward a first camera pose, the first motion being within the camera coordinate frame; determining a second movement corresponding to the first movement using the camera-to-robot base transformation, the second movement being in the robot coordinate frame; and 2. The method of claim 1, further comprising: robotically commanding the one or more camera poses to the calibration pattern based on the second movement. (10) A system for automating calibration of an endoscope, comprising: a robotic arm configured to facilitate movement between the endoscope and a calibration pattern; A control circuit; and a memory storing computer-executable instructions that, when executed, cause the control circuitry to: robotically commanding the robot arm to facilitate a first movement between the endoscope and the calibration pattern to achieve one or more robot poses in a robot coordinate frame; recording robot pose data and corresponding first camera pose data at each robot pose from the one or more robot poses; calculating a camera-to-robot base transformation based on the robot pose data recorded at each robot pose and the corresponding first camera pose data; robotically commanding the robot arm to facilitate a second movement between the endoscope and the calibration pattern to achieve one or more camera poses in a camera coordinate frame using the camera-to-robot based transformation; recording second camera pose data and corresponding position sensor pose data at each camera pose from the one or more camera poses; calculating a position sensor-to-camera transformation based on the second camera pose data recorded at each of the camera poses and the corresponding position sensor pose data; The system stores data indicative of the position sensor-to-camera transformation in a computer readable medium configured to be coupled to the endoscope.

[0090] (11) A method of calibrating an endoscope configured to be coupled to a robotic medical system, the method comprising: adjusting a relative position between the endoscope and the calibration pattern such that the endoscope and the calibration pattern are disposed according to a first determinable configuration; While in said first determinable configuration, recording first position data from a position sensor coupled to the endoscope; recording first camera data from a camera coupled to the endoscope; adjusting the relative position between the endoscope and the calibration pattern such that the endoscope and the calibration pattern are disposed according to a second determinable configuration that is different from the first determinable configuration; While in said second determinable configuration, recording second position data from the position sensor coupled to the endoscope; recording second camera data from the camera coupled to the endoscope; generating a position sensor-to-camera transformation based on the first position data, the first camera data, the second position data, and the second camera data; storing data indicative of the position sensor-to-camera transformation in a computer readable medium configured to be coupled to the endoscope; A method comprising: (12) Adjusting the relative position between the endoscope and the calibration pattern such that the endoscope and the calibration pattern are disposed according to the first determinable configuration includes: 12. The method of claim 11, comprising moving the calibration pattern relative to the endoscope. (13) Moving the calibration pattern relative to the endoscope includes: 13. The method of claim 12, comprising manual adjustment of the positioning of the calibration pattern. (14) Moving the calibration pattern relative to the endoscope includes: 13. The method of claim 12, comprising robotic adjustment of the positioning of the calibration pattern. (15) Adjusting the relative position between the endoscope and the calibration pattern such that the endoscope and the calibration pattern are disposed according to the first determinable configuration includes: 12. The method of claim 11, comprising moving the endoscope relative to the calibration pattern.

[0091] (16) Moving the endoscope relative to the calibration pattern includes: 16. The method of claim 15, further comprising manual adjustment of the positioning of the endoscope. (17) Moving the endoscope relative to the calibration pattern includes: 16. The method of embodiment 15, comprising robotic adjustment of the positioning of the endoscope. (18) The method of embodiment 11, wherein the position sensor includes at least one of an electromagnetic sensor, a shape sensing fiber, an accelerometer, a gyroscope, and a magnetometer. (19) The method of embodiment 11, wherein the computer-readable medium includes a radio frequency identification (RFID) tag configured to transmit the position sensor-to-camera transformation to an RFID reader disposed on the robotic medical system. (20) The method of embodiment 11, wherein the position sensor includes an electromagnetic sensor configured to generate the first position data based on positioning relative to an electromagnetic field generator, and the calibration pattern is disposed on the electromagnetic field generator.

[0092] (21) The method of claim 20, wherein movement of the electromagnetic field generator causes a change in (a) a relative arrangement between the position sensor and the electromagnetic field generator, and (b) a relative arrangement between the camera and the calibration pattern. (22) A method for automating calibration of an endoscope of a robotic medical system, comprising: robotically commanding a first movement between the endoscope and a calibration pattern to achieve one or more robot poses in a robot coordinate frame; recording robot pose data and corresponding first position sensor pose data at each robot pose from the one or more robot poses; calculating a position sensor-to-robot base transformation based on the robot pose data recorded at each robot pose and the corresponding first position sensor pose data; robotically commanding a second movement between the endoscope and the calibration pattern to achieve one or more position sensor poses in a position sensor coordinate frame using the position sensor-to-robot based transformation; recording second position sensor orientation data and corresponding camera orientation data at each position sensor orientation from the one or more position sensor orientations; calculating a position sensor-to-camera transformation based on the second position sensor pose data and the corresponding camera pose data recorded at each position sensor pose; storing data indicative of the position sensor-to-camera transformation in a computer readable medium configured to be coupled to the endoscope; A method comprising: (23) A system for automating calibration of an endoscope, comprising: a robotic arm configured to facilitate movement between the endoscope and a calibration pattern; A control circuit; and a memory storing computer-executable instructions that, when executed, cause the control circuitry to: robotically commanding the robot arm to facilitate a first movement between the endoscope and the calibration pattern to achieve one or more robot poses in a robot coordinate frame; recording robot pose data and corresponding first position sensor pose data at each robot pose from the one or more robot poses; calculating a position sensor-to-robot base transformation based on the robot pose data recorded at each robot pose and the corresponding first position sensor pose data; robotically commanding the robot arm to facilitate a second movement between the endoscope and the calibration pattern to achieve one or more position sensor poses in a position sensor coordinate frame using the position sensor-to-robot based transformation; recording second position sensor orientation data and corresponding camera orientation data at each position sensor orientation from the one or more position sensor orientations; calculating a position sensor-to-camera transformation based on the second position sensor pose data and the corresponding camera pose data recorded at each of the position sensor poses; The system stores data indicative of the position sensor-to-camera transformation in a computer readable medium configured to be coupled to the endoscope.

Claims

1. 1. A method of calibrating a robotic medical system, comprising: moving a robotic arm of the robotic medical system to achieve one or more robot poses in a first coordinate frame, the robotic arm being coupled to a calibration pattern or an endoscope having a camera and a position sensor disposed thereon, and the movement of the robotic arm changing the relative alignment between the camera and the calibration pattern; determining one or more first poses of the camera in a second coordinate frame based on the relative alignment between the camera and the calibration pattern in response to moving the robot arm to achieve the one or more robot poses; calculating a camera-to-robot base transformation between the first coordinate frame and the second coordinate frame based on the one or more robot poses and the one or more first poses of the camera; moving the robot arm based on the camera-to-robot base transformation to achieve one or more second poses of the camera in the second coordinate frame; determining one or more poses of the position sensor in a third coordinate frame based on a relative alignment between the position sensor and the calibration pattern in response to moving the robot arm to achieve the one or more second poses of the camera, respectively; calculating a position sensor-to-camera transformation between the second coordinate frame and the third coordinate frame based on the one or more second poses of the camera and the one or more poses of the position sensor; driving the endoscope based at least in part on the position sensor-to-camera transformation; A method comprising:

2. The method of claim 1, wherein the position sensor includes at least one of an electromagnetic sensor, a shape-sensing fiber, an accelerometer, a gyroscope, or a magnetometer.

3. The method of claim 1, further comprising storing the position sensor-to-camera transformation on a radio frequency identification (RFID) tag coupled to the endoscope, the RFID tag configured to transmit the position sensor-to-camera transformation to an RFID reader located on the robotic medical system.

4. 2. The method of claim 1 , wherein the one or more second poses of the camera are each associated with one or more feature positions in the calibration pattern, or with one or more object point positions in an image captured by the camera and one or more intrinsic parameters of the camera.

5. Moving the robot arm to achieve the one or more second poses of the camera includes: estimating a current pose of the camera in the second coordinate frame; determining a desired movement of the camera in the second coordinate frame that transforms the current pose to one of the one or more second poses; and mapping the desired movement of the camera in the second coordinate frame to a movement of the robot arm in the first coordinate frame based on the camera-to-robot base transformation.

6. A system for calibrating an endoscope, comprising: a robotic arm coupled to the endoscope having a calibration pattern or a camera and a position sensor disposed thereon, wherein movement of the robotic arm changes the relative alignment between the camera and the calibration pattern; A control circuit comprising: moving the robot arm to achieve one or more robot poses in a first coordinate frame; determining one or more first poses of the camera in a second coordinate frame based on the relative alignment between the camera and the calibration pattern in response to moving the robot arm to achieve the one or more robot poses; calculating a camera-to-robot base transformation between the first coordinate frame and the second coordinate frame based on the one or more robot poses and the one or more first poses of the camera; moving the robot arm based on the camera-to-robot base transformation to achieve one or more second poses of the camera in the second coordinate frame; determining one or more poses of the position sensor in a third coordinate frame based on a relative alignment between the position sensor and the calibration pattern in response to moving the robot arm to achieve the one or more second poses of the camera, respectively; calculating a position sensor-to-camera transformation between the second coordinate frame and the third coordinate frame based on the one or more second poses of the camera and the one or more poses of the position sensor; and a control circuit configured to drive the endoscope based at least in part on the position sensor-to-camera transformation.

7. 1. A method for calibrating an endoscope associated with a robotic medical system, the method comprising: adjusting a relative alignment between the endoscope and the calibration pattern such that the endoscope and the calibration pattern are positioned according to a first configuration; recording first position data from a position sensor coupled to the endoscope while the endoscope and the calibration pattern are positioned in the first configuration; recording first camera data from a camera coupled to the endoscope while the endoscope and the calibration pattern are positioned in the first configuration; adjusting the relative alignment between the endoscope and the calibration pattern such that the endoscope and the calibration pattern are arranged according to a second configuration different from the first configuration; recording second position data from the position sensor while the endoscope and the calibration pattern are positioned in the second configuration; recording second camera data from the camera while the endoscope and the calibration pattern are positioned in the second configuration; determining a position sensor-to-camera transformation between a coordinate frame associated with the position sensor and a coordinate frame associated with the camera based on the first position data, the first camera data, the second position data, and the second camera data; driving the endoscope based at least in part on the position sensor-to-camera transformation; A method comprising:

8. adjusting the relative alignment between the endoscope and the calibration pattern such that the endoscope and the calibration pattern are positioned according to the first configuration; The method of claim 7 , comprising moving the calibration pattern relative to the endoscope.

9. adjusting the relative alignment between the endoscope and the calibration pattern such that the endoscope and the calibration pattern are positioned according to the first configuration; The method of claim 7 , comprising moving the endoscope relative to the calibration pattern.

10. The calibration pattern includes an electromagnetic field generator, and the position sensor includes an electromagnetic sensor, the electromagnetic sensor configured to generate the first position data based on an attitude of the electromagnetic sensor relative to the electromagnetic field generator; 8. The method of claim 7, wherein adjusting the relative alignment between the endoscope and the calibration pattern changes the orientation of the position sensor relative to the electromagnetic field generator and changes the orientation of the camera relative to the calibration pattern.

Images