Surgical robotic system for control of steerable instrument using multiple imaging modalities

Abstract

A surgical robotic system includes a first robotic arm having a first instrument drive unit and an instrument coupled to and actuatable by the first instrument drive unit, and a surgeon console including a handle controller for receiving a user input to move the instrument in a first coordinate system. The system also includes a second robotic arm having a second instrument drive unit and an imaging device coupled to and actuatable by the second instrument drive unit. The imaging device is configured to capture an image of the instrument using an imaging modality in a second coordinate system, different from the first coordinate system. A tracking system is also provided for obtaining tracking data of the instrument and the imaging device. The surgical robotic system further includes a processor configured to register the instrument in the second coordinate system based on the tracking data, align the first coordinate system with the second coordinate system, and receive inputs controlling the instrument in the second coordinate system.

Description

SURGICAL ROBOTIC SYSTEM FOR CONTROL OF STEERABLE INSTRUMENT USING MULTIPLE IMAGING MODALITIESCROSS-REFERENCE TO RELATED APPLICATION

[0001] This application claims the benefit of U.S. Provisional Patent Application Serial No. 63 / 574,401 , filed April 4, 2024, the entire content of which is incorporated herein by reference.BACKGROUND

[0002] Surgical robotic systems are currently being used in a variety of medical procedures, including minimally invasive surgical procedures. Some surgical robotic systems include a surgeon console controlling a surgical robotic arm and a surgical instrument having an end effector (e.g., forceps or grasping instrument) coupled to and actuated by the robotic arm. In operation, the robotic arm is moved to a position over a patient and then guides the surgical instrument into a small incision via a surgical port or a natural orifice of a patient to position the end effector at a work site within the patient’s body. A variety of different types of instruments are used with surgical robotic systems that are designed to perform specific functions, such as steerable catheters and endoscopes.

[0003] Image guided medical and surgical procedures utilize patient images obtained prior to or during a medical procedure to guide a physician performing the procedure. Such procedures can be referred to as computer-assisted procedures. Recent advances in imaging technology, especially in imaging technologies that produce highly-detailed, two, three, and four dimensional images, such as computed tomography (CT), magnetic resonance imaging (MRI), fluoroscopic imaging (such as with a C-arm device), positron emission tomography (PET), and ultrasound imaging (US) have increased the interest in image guided medical procedures.

[0004] Typical image guided navigation systems generally require a dynamic reference frame to track the position of the patient should patient movement occur during the procedure. The dynamic reference frame is generally affixed to the patient in a generally permanent or immovable fashion. The dynamic reference frame (DRF) may also be used as a fiducial marker and may, therefore, be attached to the patient during the acquisition of pre-operative images. This enables the image space to be aligned with patient space during the navigated procedure.

[0005] Various instruments that are desired to be tracked may be used during an operative procedure. Image data is generally acquired, either intra- operatively or pre-operatively, and the instrument is generally illustrated, and superimposed on the captured image data to identify the position of the instrument relative to the patient space. Therefore, the instrument may include tracking sensors, such as electromagnetic coils or optical detection points, such as LEDs or reflectors that may be detected by a suitable tracking system. Also, the DRF can be used by the tracking system to maintain a registration or localization of the patient space to the image space. The DRF can be also any appropriate tracking sensor that is fixed to a portion of the patient that allows the system to determine whether the patient has moved relative to the image space.

[0006] Flexible surgical instruments, such as catheters, scopes, etc., are used in a variety of procedures in treatment of heart, lungs, and other organs. These instruments are used with the aid of various visualization techniques to aid in navigation of the instruments and to perform the procedure. In certain cases, multiple visualization modalities are used simultaneously, such as ultrasound and fluoroscopy, to guide the instrument. In such situations, the surgeon needs to manually perform mental-registration and compensation of the images and the instrument. In particular, the surgeon needs to understand how each imaging modality relates to the other imaging modalities as well as how the instrument is being imaged in the different imaging modalities, and how movement inputs affect the movement of the instrument. These mental processes put a lot of mental burden as well as waste a significant amount of time during the procedure.SUMMARY

[0007] The present disclosure provides for a surgical robotic system and a navigation system for navigating a surgical instrument using multiple imaging modalities. Suitable imaging modalities include a fluoroscopy imaging system and an ultrasound imaging system. The navigation system includes fiducial markers, electromagnetic trackers, or any other index markers / trackers that are used to localize imaging components, the instrument, the patient, etc. The markers may be tracked using either white or infrared cameras, electromagnetic receivers, and the like.

[0008] The robotic system is configured to combine multiple imaging modalities, such as fluoroscopy and ultrasound. In particular, localization of imagining components allows for combining of fluoroscopy and ultrasound images using the same coordinate system by locatingimage acquisition components (e.g., ultrasound transducer, X-ray sensors, etc.) in space with relation to the camera, receiver, or some other reference point.

[0009] Localization information is then processed using properties of the imagers to enable orientation of the output from both devices on display screen(s) both in rotation as well as scale / zoom while also providing the operators with the potential instructions to align the imaging systems as desired. In embodiments where the imaging systems are controlled and actuated by a robotic system, automatic alignment may also be used.

[0010] The system may also be used to align the robotic visual reference to the images. The robotic system components, e.g., robotic arms, and the image components are aligned to be in the same plane or have a live registration of the robot motion inside the image to detect where the robot coordinate system is within the current imaging output. After that registration occurs, the robot control may be aligned such that each cartesian control command happens with the same axes as the image - e.g., right input moves the instrument to the right, up input moves the instrument up, etc.

[0011] According to one embodiment of the present disclosure, a surgical robotic system is disclosed. The surgical robotic system includes a first robotic arm having a first instrument drive unit and an instrument coupled to and actuatable by the first instrument drive unit, and a surgeon console including a handle controller for receiving a user input to move the instrument in a first coordinate system. The system also includes a second robotic arm having a second instrument drive unit, and an imaging device coupled to and actuatable by the second instrument drive unit. The imaging device is configured to capture an image of the instrument using an imaging modality in a second coordinate system, different from the first coordinate system. A tracking system is also provided for obtaining tracking data of the instrument and the imaging device. The surgical robotic system further includes a processor configured to register the instrument in the second coordinate system based on the tracking data, align the first coordinate system with the second coordinate system, and receive inputs controlling the instrument in the second coordinate system.

[0012] Implementations of the above embodiment may include one or more of the following features. According to one aspect of the above embodiment, the imaging device may be one of an ultrasound probe or a fluoroscope. The instrument may be one of a catheter or a flexible scope. The tracking system may be one of an optical tracking system or an electromagnetic tracking system. The handle controller may include a plurality of directional inputs. The processor may beconfigured to map the plurality of directional inputs from the first coordinate system to the second coordinate system. The surgeon console further may include a display screen for displaying the image captured by the imaging device. The processor is configured to display a first 3D axis symbol corresponding to the first coordinate system and a second 3D axis symbol corresponding to the second coordinate system. The processor may be configured to align the second 3D axis symbol to the first 3D axis symbol when the first coordinate system is aligned with the second coordinate system.

[0013] According to another embodiment of the present disclosure, a method for controlling an imaging system with a surgical robotic system is described. The method includes: receiving a user input at a handle controller for moving an instrument in a first coordinate system, where the instrument is coupled to a first instrument drive unit of a first robotic arm; and capturing an image of the instrument through an imaging device using an imaging modality in a second coordinate system, where the imaging device is coupled to a second instrument drive unit of a second robotic arm; obtaining tracking data of the instrument and the imaging device using a tracking system. The method also includes registering the instrument in the second coordinate system based on the tracking data. The method further includes aligning the first coordinate system with the second coordinate system. The method additionally includes receiving inputs controlling the instrument in the second coordinate system.

[0014] Implementations of the above embodiment may include one or more of the following features. According to one aspect of the above embodiment, the handle controller may include a plurality of directional inputs. The method may further include mapping the plurality of directional inputs from the first coordinate system to the second coordinate system. The method may further include displaying the image captured by the imaging device on a display screen. The method may further include displaying a first 3D axis symbol corresponding to the first coordinate system and a second 3D axis symbol corresponding to the second coordinate system. The method may also include aligning the second 3D axis symbol to the first 3D axis symbol when the first coordinate system is aligned with the second coordinate system.

[0015] According to a further embodiment of the present disclosure, a method for controlling a medical instrument in a plurality of different imaging modalities is described. The method includes capturing a first image of an instrument through a first imaging device using a first imaging modality in a first coordinate system, where the first imaging device is coupled to a first instrumentdrive unit of a first robotic arm. The method also includes capturing a second image of the instrument through a second imaging device using a second imaging modality in a second coordinate system, where the second imaging device is coupled to a second instrument drive unit of a second robotic arm and the second imaging modality is different from the first imaging modality and the second coordinate system is different from the first coordinate system. The method also includes obtaining tracking data of the instrument, the first imaging device, and the second imaging device using a tracking system. The method further includes registering the instrument in the first coordinate system and the second coordinate system based on the tracking data. The method additionally includes aligning the first coordinate system with the second coordinate system.

[0016] Implementations of the above embodiment may include one or more of the following features. According to one aspect of the above embodiment, the method may further include displaying the first image captured by the first imaging device side-by-side with the second image captured by the second imaging device. The method may further include displaying a first 3D axis symbol corresponding to the first coordinate system and a second 3D axis symbol corresponding to the second coordinate system. The method may also include aligning the second 3D axis symbol to the first 3D axis symbol when the first coordinate system of the first imaging device is aligned with the second coordinate system of the second imaging device. Aligning the first coordinate system with the second coordinate system may include moving at least one of the first robotic arm or the second robotic arm.BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Various embodiments of the present disclosure are described herein with reference to the drawings, wherein:

[0018] FIG. 1 is a perspective view of a surgical robotic system including a control tower, a console, and one or more surgical robotic arms, each disposed on a movable cart according to an embodiment of the present disclosure;

[0019] FIG. 2 is a perspective view of a surgical robotic arm of the surgical robotic system of FIG. 1 according to an embodiment of the present disclosure;

[0020] FIG. 3 is a perspective view of a movable cart having a setup arm with the surgical robotic arm of the surgical robotic system of FIG. 1 according to an embodiment of the present disclosure;

[0021] FIG. 4 is a schematic diagram of a computer architecture of the surgical robotic system of FIG. 1 according to an embodiment of the present disclosure;

[0022] FIG. 5 is a navigation system for use with the surgical robotic system of FIG. 1 according to an embodiment of the present disclosure;

[0023] FIG. 6 is a perspective view of a handle controller according to one embodiment of the present disclosure;

[0024] FIG. 7 is a view of a workstation including a first display screen showing an image of a first imaging modality and a second display screen showing an image of a second imaging modality according to an embodiment of the present disclosure;

[0025] FIG. 8 is a view of graphical user interface (GUI) showing an image of the first imaging modality and coordinate system icons representing registration according to an embodiment of the present disclosure;

[0026] FIG. 9 is a flow chart of a method for controlling a steerable instrument using a single imaging modality according to an embodiment of the present disclosure; and

[0027] FIG. 10 is a flow chart of a method for controlling a steerable instrument using multiple imaging modalities according to an embodiment of the present disclosure.DETAILED DESCRIPTION

[0028] Embodiments of the presently disclosed surgical robotic system are described in detail with reference to the drawings, in which like reference numerals designate identical or corresponding elements in each of the several views.

[0029] With reference to FIG. 1, a surgical robotic system 10 includes a control tower 20, which is communicatively coupled to all of the components of the surgical robotic system 10 including a surgeon console 30 and one or more movable carts 60. Each of the movable carts 60 includes a robotic arm 40 having a surgical instrument 50 coupled thereto. The robotic arms 40 also couple to the movable carts 60. The robotic system 10 may include any number of movable carts 60 and / or robotic arms 40.

[0030] The surgical instrument 50 is configured for use during minimally invasive surgical procedures. In embodiments, the surgical instrument 50 may be configured for open surgical procedures. One of the robotic arms 40 may include an endoscopic camera 51 configured to capture video of the surgical site. The endoscopic camera 51 may be a stereoscopic endoscopeconfigured to capture two side-by-side (i.e., left and right) images of the surgical site to produce a video stream of the surgical scene. The endoscopic camera 51 is coupled to a video processing device 56, which may be disposed within the control tower 20. The video processing device 56 may be any computing device as described below configured to receive the video feed from the endoscopic camera 51 and output the processed video stream.

[0031] The surgeon console 30 includes a first display 32, which displays a video feed of the surgical site provided by a camera 51 disposed on the robotic arm 40, and a second display 34, which displays a user interface for controlling the surgical robotic system 10. The first display 32 and the second display 34 may be touchscreens allowing for displaying various graphical user inputs selectable or movable by the user.

[0032] The surgeon console 30 also includes a plurality of user interface devices, such as foot pedals 36 and a pair of handle controllers 38a and 38b, which are used by a user to remotely control the robotic arms 40. The surgeon console further includes an armrest 33 used to support clinician’ s arms while the clinician is operating the handle controllers 38a and 38b.

[0033] The control tower 20 can also include a display 23, which may be a touchscreen and may display one or more graphical user interface(s) (GUIs). The control tower 20 also acts as an interface between the surgeon console 30 and one or more of the robotic arms 40. In particular, the control tower 20 is configured to control the robotic arms 40, such as to move the robotic arms 40 and the corresponding surgical instrument 50, based on a set of programmable instructions and / or input commands from the surgeon console 30. In response to the instructions and / or input, the robotic arms 40 and the surgical instrument 50 execute a desired movement sequence in response to input from the foot pedals 36 and the handle controllers 38a and 38b. The system 10 can be configured so that the foot pedals 36 may be used to affect one or more of a wide variety of system functions, such as to enable and lock the hand controllers 38a and 38b, reposition camera movement, and activate / deactivate an electrosurgical instrument. In particular, the foot pedals 36 may be used to perform a clutching action on the hand controllers 38a and 38b. Clutching is initiated by pressing one of the foot pedals 36, which disconnects (i.e., prevents movement inputs from) the hand controllers 38a and / or 38b such that the robotic arm 40 and corresponding instrument 50 or camera 51 are not actuated. This allows the user to reposition the hand controllers 38a and 38b without moving the robotic arm(s) 40 and the instrument 50 and / or camera 51. This is useful when reaching control boundaries of the surgical space, for instance.

[0034] Each of the control tower 20, the surgeon console 30, and the robotic arm 40 includes a respective computer 21, 31, 41. The computers 21, 31, 41 are interconnected to each other using any suitable communication network based on wired or wireless communication protocols. The term “network,” whether plural or singular, as used herein, denotes a data network, including, but not limited to, the Internet, Intranet, a wide area network, or a local area network. Suitable protocols include, but are not limited to, transmission control protocol / internet protocol (TCP / IP), datagram protocol / internet protocol (UDP / IP), and / or datagram congestion control protocol (DCCP). Wireless communication may be achieved via one or more wireless configurations, e.g., radio frequency (RF), optical, Wi-Fi, Bluetooth (an open wireless protocol for exchanging data over short distances, using short-length radio waves, from fixed and mobile devices, creating personal area networks (PANs), ZigBee® (a specification for a suite of high- level communication protocols using small, low-power digital radios based on the IEEE 122.15.4- 1203 standard for wireless personal area networks (WPANs)).

[0035] The computers 21, 31, 41 may include any suitable processor (not shown) connected operably to a memory (not shown), which may include one or more of volatile, non-volatile, magnetic, optical, or electrical media, such as read-only memory (ROM), random access memory (RAM), electrically erasable programmable ROM (EEPROM), non-volatile RAM (NVRAM), or flash memory. The processor may be any suitable processor (e.g., control circuit) adapted to perform the operations, calculations, and / or set of instructions described in the present disclosure, such as a hardware processor, a field programmable gate array (FPGA), a digital signal processor (DSP), a central processing unit (CPU), a microprocessor, and combinations thereof. Those skilled in the art will appreciate that the processor may be substituted by using any logic processor (e.g., control circuit) adapted to execute algorithms, calculations, and / or set of instructions described herein.

[0036] With reference to FIG. 2, each of the robotic arms 40 may include a plurality of links 42a, 42b, 42c, which are interconnected at joints 44b and 44c, respectively. Other configurations of links and joints may be utilized as known by those skilled in the art. The joint 44a is configured to secure the robotic arm 40 to the movable cart 60 and defines a first longitudinal axis. With reference to FIG. 3, the movable cart 60 includes a lift 67 and a setup arm 61, which provides a base for mounting of the robotic arm 40. The lift 67 allows for vertical movement of the setup arm 61 and, thereby, of the robotic arms 40 mounted on the setup arm 61. The movable cart 60also includes a display 69 for displaying information pertaining to the robotic arm 40. In embodiments, the robotic arms 40 may include any type and / or number of joints.

[0037] With further reference to FIG. 3, the setup arm 61 includes a first link 62a, a second link 62b, and a third link 62c, which provide for lateral maneuverability of the robotic arms 40. The links 62a, 62b, 62c are interconnected at joints 63a and 63b, each of which may include an actuator (not shown) for rotating the links 62b and 62b relative to each other and the link 62c. In particular, the links 62a, 62b, 62c are movable in corresponding lateral planes, which are parallel to each other, thereby allowing for extension of the robotic arm 40 relative to the patient (e.g., surgical table). In embodiments, the robotic arm 40 may be coupled to the surgical table (not shown). The setup arm 61 includes controls 65 for adjusting movement of the links 62a, 62b, 62c as well as the lift 67. In embodiments, the setup arm 61 may include any type and / or number of joints.

[0038] The third link 62c may include a rotatable base 64 having two degrees of freedom. In particular, the rotatable base 64 includes a first actuator 64a and a second actuator 64b. The first actuator 64a is rotatable about a first stationary arm axis, which is perpendicular to a plane defined by the third link 62c. And the second actuator 64b is rotatable about a second stationary arm axis which is transverse to the first stationary arm axis. The first and second actuators 64a and 64b along with the lift 67 allow for full three-dimensional orientation of the robotic arm 40.

[0039] Returning to FIG. 2, the actuator 48b of the joint 44b is coupled to the joint 44c via the belt 45a, and the joint 44c is in turn coupled to the joint 46b via the belt 45b. Joint 44c may include a transfer case coupling the belts 45a and 45b, such that the actuator 48b is configured to rotate each of the links 42b, 42c and a holder 46 relative to each other. More specifically, links 42b, 42c, and the holder 46 are passively coupled to the actuator 48b which enforces rotation about a pivot point “P” which lies at an intersection of the first axis defined by the link 42a and the second axis defined by the holder 46. In other words, the pivot point “P” is a remote center of motion (RCM) for the robotic arm 40. Thus, the actuator 48b controls the angle 0 between the first and second axes allowing for orientation of the surgical instrument 50. Due to the interlinking of the links 42a, 42b, 42c, and the holder 46 via the belts 45a and 45b, the angles between the links 42a, 42b, 42c, and the holder 46 are also adjusted in order to achieve the desired angle 0. In embodiments, some or all of the joints 44a, 44b, 44c may include an actuator to obviate the need for mechanical linkages.

[0040] The joints 44a and 44b include respective actuators 48a and 48b configured to drive the joints 44a, 44b, 44c relative to each other through a series of belts 45a and 45b or other mechanical linkages such as a drive rod, a cable, or a lever and the like. In particular, the actuator 48a is configured to rotate the robotic arm 40 about a longitudinal axis defined by the link 42a.

[0041] With reference to FIG. 2, the holder 46 defines a second longitudinal axis and configured to receive an instrument drive unit (IDU) 52 (FIG. 1). The IDU 52 is configured to couple to an actuation mechanism of the surgical instrument 50 and the camera 51 and is configured to move (e.g., rotate) and actuate the instrument 50 and / or the camera 51. IDU 52 transfers actuation forces from its actuators to the surgical instrument 50 to actuate components of an end effector 49 of the surgical instrument 50. The holder 46 includes a sliding mechanism 46a, which is configured to move the IDU 52 along the second longitudinal axis defined by the holder 46. The holder 46 also includes a joint 46b, which rotates the holder 46 relative to the link 42c. During endoscopic procedures, the instrument 50 may be inserted through an endoscopic access port 55 (FIG. 3) held by the holder 46. The holder 46 also includes a port latch 46c for securing the access port 55 to the holder 46 (FIG. 2).

[0042] The IDU 52 is attached to the holder 46, followed by a sterile interface module (SIM) 43 being attached to a distal portion of the IDU 52. The SIM 43 is configured to secure a sterile drape (not shown) to the IDU 52. The instrument 50 is then attached to the SIM 43. The instrument 50 is then inserted through the access port 55 by moving the IDU 52 along the holder 46. The SIM 43 includes a plurality of drive shafts configured to transmit rotation of individual motors of the IDU 52 to the instrument 50 thereby actuating the instrument 50. In addition, the SIM 43 provides a sterile barrier between the instrument 50 and the other components of the robotic arm 40, including the IDU 52.

[0043] The robotic arm 40 also includes a plurality of manual override buttons 53 (FIG. 1) disposed on the IDU 52 and the setup arm 61, which may be used in a manual mode. The user may press one or more of the buttons 53 to move the component associated with the one or more buttons 53.

[0044] With reference to FIG. 4, each of the computers 21, 31, 41 of the surgical robotic system 10 may include a plurality of controllers, which may be embodied in hardware and / or software. The computer 21 of the control tower 20 includes a controller 21a and safety observer 21b. The controller 21a receives data from the computer 31 of the surgeon console 30 about the currentposition and / or orientation of the handle controllers 38a and 38b and the state of the foot pedals 36 and other buttons. The controller 21a processes these input positions to determine desired drive commands for each joint of the robotic arm 40 and / or the IDU 52 and communicates these to the computer 41 of the robotic arm 40. The controller 21a also receives the actual joint angles measured by encoders of the actuators 48a and 48b and uses this information to determine force feedback commands that are transmitted back to the computer 31 of the surgeon console 30 to provide haptic feedback through the handle controllers 38a and 38b. The safety observer 21b performs validity checks on the data going into and out of the controller 21a and notifies a system fault handler if errors in the data transmission are detected to place the computer 21 and / or the surgical robotic system 10 into a safe state.

[0045] The computer 41 includes a plurality of controllers, namely, a main cart controller 41a, a setup arm controller 41b, a robotic arm controller 41c, and an instrument drive unit (IDU) controller 41 d. The main cart controller 41a receives and processes joint commands from the controller 21a of the computer 21 and communicates them to the setup arm controller 41b, the robotic arm controller 41c, and the IDU controller 41 d. The main cart controller 41a also manages instrument exchanges and the overall state of the movable cart 60, the robotic arm 40, and the IDU 52. The main cart controller 41a also communicates actual joint angles back to the controller 21a.

[0046] Each of joints 63a and 63b and the rotatable base 64 of the setup arm 61 are passive joints (i.e., no actuators are present therein) allowing for manual adjustment thereof by a user. The joints 63a and 63b and the rotatable base 64 include brakes that are disengaged by the user to configure the setup arm 61. The setup arm controller 41b monitors slippage of each of joints 63a and 63b and the rotatable base 64 of the setup arm 61, when brakes are engaged or can be freely moved by the operator when brakes are disengaged, but do not impact controls of other joints. The robotic arm controller 41c controls each joint 44a and 44b of the robotic arm 40 and calculates desired motor torques required for gravity compensation, friction compensation, and closed loop position control of the robotic arm 40. The robotic arm controller 41c calculates a movement command based on the calculated torque. The calculated motor commands are then communicated to one or more of the actuators 48a and 48b in the robotic arm 40. The actual joint positions are then transmitted by the actuators 48a and 48b back to the robotic arm controller 41c.

[0047] The IDU controller 41 d receives desired joint angles for the surgical instrument 50, such as wrist and jaw angles, and computes desired currents for the motors in the IDU 52. The IDUcontroller 41 d calculates actual angles based on the motor positions and transmits the actual angles back to the main cart controller 41a.

[0048] The robotic arm 40 is controlled in response to a pose of the handle controller controlling the robotic arm 40, e.g., the handle controller 38a, which is transformed into a desired pose of the robotic arm 40 through a hand eye transform function executed by the controller 21a. The hand eye function, as well as other functions described herein, is / are embodied in software executable by the controller 21a or any other suitable controller described herein. The pose of one of the handle controllers 38a may be embodied as a coordinate position and roll-pitch-yaw (RPY) orientation relative to a coordinate reference frame, which is fixed to the surgeon console 30. The desired pose of the instrument 50 is relative to a fixed frame on the robotic arm 40. The pose of the handle controller 38a is then scaled by a scaling function executed by the controller 21a. In embodiments, the coordinate position may be scaled down and the orientation may be scaled up by the scaling function. In addition, the controller 21a may also execute a clutching function, which disengages the handle controller 38a from the robotic arm 40. In particular, the controller 21a stops transmitting movement commands from the handle controller 38a to the robotic arm 40 if certain movement limits or other thresholds are exceeded and in essence acts like a virtual clutch mechanism, e.g., limits mechanical input from effecting mechanical output.

[0049] The desired pose of the robotic arm 40 is based on the pose of the handle controller 38a and is then passed by an inverse kinematics function executed by the controller 21a. The inverse kinematics function calculates angles for the joints 44a, 44b, 44c of the robotic arm 40 that achieve the scaled and adjusted pose input by the handle controller 38a. The calculated angles are then passed to the robotic arm controller 41c, which includes a joint axis controller having a proportional-derivative (PD) controller, the friction estimator module, the gravity compensator module, and a two-sided saturation block, which is configured to limit the commanded torque of the motors of the joints 44a, 44b, 44c. In aspects, handle controller 38a may be substituted for and / or employed in conjunction with handle controller 38b.

[0050] FIG. 5 shows an image-guided navigation system 100 for use with a surgical robotic system 10 in non-line-of-site navigation of an instrument, such as a catheter during cardiac therapy or any other surgical procedure. The navigation system 100 may be used with any type of instrument or delivery system, including guide wires, needles, drug delivery systems, cell delivery systems, gene delivery systems and biopsy systems. Moreover, these instruments may be used for cardiac therapyor any other therapy in the body or be used to navigate or map any other regions of the body, such as moving body structures. The navigation system 100 addresses multiple cardiac, neurological, organ and other soft tissue therapies, including drug delivery, cell transplantation, gene delivery, electrophysiology ablations, transmyocardial vascularization (TMR), biopsy guidance, and virtual echography imaging.

[0051] The navigation system 100 may include an imaging device 112 that is used to acquire preoperative or real-time images of a patient 114. The imaging device 112 may be a fluoroscopic x- ray imaging device, a magnetic resonance imager (MRI), a computed tomography (CT) imager, a positron emission tomography (PET) imager, an isocentric fluoroscopy imager, a biplane fluoroscopy imager, an ultrasound imager, a multi-slice computed tomography (MSCT) imager, positron emission tomography-computed tomography (PET / CT), high definition computed tomography (HDCT), dual source computed tomography, a high-frequency ultrasound (HIFU) imager, an optical coherence tomography (OCT) imager, an intra-vascular ultrasound imager (IVUS), an intra-operative CT imager, an intra-operative MRI imager, a single photon emission computer tomography (SPECT) imager, and a combination thereof.

[0052] The imaging device 112 may include an arm 116 having an imaging (e.g., x-ray) source 118, a receiving section 120, an optional calibration and tracking target 122 and optional radiation sensors 124. The calibration and tracking target 122 includes calibration markers. An arm controller 128 captures the x-ray images received at the x-ray receiving section 120 and stores the images for later use. The arm controller 128 may also control the rotation of the arm 116. For example, the arm 116 may move in the direction of arrow 130 or rotate about the long axis of the patient, allowing anterior or lateral views of the patient 114 to be imaged. Each of these movements involves rotation about a mechanical connector 132 of the arm 116. In this example, the long axis of the patient 114 is substantially in line with the mechanical connector 132 of the arm 116. This enables the arm 116 to be rotated relative to the patient 114, allowing images of the patient 114 to be taken from multiple directions or about multiple planes. In embodiments, the arm 116 may be coupled to the movable cart 60 and controllable by the robotic system 100.

[0053] Two dimensional fluoroscopic images taken by the imaging device 112 are captured and stored in the arm controller 128. Multiple two-dimensional images taken by the imaging device 112 may also be captured and assembled to provide a larger view or image of a whole region of a patient, as opposed to being directed to only a portion of a region of the patient. These images arethen forwarded from the arm controller 128 to a controller or workstation 134 having one or more displays 136 and a user interface 138. The workstation 134 may be a special purpose computing device and may include any suitable processor (not shown) connected operably to a memory (not shown), which may include one or more of volatile, non-volatile, magnetic, optical, or electrical media, such as read-only memory (ROM), random access memory (RAM), electrically erasable programmable ROM (EEPROM), non-volatile RAM (NVRAM), or flash memory. The processor may be any suitable processor (e.g., control circuit) adapted to perform the operations, calculations, and / or set of instructions described in the present disclosure, such as a hardware processor, a field programmable gate array (FPGA), a digital signal processor (DSP), a central processing unit (CPU), a microprocessor, and combinations thereof. Those skilled in the art will appreciate that the processor may be substituted by using any logic processor (e.g., control circuit) adapted to execute algorithms, calculations, and / or set of instructions described herein.

[0054] The workstation 134 performs a plurality of functions, such as displaying on the display screen(s) 136, saving, digitally manipulating, or printing a hard copy of the received images. The user interface 138, which may be a keyboard, mouse, touch pen, touch screen or other suitable device, allows a physician or user to provide inputs to control the imaging device 112, via the arm controller 128, or adjust the display settings of the display screen(s) 136. The workstation 134 may also direct the arm controller 128 to adjust the mechanical connector 132 of the arm 116 to obtain various two-dimensional images along different planes to generate representative two-dimensional and three-dimensional images. When the x-ray source 118 generates the x-rays that propagate to the x-ray receiving section 120, the radiation sensors 124 sense the presence of radiation, which is forwarded to the arm controller 128, to identify whether or not the imaging device 112 is actively imaging. This information is also transmitted to a coil array controller 148. Alternatively, a person or physician may manually indicate when the imaging device 112 is actively imaging or this function can be built into the x-ray source 118, x-ray receiving section 120, or the controller 218.

[0055] While the fluoroscopic imaging device 112 is shown in FIG. 5, any other alternative 2D, 3D or 4D imaging modality may also be used. For example, any 2D, 3D or 4D imaging device, such as isocentric fluoroscopy, bi-plane fluoroscopy, ultrasound, computed tomography (CT), multi-slice computed tomography (MSCT), magnetic resonance imaging (MRI), high frequency ultrasound (HIFU), positron emission tomography (PET), optical coherence tomography (OCT), intra-vascular ultrasound (IVUS), ultrasound, intra-operative CT or MRI may also be used toacquire 2D, 3D or 4D pre-operative or real-time images or image data of the patient 114. The images may also be obtained and displayed in two, three or four dimensions. In more advanced forms, four-dimensional surface rendering of the heart or other regions of the body may also be achieved by incorporating heart data or other soft tissue data from an atlas map or from preoperative image data captured by MRI, CT, or echocardiography modalities.

[0056] Image datasets from hybrid modalities, such as positron emission tomography (PET) combined with CT, or single photon emission computer tomography (SPECT) combined with CT, could also provide functional image data superimposed onto anatomical data to be used to confidently reach target sights within the heart or other areas of interest. It should further be noted that the fluoroscopic imaging device 112, as shown in FIG. 5, provides a virtual bi-plane image using a single-head arm fluoroscope (i.e., imaging device 112) by simply rotating the arm 116 about at least two planes, which could be orthogonal planes to generate two-dimensional images that can be converted to three-dimensional volumetric images. By acquiring images in more than one plane, an icon representing the location of a catheter or other instrument, introduced and advanced in the patient 114, may be superimposed in more than one view on display screen(s) 136 allowing simulated bi-plane or even multi-plane views, including two and three-dimensional views.

[0057] These types of imaging modalities may provide certain distinct benefits and disadvantages for their use. For example, magnetic resonance imaging (MRI) is generally performed pre- operatively using a non-ionizing field. This type of imaging provides very good tissue visualization in three-dimensional form and also provides anatomy and functional information from the imaging. MRI imaging data is generally registered and compensated for motion correction using dynamic reference frames that are discussed herein.

[0058] Positron emission tomography (PET) imaging is generally a pre-operative imaging procedure that exposes the patient to some level of radiation to provide a 3D image. PET imaging provides functional information and also generally requires registration and motion correction using dynamic reference frames.

[0059] Computed tomography (CT) imaging is also generally a pre-operative technique that exposes the patient to a limited level of radiation. CT imaging, however, is a very fast imaging procedure. A multi-slice CT system provides 3D images having good resolution and anatomyinformation. Again, CT imaging is generally registered and needs to account for motion correction, via dynamic reference frames.

[0060] Fluoroscopy imaging is generally an intra-operative imaging procedure that exposes the patient to certain amounts of radiation to provide either two-dimensional or rotational three- dimensional images. Fluoroscopic images generally provide good resolution and anatomy information. Fluoroscopic images can be either manually or automatically registered and also need to account for motion correction using dynamic reference frames.

[0061] Ultrasound imaging is also generally intra-operative procedure using a non-ionizing field to provide either 2D, 3D, or 4D imaging, including anatomy and blood flow information. Ultrasound imaging provides automatic registration and does not need to account for any motion correction.

[0062] The navigation system 100 may include a second imaging device 166, which may be an ultrasound probe, such as a drop-in, laparoscopic ultrasound probe, an ultrasound probe, and the like. The ultrasound probe is configured to generate ultrasound data by emitting acoustic waves into the tissue and to receive the reflected acoustic waves, i.e., ultrasound data. The received reflected acoustic waves are then processed by the workstation 134 to identify various properties of the tissues through which the acoustic wave traveled, such as the density of the tissue. The ultrasound probe may be grasped by a robotic instrument or coupled directly to the IDU 52. In embodiments, the second imaging device 166 may be a fluoroscopic imager, a magnetic resonance imager (MRI), a computed tomography (CT) imager, a positron emission tomography (PET) imager, an isocentric fluoroscopy imager, a bi-plane fluoroscopy imager, an ultrasound imager, a multi-slice computed tomography (MSCT) imager, positron emission tomography-computed tomography (PET / CT), high definition computed tomography (HDCT), dual source computed tomography, a high-frequency ultrasound (HIFU) imager, an optical coherence tomography (OCT) imager, an intra-vascular ultrasound imager (IVUS), an intra-operative CT imager, an intraoperative MRI imager, a single photon emission computer tomography (SPECT) imager, and a combination thereof.

[0063] The navigation system 100 may further include an electromagnetic navigation system 144 that includes a transmitter coil array 146, the coil array controller 148, a navigation probe interface 150, a dynamic reference frame 154, and the instrument 50, which may be an electromagnetically tracked flexible instrument, such as a catheter, scope, or any other type of instrument.

[0064] In embodiments, the entire tracking system 144 or parts of the tracking system 144 may be incorporated into the imaging devices 112 and 166, including the workstation 134 and radiation sensors 124. Incorporating the tracking system 144 will provide an integrated imaging and tracking system. Any combination of these components may also be incorporated into the imaging system 112, which again can include a fluoroscopic arm imaging device or any other appropriate imaging device.

[0065] The transmitter coil array 146 is shown attached to the x-ray receiving section 120 of the arm 116. However, it should be noted that the transmitter coil array 146 may also be positioned at any other location as well. For example, the transmitter coil array 146 may be positioned at the x- ray source 118, within or atop the OR table 156 positioned below the patient 114, on siderails associated with the table 156, or positioned on the patient 114 in proximity to the region being navigated, such as on the patient's chest. The transmitter coil array 146 includes a plurality of coils that are each operable to generate distinct electromagnetic fields into the navigation region of the patient 114, which is sometimes referred to as patient space.

[0066] The transmitter coil array 146 is controlled or driven by the coil array controller 148. The coil array controller 148 drives each coil in the transmitter coil array 146 in a time division multiplex or a frequency division multiplex manner. In this regard, each coil may be driven separately at a distinct time, or all of the coils may be driven simultaneously with each being driven by a different frequency. Upon driving the coils in the transmitter coil array 146 with the coil array controller 148, electromagnetic fields are generated within the patient 114 in the area where the medical procedure is being performed, which is again sometimes referred to as patient space. The electromagnetic fields generated in the patient space induces currents in sensors 158 positioned in the instrument 50. These induced signals from the instrument 50 are delivered to the navigation probe interface 150 and subsequently forwarded to the coil array controller 148. The navigation probe interface 150 provides all the necessary electrical isolation for the navigation system 100. The navigation probe interface 150 also includes amplifiers, filters and buffers required to directly interface with the sensors 158 in instrument 50. Alternatively, the instrument 50 may employ a wireless communications channel as opposed to being coupled directly to the navigation probe interface 150.

[0067] The instrument 50 may be equipped with at least one, and generally multiple, localization sensors 158. The instrument 50 is also generally a steerable catheter that includes a handle at aproximal end and the multiple location sensors 158 fixed to the catheter body and spaced axially from one another along the distal segment of the instrument 50. The instrument 50, as shown in FIG. 5 may include four localization sensors 158. The localization sensors 158 are generally formed as electromagnetic receiver coils, such that the electromagnetic field generated by the transmitter coil array 146 induces current in the electromagnetic receiver coils or sensors 158. The instrument 50 may also be equipped with one or more sensors, which are operable to sense various physiological signals. For example, the instrument 50 may be provided with electrodes for sensing myopotentials or action potentials. An absolute pressure sensor may also be included, as well as other electrode sensors. The instrument 50 may also be provided with an open lumen to allow the delivery of a medical device or pharmaceutical / cell / gene agents. For example, the instrument 50 may be used as a guide catheter for deploying a medical lead, such as a cardiac lead for use in cardiac pacing and / or defibrillation or tissue ablation. The open lumen may alternatively be used to locally deliver pharmaceutical agents, cell, or genetic therapies.

[0068] The dynamic reference frame 154 of the electromagnetic tracking system 144 is also coupled to the navigation probe interface 150 to forward the information to the coil array controller 148. The dynamic reference frame 154 is a small magnetic field detector that is designed to be fixed to the patient 114 adjacent to the region being navigated so that any movement of the patient 114 is detected as relative motion between the transmitter coil array 146 and the dynamic reference frame 154. This relative motion is forwarded to the coil array controller 148, which updates registration correlation and maintains accurate navigation. The dynamic reference frame 154 can be configured as a pair of orthogonally oriented coils, each having the same center or may be configured in any other non-coaxial coil configuration. The dynamic reference frame 154 may be affixed externally to the patient 114, adjacent to the region of navigation, such as on the patient's chest, as shown in FIG. 5 or on the patient's back. The dynamic reference frame 154 can be affixed to the patient's skin, by way of a stick-on adhesive patch. The dynamic reference frame 154 may also be removably attachable to fiducial markers 160 also positioned on the patient's body.

[0069] Alternatively, the dynamic reference frame 154 may be internally attached, for example, to the wall of the patient's heart or other soft tissue using a temporary lead that is attached directly to the heart. This provides increased accuracy since this lead will track the regional motion of the heart. Gating will also increase the navigational accuracy of the system 100. It should further be noted that multiple dynamic reference frames 154 may also be employed. For example, an externaldynamic reference frame 154 may be attached to the chest of the patient 114, as well as to the back of the patient 114. Since certain regions of the body may move more than others due to motions of the heart or the respiratory system, each dynamic reference frame 154 may be appropriately weighted to increase accuracy even further. In this regard, the dynamic reference frame 154 attached to the back may be weighted higher than the dynamic reference frame 154 attached to the chest, since the dynamic reference frame 154 attached to the back is relatively static in motion.

[0070] In addition to electromagnetic tracking, the navigation system 100 may also include an optical tracking system 170, which may include any suitable camera 172 or cameras, such as one or more infrared cameras (e.g., bifocal cameras), able to identify, for example, active and passive tracking markers 174 in a given measurement volume viewable from the perspective of the camera 172. The camera 172 may scan the given measurement volume and detect the light that comes from the markers 174 in order to identify and determine the position of the markers 174 in three- dimensions. For example, active markers 174 may include infrared-emitting markers that are activated by an electrical signal (e.g., infrared light emitting diodes (LEDs)), and passive markers 174 may include retro-reflective markers that reflect infrared light (e.g., they reflect incoming IR radiation into the direction of the incoming light), for example, emitted by illuminators on the camera 172, or other suitable device. The markers 174 may be placed on the imaging devices 112 and 166 to allow for their registration and localization.

[0071] FIG. 6 shows the left-handle controller 38b, which is a mirror copy of the right-handle controller 38a, which may be used to control, i.e., move, activate, etc. tool coupled to the IDU 52 of the robotic arms 40, such as the instrument 50, the laparoscopic camera 51, and the ultrasound probe 166. Each of the handle controllers 38a and 38b includes a handle 701 and a paddle 708 that is pivotally coupled to the handle 701 at one end (e.g., proximal) of the paddle 708. The paddle 708 is configured to actuate a function of the instrument 50. The paddle 708 may include a finger sensor configured to detect presence or movement of a finger, such as touch sensors, capacitive sensors, optical sensors, and the like. In embodiments, the finger sensor 704 may be disposed on any portion of the handle controllers 38a and 38b. Each of the handle controllers 38a and 38b may also include a trigger 705a and one or more buttons 705b for activating various functions of the instrument 50. In addition, each of the handle controllers 38a and 38b may include a gimbal assembly 706 allowing for movement and rotation of the handle controllers 38a and 38b in a coordinate system of the handle controller 38a. The coordinate system is represented by a 3Daxis symbol 600 including the X-axis 600x, Y-axis 600y, Z-axis 600z. The gimbal assembly includes a plurality of frames 710a, 710b, 710c interconnected by rotatable joints 711 between each of the frames 710a, 710b, 710c, the handle 701, and a support frame 712. The joints include encoders or other sensors suitable for measuring rotation, which is then used as input to control movement (e.g., pitch, roll, yaw, etc.) of the instrument 50. In embodiments, the handle controller 38b may be any other directional input device, such as an analog joystick, a directional pad, a touchpad, trackball, mouse, and the like.

[0072] With reference to FIG. 7, the display screens 136 of the workstation 134 are used to display images obtained from the first imaging device 112 and / or the second imaging device 166. Each of the display screens, i.e., first display screen 136a and second display screen 136b, may be display an image from one of the image modalities, i.e., from the first imaging device 112 and the second imaging device 166, respectively. In embodiments, the images may be displayed on a single display in separate windows, regions, or picture-in-picture formats.

[0073] The first display screen 136a displays the image 180a from the first imaging modality, i.e., the first imaging device 112, which may be a fluoroscope as described above. The display screen 136a also shows a coordinate system of the first imaging modality, which is depicted using a 3D axis symbol 182, including the X-axis 182x, Y-axis 182y, Z-axis 182z. The orientation of the axes 182x, y, z is obtained using electromagnetic and / or optical tracking systems described above. The 3D axis symbol 182 may be overlayed on the image 180a.

[0074] The second display screen 136b displays the image 180b from the imaging second modality, i.e., the second imaging device 166, which may be an ultrasound probe as described above. The display screen 136b also shows a coordinate system of the second imaging modality, which is depicted using a 3D axis symbol 184, including the X-axis 184x, Y-axis 184y, Z-axis 184z. The orientation of the axes 184x, y, z is obtained using electromagnetic and / or optical tracking systems described above. The 3D axis symbol 184 may be overlayed on the image 180b.

[0075] As shown in FIG. 7, the respective coordinate systems of the first and second imaging modalities are not aligned due to different, i.e., misaligned, orientations of the corresponding imaging devices 112 and 166. Furthermore, the coordinate system of the hand controllers 38a and 38b is also not aligned with either of the coordinate systems of the first and second imaging modalities. Accordingly, the movement inputs at the handle controllers 38a and 38b in onedirection may result in movement of the catheter 152 in the first and second images in completely different direction due to the misalignment.

[0076] FIG. 9 shows a method for using a single imaging modality of navigation system 100 with the surgical robotic system 10. The method may be embodied in software instructions stored in memory and executable by a processor of the workstation 134, surgeon console 30, control tower 20, etc. At step 200, the surgical robotic system 10 is initialized, which includes attaching the instrument 50 to the IDU 52 of the robotic arm 40. The instrument 50 is controlled by user inputs through the handle controller 38a. In embodiments, pivoting the handle 701 about the Y-axis 600y, controls the yaw motion (e.g., movement between right and left) and pivoting the handle 701 about X-axis 600x, controls the pitch motion (e.g., movement between up and down). Similarly, the Z-axis 600z rotation provides for roll controls of the instrument. In embodiments, only some of the movements of the handle controller 38a may be mapped to the instrument 50. In other words, the instrument 50 may have less degrees of freedom than the handle controller 38a.

[0077] At step 202, the instrument 50 is advanced to the surgical site, which may be done by moving the holder 46, or other actuation means of the instrument 50. The first imaging device 112 is also activated to image the surgical site and the instrument 50 as it is advanced. The instrument 50 is advanced until the instrument 50 is within the imaging frame of the first imaging device 112. At this point, the instrument 50 is movable according to the original coordinate system of the handle controller 38a described above with respect to step 200.

[0078] At step 204, the instrument 50, and by extension the handle controller 38a is registered to the coordinate system of the first imaging device 112. As described above, registration may be performed using various tracking systems, e.g., the electromagnetic tracking system 144, the optical tracking system 170, etc. to identify location and orientation of the first imaging device 112.

[0079] The registration status may be displayed using 3D axis symbols as shown in FIG. 7. The coordinate system of the first imaging device 112 may be displayed as a first 3D axis symbol 194, which may be overlay ed on an image 190 from the first imaging device 112. In embodiments, the first 3D axis symbol 194 may be displayed over an end effector or distal portion of the instrument 50. The image 190 also includes a second 3D axis symbol 196, which represents the coordinate system of the handle controller 38a. The second 3D axis symbol 196 may be displayed in any portion of the image 190 or the display screen 136, e.g., upper corner. As shown in a table of FIG.8, prior to registration, the coordinate systems of the first imaging device 112 and the handle controller 38a are not aligned. Once registration is complete, the coordinate system of the handle controller 38a is remapped to align with the coordinate system of first imaging device 112 at step 206. The registration may be represented by rotating the second 3D axis symbol 196 to match the first 3D axis symbol 194. The timing of displaying the axis alignment may occur simultaneously with alignment or within a time period of alignment taking place, e.g., a delay of 0.5 seconds or more.

[0080] The registration process may be performed automatically by the surgical robotic system 10, e.g., based on the determination that the instrument 50 has reached the surgical site. In embodiments, the registration process may be started manually in response to a user command. Once the registration process is completed, a notification may be output by the robotic system 10 indicating that the instrument 50 may now be controlled by the registered handle controller 38a at step 208.

[0081] To further aid with the movement of the instrument 50, a GUI may be displayed as an overlay on the image 190. The GUI may show the resulting movement of the instrument 50 prior to actual movement thereof. In response to a movement input through the handle controller 38a, the system 10 may display an indicator, which may be a virtual representation 50’ of the instrument 50 of where the instrument 50 will move to and / or an arrow 198 corresponding to the movement. In embodiments, the GUI may display a confirmation prompt along with the movement indicator asking the user to confirm the indicated movement corresponds to desired movement.

[0082] FIG. 10 shows a method for using multiple imaging modalities of the navigation system 100. The method may also be embodied in software instructions stored in memory and executable by a processor of the workstation 134, surgeon console 30, control tower 20, etc. At step 300, the surgical robotic system 10 is initialized, which includes attaching the instrument 50 to the IDU 52 of the robotic arm 40. The instrument 50 is controlled by user inputs through the handle controller 38a as described above with respect to the method of FIG. 9.

[0083] At step 302, the instrument 50 is advanced to the surgical site, which may be done by moving the holder 46, or other actuation means of the instrument 50. The first imaging device 112 and the second imaging device 166 are also activated to image the surgical site and the instrument 50 as it is advanced to provide for dual modality imaging, e.g., fluoroscopy and ultrasound. The instrument 50 is advanced until the instrument 50 is within the imaging frame of the first imagingdevice 112 and the second imaging device 166. At this point, the instrument 50 is movable according to the original coordinate system of the handle controller 38a described above with respect to step 300.

[0084] At step 304, the instrument 50 is registered to the coordinate system of each of the first imaging device 112 and the second imaging device 166. As described above, registration may be performed using various tracking systems, e.g., the electromagnetic tracking system 144, the optical tracking system 170, etc., to identify location and orientation of the instrument 50 relative to the first imaging device 112 and the second imaging device 166.

[0085] The registration status may be displayed using 3D axis symbols as shown in FIG. 8. The first image 180a from the first imaging device 112 shows a coordinate system of the first imaging modality, which is depicted using the 3D axis symbol 182 as an overlay on the image 180a. The second image 180b from the second imaging device 166 shows a coordinate system of the second imaging modality, which is depicted using the 3D axis symbol 184. The orientation of the coordinate systems of the first and second modalities is obtained using electromagnetic and / or optical tracking systems described above.

[0086] At step 306, the coordinate systems of the first and second image modalities are aligned. This may be accomplished by moving one or both of the imaging devices 112 and 166 until the coordinate systems are aligned. Movement of the imaging device 112 and 166 is done by the robotic arms 116 or 40, respectively. Alignment of the coordinate systems may be indicated by the 3D axis symbols 182 and 184 being aligned at step 308. After two imaging modalities are aligned, the method of FIG. 9 may be performed to align the movement control system, i.e., the handle controller 38a, with both imaging modalities as described above.

[0087] The methods of FIGS. 9 and 10 are suitable for use during cardiac procedures where navigation and guidance of flexible instruments is particularly challenging. To further aid in such procedures, the navigation system 100 is also configured to compensate for the effects of respiration and the beating heart that can normally complicate navigation of instruments in and around the heart. Various patient parameters may be monitored such as blood pressure using a pressure monitor, blood oxygenation using a pulse oximeter, heart activity using an electrocardiogram monitor, etc. The imaging devices 112 and 166 may be operated based on the measured patient parameters.

[0088] The imaging process may be synced to physiological activity of the patient. Such that the image device 112 and / or 166 will capture images at this desired synced time. For example, the physiological change may be the beating heart, which is identified by ECG. This allows for the image acquisition to take place at the end of diastole. Diastole is the period of time between contractions of the atria or the ventricles during which blood enters the relaxed chambers from systemic circulation and the lungs. Diastole is often measured as the blood pressure at the instant of maximum cardiac relaxation. This makes this time period suitable for imaging. In further embodiments, image compensation may be accomplished by displaying images obtained during a specific time period, e.g., diastole. Thus, imaging may be performed continuously, but only image frames captured during certain time periods are displayed.

[0089] It will be understood that various modifications may be made to the embodiments disclosed herein. Therefore, the above description should not be construed as limiting, but merely as exemplifications of various embodiments. Those skilled in the art will envision other modifications within the scope and spirit of the claims appended thereto.

[0090] The following examples are illustrative of the techniques described herein.

[0091] Example 1. A surgical robotic system comprising: a first robotic arm including a first instrument drive unit and an instrument coupled to and actuatable by the first instrument drive unit; a surgeon console including a handle controller for receiving a user input to move the instrument in a first coordinate system; a second robotic arm including a second instrument drive unit and an imaging device coupled to and actuatable by the second instrument drive unit, the imaging device configured to capture an image of the instrument using an imaging modality in a second coordinate system, different from the first coordinate system; a tracking system for obtaining tracking data of the instrument and the imaging device; and a processor configured to: register the instrument in the second coordinate system based on the tracking data; align the first coordinate system with the second coordinate system; and receive inputs controlling the instrument in the second coordinate system.

[0092] Example 2. The surgical robotic system according to Example 1, wherein the imaging device is selected from the group consisting of an ultrasound probe and a fluoroscope, and the instrument is selected from the group consisting of a catheter and a flexible scope.

[0093] Example 3. The surgical robotic system according to Example 1, wherein the handle controller includes a plurality of directional inputs.

[0094] Example 4. The surgical robotic system according to Example 3, wherein the processor is configured to map the plurality of directional inputs from the first coordinate system to the second coordinate system.

[0095] Example 5. The surgical robotic system according to Example 1, wherein the surgeon console further includes a display screen for displaying the image captured by the imaging device.

[0096] Example 6. The surgical robotic system according to Example 5, wherein the processor is configured to display a first 3D axis symbol corresponding to the first coordinate system and a second 3D axis symbol corresponding to the second coordinate system.

[0097] Example 7. The surgical robotic system according to Example 6, wherein the processor is configured to align the second 3D axis symbol to the first 3D axis symbol when the first coordinate system is aligned with the second coordinate system.

[0098] Example 8. The surgical robotic system according to Example 1, wherein the processor is configured to display a movement indicator of where the instrument will move to in response to the inputs.

[0099] Example 9. The surgical robotic system according to Example 8, wherein the movement indicator is at least one of a virtual representation or an arrow.

[0100] Example 10. A method for controlling an imaging system with a surgical robotic system, the method comprising: receiving a user input at a handle controller for moving an instrument in a first coordinate system, wherein the instrument is coupled to a first instrument drive unit of a first robotic arm; capturing an image of the instrument through an imaging device using an imaging modality in a second coordinate system, wherein the imaging device is coupled to a second instrument drive unit of a second robotic arm; obtaining tracking data of the instrument and the imaging device using a tracking system; registering the instrument in the second coordinate system based on the tracking data; aligning the first coordinate system with the second coordinate system; and receiving inputs controlling the instrument in the second coordinate system.

[0101] Example 11. The method according to Example 10, wherein the handle controller includes a plurality of directional inputs.

[0102] Example 12. The method according to Example 11, further comprising mapping the plurality of directional inputs from the first coordinate system to the second coordinate system.

[0103] Example 13. The method according to Example 10, further comprising displaying the image captured by the imaging device on a display screen.

[0104] Example 14. The method according to Example 13, further comprising displaying a first 3D axis symbol corresponding to the first coordinate system and a second 3D axis symbol corresponding to the second coordinate system.

[0105] Example 15. The method according to Example 14, further comprising aligning the second 3D axis symbol to the first 3D axis symbol when the first coordinate system is aligned with the second coordinate system.

[0106] Example 16. A method for controlling a medical instrument in a plurality of different imaging modalities, the method comprising: capturing a first image of an instrument through a first imaging device using a first imaging modality in a first coordinate system, wherein the first imaging device is coupled to a first instrument drive unit of a first robotic arm; capturing a second image of the instrument through a second imaging device using a second imaging modality in a second coordinate system, wherein the second imaging device is coupled to a second instrument drive unit of a second robotic arm, the second imaging modality is different from the first imaging modality, and the second coordinate system is different from the first coordinate system; obtaining tracking data of the instrument, the first imaging device, and the second imaging device using a tracking system; registering the instrument in the first coordinate system and the second coordinate system based on the tracking data; and aligning the first coordinate system with the second coordinate system.

[0107] Example 17. The method according to Example 16, further comprising displaying the first image captured by the first imaging device side-by-side with the second image captured by the second imaging device.

[0108] Example 18. The method according to Example 17, further comprising displaying a first 3D axis symbol corresponding to the first coordinate system and a second 3D axis symbol corresponding to the second coordinate system.

[0109] Example 19. The method according to Example 18, further comprising aligning the second 3D axis symbol to the first 3D axis symbol when the first coordinate system of the first imaging device is aligned with the second coordinate system of the second imaging device.

[0110] Example 20. The method according to Example 16, wherein aligning the first coordinate system with the second coordinate system includes moving at least one of the first robotic arm or the second robotic arm.

Claims

WHAT IS CLAIMED IS:

1. A surgical robotic system (10) comprising: a first robotic arm (40) including a first instrument drive unit (52) and an instrument (50) coupled to and actuatable by the first instrument drive unit; a surgeon console (30) including a handle controller (38a, 38b) for receiving a user input to move the instrument in a first coordinate system; a second robotic arm (40, 116) including a second instrument drive unit (52) and an imaging device (112, 166) coupled to and actuatable by the second instrument drive unit, the imaging device configured to capture an image of the instrument using an imaging modality in a second coordinate system, different from the first coordinate system; a tracking system (144) for obtaining tracking data of the instrument and the imaging device; and a processor (21a) configured to: register the instrument in the second coordinate system based on the tracking data; align the first coordinate system with the second coordinate system; and receive inputs controlling the instrument in the second coordinate system.

2. The surgical robotic system according to claim 1, wherein the imaging device is selected from the group consisting of an ultrasound probe and a fluoroscope, and the instrument is selected from the group consisting of a catheter and a flexible scope.

3. The surgical robotic system according to any preceding claim, wherein the handle controller includes a plurality of directional inputs.

4. The surgical robotic system according to claim 3, wherein the processor is configured to map the plurality of directional inputs from the first coordinate system to the second coordinate system.

5. The surgical robotic system according to any preceding claim, wherein the surgeon console further includes a display screen (32, 34) for displaying the image captured by the imaging device.

6. The surgical robotic system according to claim 5, wherein the processor is configured to display a first 3D axis symbol (194) corresponding to the first coordinate system and a second 3D axis symbol (196) corresponding to the second coordinate system.

7. The surgical robotic system according to claim 6, wherein the processor is configured to align the second 3D axis symbol to the first 3D axis symbol when the first coordinate system is aligned with the second coordinate system.

8. The surgical robotic system according to any preceding claim, wherein the processor is configured to display a movement indicator (50’, 198) of where the instrument will move to in response to the inputs.

9. The surgical robotic system according to claim 8, wherein the movement indicator is at least one of a virtual representation (50’) or an arrow (198).

10. A method for controlling an imaging system with a surgical robotic system, the method comprising: receiving a user input at a handle controller (38a, 38b) for moving an instrument (50) in a first coordinate system, wherein the instrument is coupled to a first instrument drive unit (52) of a first robotic arm (40); capturing an image of the instrument through an imaging device (112, 166) using an imaging modality in a second coordinate system, wherein the imaging device is coupled to a second instrument drive unit (52) of a second robotic arm (40, 116); obtaining tracking data of the instrument and the imaging device using a tracking system; registering the instrument in the second coordinate system based on the tracking data; aligning the first coordinate system with the second coordinate system; and receiving inputs controlling the instrument in the second coordinate system.

11. The method according to claim 10, wherein the handle controller includes a plurality of directional inputs.

12. The method according to claim 11, further comprising mapping the plurality of directional inputs from the first coordinate system to the second coordinate system.

13. The method according to any preceding claim, further comprising displaying the image captured by the imaging device on a display screen (32, 34).

14. The method according to claim 13, further comprising displaying a first 3D axis symbol (194) corresponding to the first coordinate system and a second 3D axis symbol (196) corresponding to the second coordinate system.

15. The method according to claim 14, further comprising aligning the second 3D axis symbol to the first 3D axis symbol when the first coordinate system is aligned with the second coordinate system.

Images