Surgical robotic system for controlling steerable instruments using multiple imaging modalities

Abstract

The surgical robotic system includes a first robotic arm having a first instrument drive unit and an instrument coupled to and actuatable by the first instrument drive unit, and a surgeon console including a hand controller for receiving user input to move the instrument in a first coordinate system. The system also includes a second robotic arm having a second instrument drive unit and an imaging device coupled to and actuatable by the second instrument drive unit. The imaging device is configured to capture images of the instrument in a second coordinate system different from the first coordinate system using an imaging modality. A tracking system is also provided for obtaining tracking data of the instrument and the imaging device. The surgical robotic system further includes a processor configured to register the instrument in the second coordinate system based on the tracking data, align the first coordinate system with the second coordinate system, and receive input to control the instrument in the second coordinate system.

Description

Cross-references to related applications

[0001] This application claims the benefit of U.S. Provisional Patent Application Serial No. 63 / 574,401, filed April 4, 2024, the entire contents of which are incorporated herein by reference. Background Technology

[0002] Surgical robotic systems are currently used in a variety of medical procedures, including minimally invasive surgery. Some surgical robotic systems consist of a surgeon's console that controls a robotic arm and surgical instruments with end effectors (e.g., clamps or gripping instruments) that are coupled to and actuated by the robotic arm. During operation, the robotic arm is moved to a position above the patient, and then the surgical instruments are guided through the patient's surgical port or natural orifice into a small incision to position the end effector at the working site within the patient's body. Surgical robotic systems are used in conjunction with various types of instruments designed to perform specific functions, such as manipulable catheters and endoscopes.

[0003] Image-guided medical and surgical procedures utilize patient images acquired before or during the procedure to guide the physician. These procedures can be termed computer-aided procedures. Recent advances in imaging technology, particularly those producing highly detailed two-dimensional, three-dimensional, and four-dimensional images (such as computed tomography (CT), magnetic resonance imaging (MRI), fluorescence imaging (e.g., using a C-arm), positron emission tomography (PET), and ultrasound imaging (US), have increased interest in image-guided medical procedures.

[0004] Typical image-guided navigation systems often require a dynamic reference frame (DRF) to track the patient's position as they move during the procedure. The DRF is typically fixed to the patient in a largely permanent or immovable manner. The DRF can also be used as a reference marker and can therefore be attached to the patient during preoperative image acquisition. This allows the image space to be aligned with the patient space during the navigation procedure.

[0005] During the procedure, various instruments may be used that are intended to be tracked. Image data is typically acquired intraoperatively or preoperatively, and instruments are usually displayed and overlaid on the captured image data to identify their position relative to the patient space. Therefore, instruments may include tracking sensors (such as electromagnetic coils) or optical detection points (such as LEDs or reflectors that can be detected by a suitable tracking system). Furthermore, the tracking system may use a digital radiometer (DRF) to maintain registration or positioning between the patient space and the image space. The DRF can also be any suitable tracking sensor fixed to a part of the patient, allowing the system to determine whether the patient has moved relative to the image space.

[0006] Flexible surgical instruments (such as catheters and endoscopes) are used in a variety of procedures to treat the heart, lungs, and other organs. These instruments are used with the aid of various visualization techniques to assist in instrument navigation and procedure execution. In some cases, multiple visualization modalities (such as ultrasound and fluoroscopy) are used simultaneously to guide the instruments. In these situations, surgeons need to manually perform mental registration and compensation of images and instruments. Specifically, surgeons need to understand how each imaging modality relates to other imaging modalities, how instruments are imaged in different modalities, and how motion inputs affect instrument movement. These mental processes impose a significant psychological burden and waste a considerable amount of time during the procedure. Summary of the Invention

[0007] This disclosure provides a surgical robot system and navigation system for navigating surgical instruments using multiple imaging modalities. Suitable imaging modalities include fluorescence fluoroscopy imaging systems and ultrasound imaging systems. The navigation system includes reference markers, electromagnetic trackers, or any other indexing markers / trackers for locating imaging components, instruments, patients, etc. These markers can be tracked using white light cameras or infrared cameras, electromagnetic receivers, etc.

[0008] The robotic system is configured to combine multiple imaging modalities, such as fluorescence fluoroscopy and ultrasound. In particular, the positioning of the imaging components allows for the combination of fluorescence fluoroscopy and ultrasound images using the same coordinate system by positioning the image acquisition components (e.g., ultrasound transducers, X-ray sensors, etc.) in space relative to a camera, receiver, or some other reference point.

[0009] The imaging characteristics are then used to process the positioning information so that the outputs from both devices can be oriented in terms of rotation and zoom / zoom on the display(s)(multiple) screens, while also providing the operator with potential instructions to align the imaging system as needed. In embodiments where the imaging system is controlled and actuated by a robotic system, automatic alignment can also be used.

[0010] This system can also be used to align a robot's visual reference with an image. Robot system components (e.g., a robotic arm) and image components are aligned to be in the same plane, or to perform real-time registration of robot motion within an image to detect the position of the robot's coordinate system within the current imaging output. After registration, robot control can be aligned so that each Cartesian control command occurs on the same axis as the image—for example, a "right" input moves the device to the right, an "up" input moves the device upward, and so on.

[0011] According to one embodiment of this disclosure, a surgical robot system is disclosed. The surgical robot system includes: a first robotic arm having a first instrument drive unit and an instrument coupled to and actuable by the first instrument drive unit; and a surgeon's console including a handle controller for receiving user input to move the instrument in a first coordinate system. The system further includes: a second robotic arm having a second instrument drive unit and an imaging device coupled to and actuable by the second instrument drive unit. The imaging device is configured to capture images of the instrument in a second coordinate system different from the first coordinate system using an imaging modality. A tracking system is also provided for acquiring tracking data of the instrument and the imaging device. The surgical robot system further includes a processor configured to: register the instrument in the second coordinate system based on the tracking data, align the first coordinate system with the second coordinate system, and receive input to control the instrument in the second coordinate system.

[0012] Implementations of the above embodiments may include one or more of the following features. According to one aspect of the above embodiments, the imaging device may be an ultrasound probe or a fluoroscopy system. The instrument may be a catheter or a flexible endoscope. The tracking system may be an optical tracking system or an electromagnetic tracking system. The handheld controller may include multiple directional inputs. The processor may be configured to map the multiple directional inputs from the first coordinate system to the second coordinate system. The surgeon's console may further include a display screen for displaying images captured by the imaging device. The processor is configured to display a first 3D axis symbol corresponding to the first coordinate system and a second 3D axis symbol corresponding to the second coordinate system. The processor may be configured to align the second 3D axis symbol with the first 3D axis symbol when the first coordinate system is aligned with the second coordinate system.

[0013] According to another embodiment of this disclosure, a method for controlling an imaging system using a surgical robot system is described. The method includes: receiving user input at a handle controller for moving an instrument in a first coordinate system, wherein the instrument is coupled to a first instrument drive unit of a first robotic arm; and capturing an image of the instrument in a second coordinate system using an imaging device with an imaging modality, wherein the imaging device is coupled to a second instrument drive unit of a second robotic arm; and obtaining tracking data of the instrument and the imaging device using a tracking system. The method further includes: registering the instrument in the second coordinate system based on the tracking data. The method further includes: aligning the first coordinate system with the second coordinate system. The method additionally includes: receiving input for controlling the instrument in the second coordinate system.

[0014] The implementation of the above embodiments may include one or more of the following features. According to one aspect of the above embodiments, the handle controller may include multiple directional inputs. The method may further include: mapping the multiple directional inputs from the first coordinate system to the second coordinate system. The method may further include: displaying an image captured by an imaging device on a display screen. The method may further include: displaying a first 3D axis symbol corresponding to the first coordinate system and a second 3D axis symbol corresponding to the second coordinate system. The method may also include: aligning the second 3D axis symbol with the first 3D axis symbol when the first coordinate system is aligned with the second coordinate system.

[0015] According to another embodiment of this disclosure, a method for controlling a medical device in multiple different imaging modalities is described. The method includes: capturing a first image of the device in a first coordinate system using a first imaging device with a first imaging modality, wherein the first imaging device is coupled to a first device drive unit of a first robotic arm. The method further includes: capturing a second image of the device in a second coordinate system using a second imaging device with a second imaging modality, wherein the second imaging device is coupled to a second device drive unit of a second robotic arm, and the second imaging modality is different from the first imaging modality, and the second coordinate system is different from the first coordinate system. The method further includes: obtaining tracking data of the device, the first imaging device, and the second imaging device using a tracking system. The method further includes: registering the device in the first coordinate system and the second coordinate system based on the tracking data. The method further includes: aligning the first coordinate system with the second coordinate system.

[0016] The implementation of the above embodiments may include one or more of the following features. According to one aspect of the above embodiments, the method may further include: displaying the first image captured by the first imaging device and the second image captured by the second imaging device side-by-side. The method may further include: displaying a first 3D axis symbol corresponding to the first coordinate system and a second 3D axis symbol corresponding to the second coordinate system. The method may also include: aligning the second 3D axis symbol with the first 3D axis symbol when the first coordinate system of the first imaging device is aligned with the second coordinate system of the second imaging device. Aligning the first coordinate system with the second coordinate system may include: moving at least one of the first robotic arm or the second robotic arm. Attached Figure Description

[0017] Various embodiments of this disclosure are described herein with reference to the accompanying drawings, in which: Figure 1This is a perspective view of a surgical robot system according to an embodiment of the present disclosure, the surgical robot system including a control tower, a console, and one or more surgical robotic arms, each of the one or more surgical robotic arms being mounted on a movable trolley; Figure 2 Examples based on this disclosure Figure 1 A 3D view of the surgical robotic arm of a surgical robot system; Figure 3 This is a perspective view of a movable trolley with a mounting arm according to an embodiment of the present disclosure, the mounting arm having... Figure 1 Surgical robotic arms in surgical robot systems; Figure 4 Examples based on this disclosure Figure 1 A schematic diagram of the computer architecture of a surgical robot system; Figure 5 This is an embodiment of the present disclosure for use with Figure 1 The navigation system used in surgical robot systems; Figure 6 This is a perspective view of a handle controller according to one embodiment of the present disclosure; Figure 7 This is a view of a workstation according to an embodiment of the present disclosure, the workstation including: a first display screen showing an image of a first imaging mode and a second display screen showing an image of a second imaging mode; Figure 8 This is a view of a graphical user interface (GUI) according to an embodiment of this disclosure, showing an image of a first imaging modality and coordinate system icons representing registration; Figure 9 This is a flowchart of a method for controlling a maneuverable device using a single imaging modality, according to embodiments of this disclosure; and Figure 10 This is a flowchart of a method for controlling a maneuverable device using multiple imaging modalities, according to embodiments of this disclosure. Detailed Implementation

[0018] Embodiments of the surgical robot system disclosed herein are described in detail with reference to the accompanying drawings, in which the same reference numerals denote the same or corresponding elements in each of several views.

[0019] refer to Figure 1The surgical robot system 10 includes a control tower 20 communicatively connected to all components of the surgical robot system 10, including a surgeon's console 30 and one or more mobile trolleys 60. Each mobile trolley 60 includes a robotic arm 40 with surgical instruments 50 attached thereto. The robotic arm 40 is also attached to the mobile trolley 60. The robot system 10 may include any number of mobile trolleys 60 and / or any number of robotic arms 40.

[0020] Surgical instrument 50 is configured for use during minimally invasive surgery. In an embodiment, surgical instrument 50 may be configured for open surgery. One of the robotic arms 40 may include an endoscope camera 51 configured to capture video of the surgical site. Endoscope camera 51 may be a stereoscopic endoscope configured to capture two side-by-side (i.e., left and right) images of the surgical site to generate a video stream of the surgical scene. Endoscope camera 51 is coupled to video processing unit 56, which may be located within control tower 20. Video processing unit 56 may be any computing device as described below, configured to receive video feeds from endoscope camera 51 and output a processed video stream.

[0021] The surgeon's console 30 includes a first display 32 and a second display 34. The first display shows a video feed of the surgical site provided by a camera 51 mounted on the robotic arm 40, and the second display shows a user interface for controlling the surgical robot system 10. The first display 32 and the second display 34 may be touchscreens, thereby allowing the display of various graphical user inputs that can be selected or moved by the user.

[0022] The surgeon's console 30 also includes multiple user interface devices, such as a foot pedal 36 and a pair of handle controllers 38a and 38b, which the user uses to remotely control the robotic arm 40. The surgeon's console further includes a handrail 33 for supporting the clinician's arm while the clinician operates the handle controllers 38a and 38b.

[0023] The control tower 20 may also include a display 23, which may be a touchscreen and can display one or more graphical user interfaces (GUIs). The control tower 20 also serves as an interface between the surgeon's console 30 and one or more robotic arms 40. Specifically, the control tower 20 is configured to control the robotic arms 40 based on a set of programmable instructions and / or input commands from the surgeon's console 30, such as moving the robotic arms 40 and corresponding surgical instruments 50. In response to these instructions and / or inputs, the robotic arms 40 and surgical instruments 50 perform desired sequences of movement in response to inputs from the foot pedal 36 and the hand controllers 38a and 38b. The system 10 may be configured to allow the foot pedal 36 to influence one or more of various system functions, such as enabling and disabling the hand controllers 38a and 38b, repositioning camera movement, and starting / disabling electric surgical instruments. Specifically, the foot pedal 36 may be used to perform clutch actions on the hand controllers 38a and 38b. Engaging the clutch by pressing one of the foot pedals 36 disconnects the hand controllers 38a and / or 38b (i.e., prevents movement input from these hand controllers), thus preventing the robotic arm 40 and the corresponding instrument 50 or camera 51 from being actuated. This allows the user to reposition the hand controllers 38a and 38b without moving the robotic arm 40 and the instrument 50 and / or camera 51. This is useful, for example, when reaching the control boundaries of the surgical space.

[0024] Each of the control tower 20, the surgeon's console 30, and the robotic arm 40 includes a corresponding computer 21, 31, or 41. Computers 21, 31, and 41 are interconnected using any suitable communication network based on wired or wireless communication protocols. As used herein, the term "network," whether plural or singular, refers to a data network, including but not limited to the Internet, intranets, wide area networks, or local area networks. Suitable protocols include, but are not limited to, Transmission Control Protocol / Internet Protocol (TCP / IP), Datagram Protocol / Internet Protocol (UDP / IP), and / or Datagram Congestion Control Protocol (DCCP). Wireless communication can be implemented via one or more wireless configurations, such as radio frequency (RF), optical, Wi-Fi, Bluetooth (an open wireless protocol used to exchange data between fixed and mobile devices over short distances using short-wavelength radio waves to create personal area networks (PANs)), and ZigBee® (a specification of advanced communication protocols using small, low-power digital radios based on the IEEE 122.15.4-1203 Wireless Personal Area Network (WPAN) standard).

[0025] Computers 21, 31, and 41 may include any suitable processor (not shown) operatively connected to a memory (not shown) that may include one or more of volatile, non-volatile, magnetic, optical, or electrical media, such as read-only memory (ROM), random access memory (RAM), electrically erasable programmable ROM (EEPROM), non-volatile RAM (NVRAM), or flash memory. The processor may be any suitable processor (e.g., control circuitry) adapted to perform the operations, calculations, and / or instruction sets described in this disclosure, such as a hardware processor, field-programmable gate array (FPGA), digital signal processor (DSP), central processing unit (CPU), microprocessor, or combinations thereof. Those skilled in the art will understand that a processor may be used instead of a logic processor (e.g., control circuitry) adapted to perform the algorithms, calculations, and / or instruction sets described herein.

[0026] refer to Figure 2 Each of the robotic arms 40 may include a plurality of links 42a, 42b, 42c, which are interconnected at joints 44b and 44c, respectively. Other configurations of links and joints may be used, as is known to those skilled in the art. Joint 44a is configured to secure the robotic arm 40 to a movable trolley 60 and define a first longitudinal axis. (Reference) Figure 3 The movable trolley 60 includes a lift 67 and a mounting arm 61, which provides a base for mounting the robotic arm 40. The lift 67 allows vertical movement of the mounting arm 61, thereby allowing vertical movement of the robotic arm 40 mounted on the mounting arm 61. The movable trolley 60 also includes a display 69 for displaying information about the robotic arm 40. In embodiments, the robotic arm 40 may include any type and / or any number of joints.

[0027] Further reference Figure 3 The mounting arm 61 includes a first link 62a, a second link 62b, and a third link 62c, which provide lateral maneuverability to the robotic arm 40. Links 62a, 62b, and 62c are interconnected at joints 63a and 63b, each joint including an actuator (not shown) for rotating links 62b and 62b relative to each other and relative to link 62c. Specifically, links 62a, 62b, and 62c are movable in corresponding lateral planes parallel to each other, thereby allowing the robotic arm 40 to extend relative to a patient (e.g., an operating table). In an embodiment, the robotic arm 40 may be coupled to an operating table (not shown). The mounting arm 61 includes controls 65 for adjusting the movement of links 62a, 62b, and 62c, as well as the lift 67. In an embodiment, the mounting arm 61 may include any type and / or any number of joints.

[0028] The third link 62c may include a rotatable base 64 having two degrees of freedom. Specifically, the rotatable base 64 includes a first actuator 64a and a second actuator 64b. The first actuator 64a is rotatable about a first fixed arm axis perpendicular to the plane defined by the third link 62c. Furthermore, the second actuator 64b is rotatable about a second fixed arm axis transverse to the first fixed arm axis. The first actuator 64a and the second actuator 64b, together with the lift 67, achieve full three-dimensional orientation of the robotic arm 40.

[0029] return Figure 2 Actuator 48b of joint 44b is connected to joint 44c via belt 45a, and joint 44c is in turn connected to joint 46b via belt 45b. Joint 44c may include a transfer case connecting belts 45a and 45b, such that actuator 48b is configured to rotate each of links 42b, 42c and retainer 46 relative to each other. More specifically, links 42b, 42c and retainer 46 are passively connected to actuator 48b, which forces rotation about a pivot point “P” located at the intersection of a first axis defined by link 42a and a second axis defined by retainer 46. In other words, pivot point “P” is the remote center of motion (RCM) of robotic arm 40. Thus, actuator 48b controls the angle θ between the first and second axes, thereby allowing orientation of surgical instrument 50. Because links 42a, 42b, 42c and retainer 46 are interconnected via belts 45a and 45b, the angle between links 42a, 42b, 42c and retainer 46 is also adjusted to achieve a desired angle θ. In an embodiment, some or all of joints 44a, 44b, 44c may include actuators to eliminate the need for mechanical linkages.

[0030] Joints 44a and 44b include corresponding actuators 48a and 48b, which are configured to drive joints 44a, 44b, and 44c relative to each other via a series of links 45a and 45b or other mechanical linkages (such as drive rods, cables, or levers). In particular, actuator 48a is configured to rotate the robotic arm 40 about a longitudinal axis defined by link 42a.

[0031] refer to Figure 2 The retainer 46 defines a second longitudinal axis and is configured to receive the instrument drive unit (IDU) 52. Figure 1IDU 52 is configured to be coupled to the actuation mechanism of surgical instrument 50 and camera 51 and is configured to move (e.g., rotate) and actuate the instrument 50 and / or camera 51. IDU 52 transmits actuating force from its actuator to surgical instrument 50 to actuate components of the end effector 49 of surgical instrument 50. Holder 46 includes a sliding mechanism 46a configured to move IDU 52 along a second longitudinal axis defined by holder 46. Holder 46 also includes a joint 46b that rotates holder 46 relative to link 42c. During endoscopic procedures, instrument 50 can pass through endoscope access port 55 held by holder 46. Figure 3 Insertion. The retainer 46 also includes a port latch 46c for securing the inlet port 55 to the retainer 46. Figure 2 ).

[0032] IDU 52 is attached to retainer 46, and then a sterile interface module (SIM) 43 is attached to the distal portion of IDU 52. SIM 43 is configured to secure a sterile drape (not shown) to IDU 52. Instrument 50 is then attached to SIM 43. Instrument 50 is then inserted through inlet port 55 by moving IDU 52 along retainer 46. SIM 43 includes multiple drive shafts configured to transmit rotation of individual motors of IDU 52 to instrument 50, thereby actuating instrument 50. Furthermore, SIM 43 provides a sterile barrier between instrument 50 and other components of robotic arm 40, including IDU 52.

[0033] The robotic arm 40 also includes a mounting arm 61 and multiple manual control buttons 53 mounted on the IDU 52. Figure 1 The mounting arm can be used in manual mode. The user can press one or more of these buttons 53 to move the component associated with the one or more buttons 53.

[0034] refer to Figure 4Each of the computers 21, 31, and 41 in the surgical robot system 10 may include multiple controllers, which may be implemented in hardware and / or software. The computer 21 of the control tower 20 includes a controller 21a and a safety observer 21b. The controller 21a receives data from the computer 31 of the surgeon's console 30 regarding the current position and / or orientation of the handle controllers 38a and 38b, as well as the status of the foot pedals 36 and other buttons. The controller 21a processes these input positions to determine the desired drive commands for each joint of the robotic arm 40 and / or the IDU 52, and transmits these desired drive commands to the computer 41 of the robotic arm 40. The controller 21a also receives the actual joint angles measured by the encoders of the actuators 48a and 48b and uses this information to determine force feedback commands, which are transmitted back to the computer 31 of the surgeon's console 30 to provide tactile feedback via the handle controllers 38a and 38b. The safety observer 21b performs validity checks on the data entering and exiting the controller 21a, and if an error is detected in the data transmission, it notifies the system fault handler to put the computer 21 and / or the surgical robot system 10 into a safe state.

[0035] Computer 41 includes multiple controllers: a trolley main controller 41a, a mounting arm controller 41b, a robotic arm controller 41c, and an instrument drive unit (IDU) controller 41d. The trolley main controller 41a receives and processes joint commands from controller 21a on computer 21 and transmits them to the mounting arm controller 41b, robotic arm controller 41c, and IDU controller 41d. The trolley main controller 41a also manages instrument exchange and the overall status of the movable trolley 60, robotic arm 40, and IDU 52. The trolley main controller 41a also transmits actual joint angles back to controller 21a.

[0036] Each of joints 63a and 63b, and the rotatable base 64 of the mounting arm 61, is a passive joint that allows manual adjustment by the user (i.e., where no actuator is present). Joints 63a and 63b, and the rotatable base 64, include brakes that are disengaged by the user to configure the mounting arm 61. When the brakes are engaged, the mounting arm controller 41b monitors slippage of each of joints 63a and 63b of the mounting arm 61 and the rotatable base 64, or when the brakes are disengaged, the operator is free to move the mounting arm controller without affecting control of other joints. The robotic arm controller 41c controls each joint 44a and 44b of the robotic arm 40 and calculates the desired motor torque required for gravity compensation, friction compensation, and closed-loop position control of the robotic arm 40. The robotic arm controller 41c calculates movement commands based on the calculated torques. The calculated motor commands are then transmitted to one or more of the actuators 48a and 48b in the robotic arm 40. The actual joint position is then transmitted back to the robotic arm controller 41c by actuators 48a and 48b.

[0037] IDU controller 41d receives the desired joint angles (such as wrist angle and gripper angle) from surgical instrument 50 and calculates the desired current of the motor in IDU 52. IDU controller 41d calculates the actual angles based on the motor position and transmits these actual angles back to trolley main controller 41a.

[0038] The robotic arm 40 is controlled in response to the posture of a handle controller (e.g., handle controller 38a) that controls the robotic arm 40, the posture being transformed into the desired posture of the robotic arm 40 via a hand-eye transformation function performed by controller 21a. The hand-eye function, as well as other functions described herein, is implemented in software executable by controller 21a or any other suitable controller described herein. The posture of one of the handle controllers 38a can be implemented as a coordinate position and roll-pitch-yaw (RPY) orientation relative to a coordinate reference system fixed to the surgeon's console 30. The desired posture of the instrument 50 is relative to a fixation system on the robotic arm 40. The posture of the handle controller 38a is then scaled by a scaling function performed by controller 21a. In an embodiment, the scaling function can reduce the coordinate position and enlarge the orientation. Additionally, controller 21a can also perform a clutch function to disengage the handle controller 38a from the robotic arm 40. Specifically, if certain movement limits or other thresholds are exceeded, controller 21a stops transmitting movement commands from handle controller 38a to robotic arm 40, and essentially acts like a virtual clutch mechanism, for example, limiting the mechanical input from affecting the mechanical output.

[0039] The desired posture of the robotic arm 40 is based on the posture of the handle controller 38a and then transmitted via an inverse kinematics function executed by the controller 21a. The inverse kinematics function calculates the angles of the joints 44a, 44b, and 44c of the robotic arm 40, realizing the scaled and adjusted posture input from the handle controller 38a. The calculated angles are then transmitted to the robotic arm controller 41c, which includes a joint axis controller with a proportional-derivative (PD) controller, a friction estimator module, a gravity compensator module, and a double-sided saturation block configured to limit the command torque of the motors of joints 44a, 44b, and 44c. In various respects, the handle controller 38a can be replaced by and / or used in conjunction with the handle controller 38b.

[0040] Figure 5 An image-guided navigation system 100 is illustrated for use with a surgical robotic system 10 for non-line-of-sight navigation of instruments, such as catheters, during cardiac treatments or any other surgical procedures. The navigation system 100 can be used with any type of instrument or delivery system, including guidewires, needles, drug delivery systems, cell delivery systems, gene delivery systems, and biopsy systems. Furthermore, these instruments can be used for cardiac treatments or any other treatments within the body, or for navigating or mapping any other area of ​​the body, such as moving body structures. The navigation system 100 is suitable for a variety of cardiac, neurological, organ, and other soft tissue treatments, including drug delivery, cell transplantation, gene delivery, electrophysiological ablation, transmyocardial revascularization (TMR), biopsy guidance, and virtual ultrasound imaging.

[0041] The navigation system 100 may include an imaging device 112 for acquiring preoperative or real-time images of the patient 114. The imaging device 112 may be a fluorescence fluoroscopy X-ray imaging device, a magnetic resonance imaging (MRI) system, a computed tomography (CT) system, a positron emission tomography (PET) system, an isocentric fluorescence fluoroscopy system, a biplane fluorescence fluoroscopy system, an ultrasound system, a multislice computed tomography (MSCT) system, a positron emission tomography-computed tomography (PET / CT) system, a high-resolution computed tomography (HDCT) system, a dual-source computed tomography system, a high-frequency ultrasound (HIFU) system, an optical coherence tomography (OCT) system, an intravascular ultrasound (IVUS) system, an intraoperative CT system, an intraoperative MRI system, a single-photon emission computed tomography (SPECT) system, or combinations thereof.

[0042] Imaging device 112 may include an arm 116 having an imaging (e.g., X-ray) source 118, a receiver 120, an optional calibration and tracking target 122, and an optional radiation sensor 124. The calibration and tracking target 122 includes calibration marks. An arm controller 128 captures and stores X-ray images received at the X-ray receiver 120 for later use. The arm controller 128 may also control rotation of the arm 116. For example, the arm 116 may move in the direction of arrow 130 or rotate about the long axis of the patient, thereby allowing imaging of anterior or lateral views of the patient 114. Each of these movements involves rotation about the mechanical connector 132 of the arm 116. In this example, the long axis of the patient 114 is substantially aligned with the mechanical connector 132 of the arm 116. This allows the arm 116 to rotate relative to the patient 114, thereby allowing images of the patient 114 to be captured from multiple directions or around multiple planes. In one embodiment, arm 116 may be coupled to a movable trolley 60 and may be controlled by a robotic system 100.

[0043] Two-dimensional fluorescence fluoroscopic images captured by imaging device 112 are captured and stored in arm controller 128. Multiple two-dimensional images captured by imaging device 112 may also be captured and stitched together to provide a larger view or image of the entire area of ​​the patient, rather than just a portion of the patient's area. These images are then forwarded from arm controller 128 to a controller or workstation 134 having one or more displays 136 and a user interface 138. Workstation 134 may be a dedicated computing device and may include any suitable processor (not shown) operatively connected to a memory (not shown) that may include one or more of volatile, non-volatile, magnetic, optical, or electrical media, such as read-only memory (ROM), random access memory (RAM), electrically erasable programmable ROM (EEPROM), non-volatile RAM (NVRAM), or flash memory. The processor may be any suitable processor (e.g., control circuitry) adapted to perform the operations, calculations, and / or instruction sets described in this disclosure, such as a hardware processor, field-programmable gate array (FPGA), digital signal processor (DSP), central processing unit (CPU), microprocessor, or combinations thereof. Those skilled in the art will understand that a processor can be used instead of a logic processor (e.g., control circuitry) suitable for executing the algorithms, computations, and / or instruction sets described herein.

[0044] Workstation 134 performs multiple functions, such as displaying, saving, digitally processing, or printing paper copies of received images on the display(s)136. User interface 138 (which may be a keyboard, mouse, stylus, touchscreen, or other suitable device) allows physicians or users to provide input to control imaging device 112 via arm controller 128, or to adjust display settings on the display(s)136. Workstation 134 can also guide arm controller 128 to adjust the mechanical connector 132 of arm 116 to obtain various two-dimensional images along different planes, thereby generating representative two-dimensional and three-dimensional images. When x-ray source 118 generates x-rays that propagate to x-ray receiver 120, radiation sensor 124 senses the presence of radiation, which is relayed to arm controller 128 to identify whether imaging device 112 is actively imaging. This information is also transmitted to coil array controller 148. Alternatively, personnel or physicians can manually indicate when the imaging device 112 is actively imaging, or this function can be built into the x-ray source 118, the x-ray receiver 120, or the controller 218.

[0045] Although Figure 5 The diagram illustrates a fluorescence fluoroscopy imaging device 112, but any other alternative 2D, 3D, or 4D imaging modality may also be used. For example, any 2D, 3D, or 4D imaging device, such as isocentric fluoroscopy, biplane fluoroscopy, ultrasound, computed tomography (CT), multislice computed tomography (MSCT), magnetic resonance imaging (MRI), high-frequency ultrasound (HIFU), positron emission tomography (PET), optical coherence tomography (OCT), intravascular ultrasound (IVUS), ultrasound, intraoperative CT, or MRI, may also be used to acquire 2D, 3D, or 4D preoperative or real-time images or image data of the patient 114. Images can also be acquired and displayed in two, three, or four dimensions. In a more advanced form, four-dimensional surface reconstruction of the heart or other areas of the body can also be achieved by combining cardiac data or other soft tissue data from atlas mapping or from preoperative image data captured via MRI, CT, or echocardiography modalities.

[0046] Image datasets from hybrid modalities (such as positron emission tomography (PET) combined with CT or single-photon emission computed tomography (SPECT) combined with CT) can also provide functional image data overlaid on anatomical data for reliably reaching target sites in the heart or other regions of interest. It should be further noted that, as Figure 5The illustrated fluoroscopic imaging device 112 provides virtual biplane images using a single-headed fluoroscopy unit (i.e., imaging device 112) by simply rotating arm 116 around at least two planes (which may be orthogonal planes) to generate a two-dimensional image that can be converted into a three-dimensional volumetric image. By acquiring images in more than one plane, icons representing the location of a catheter or other instrument (which is introduced and advanced into the patient 114) can be superimposed on more than one view on display(s) 136, thereby allowing simulated biplane or even multiplane views, including two-dimensional and three-dimensional views.

[0047] These types of imaging modalities offer certain distinct advantages and disadvantages for their use. For example, magnetic resonance imaging (MRI) is typically performed preoperatively using a non-ionizing field. This type of imaging provides excellent tissue visualization in three dimensions and also offers anatomical and functional information from the imaging. MRI imaging data is typically registered and motion-corrected using the dynamic reference frame discussed in this paper.

[0048] Positron emission tomography (PET) imaging is typically a preoperative imaging procedure that exposes the patient to a level of radiation to provide 3D images. PET imaging provides functional information and often requires registration and motion correction using a dynamic reference frame.

[0049] Computed tomography (CT) imaging is often a preoperative technique that exposes patients to limited levels of radiation. However, CT imaging is a very rapid imaging procedure. Multislice CT systems provide 3D images with good resolution and anatomical information. Similarly, CT images are typically registered and require motion correction via a dynamic reference frame.

[0050] Fluorescence fluoroscopy is typically an intraoperative imaging procedure that exposes the patient to a certain amount of radiation to provide two-dimensional or rotated three-dimensional images. Fluorescence fluoroscopy images generally provide good resolution and anatomical information. Fluorescence fluoroscopy images can be registered manually or automatically, and motion corrections need to be considered using a dynamic reference frame.

[0051] Ultrasound imaging is often an intraoperative procedure that uses a non-ionizing field to provide 2D, 3D, or 4D imaging (including anatomical and blood flow information). Ultrasound imaging provides automatic registration and does not require any motion correction.

[0052] The navigation system 100 may include a second imaging device 166, which may be an ultrasound probe, such as an implantable laparoscopic ultrasound probe, an ultrasound probe, etc. The ultrasound probe is configured to generate ultrasound data by emitting sound waves into tissue and receiving reflected sound waves (i.e., ultrasound data). The received reflected sound waves are then processed by workstation 134 to identify various characteristics of the tissue through which the sound waves travel, such as tissue density. The ultrasound probe may be gripped by a robotic instrument or directly coupled to IDU 52. In an embodiment, the second imaging device 166 may be a fluorescence fluoroscopy imager, a magnetic resonance imaging (MRI) imager, a computed tomography (CT) imager, a positron emission tomography (PET) imager, an isocentric fluorescence fluoroscopy imager, a biplane fluorescence fluoroscopy imager, an ultrasound imager, a multislice computed tomography (MSCT) imager, a positron emission tomography-computed tomography (PET / CT) imager, a high-definition computed tomography (HDCT) imager, a dual-source computed tomography imager, a high-frequency ultrasound (HIFU) imager, an optical coherence tomography (OCT) imager, an intravascular ultrasound imager (IVUS), an intraoperative CT imager, an intraoperative MRI imager, a single-photon emission computed tomography (SPECT) imager, or a combination thereof.

[0053] The navigation system 100 may further include an electromagnetic navigation system 144, which includes a transmitting coil array 146, a coil array controller 148, a navigation probe interface 150, a dynamic reference frame 154, and an instrument 50, which may be a flexible instrument with electromagnetic tracking, such as a catheter, endoscope, or any other type of instrument.

[0054] In this embodiment, the entire tracking system 144, or portions thereof, may be incorporated into imaging devices 112 and 166, including workstation 134 and radiation sensor 124. Incorporation into tracking system 144 will provide an integrated imaging and tracking system. Any combination of these components may also be incorporated into imaging system 112, which may similarly include a fluorescence imaging arm or any other suitable imaging device.

[0055] The transmitting coil array 146 is shown as an x-ray receiver 120 attached to arm 116. However, it should be noted that the transmitting coil array 146 can also be positioned in any other location. For example, the transmitting coil array 146 can be positioned at x-ray source 118, inside or on top of operating table 156 located below patient 114, on guardrail associated with operating table 156, or positioned on patient 114 near the navigated area, such as on the patient's chest. The transmitting coil array 146 may include multiple coils, each operable to generate different electromagnetic fields in the navigated area of ​​patient 114, sometimes referred to as patient space.

[0056] The transmitting coil array 146 is controlled or driven by a coil array controller 148. The coil array controller 148 drives each coil in the transmitting coil array 146 in a time-division multiplexing or frequency-division multiplexing manner. In this respect, each coil can be driven individually at different times, or all coils can be driven simultaneously, with each coil driven by a different frequency. When the coils in the transmitting coil array 146 are driven with the coil array controller 148, an electromagnetic field is generated in the area within the patient 114 where the medical procedure is being performed; this area is sometimes referred to as the patient space. The electromagnetic field generated in the patient space induces a current in a sensor 158 located in the device 50. These induced signals from the device 50 are delivered to a navigation probe interface 150 and subsequently forwarded to the coil array controller 148. The navigation probe interface 150 provides all necessary electrical isolation for the navigation system 100. The navigation probe interface 150 also includes amplifiers, filters, and buffers required for direct interface connection with the sensor 158 in the device 50. Alternatively, the device 50 can use a wireless communication channel instead of being directly coupled to the navigation probe interface 150.

[0057] The device 50 may be equipped with at least one, and typically multiple, positioning sensors 158. The device 50 is also typically a maneuverable catheter, which includes: a handle located proximally; and multiple positioning sensors 158 fixed to the catheter body and axially spaced apart from each other along the distal segment of the device 50. Figure 5 As shown, device 50 may include four positioning sensors 158. Positioning sensors 158 are typically formed as electromagnetic receiving coils, such that an electromagnetic field generated by the transmitting coil array 146 induces a current in the electromagnetic receiving coils or sensors 158. Device 50 may also be equipped with one or more sensors operable to sense various physiological signals. For example, device 50 may be provided with electrodes for sensing myocardial potentials or action potentials. Absolute pressure sensors and other electrode sensors may also be included. Device 50 may also be provided with an open lumen to allow delivery of medical devices or drug / cell / genetic agents. For example, device 50 may be used as a guiding catheter for deploying medical leads (such as cardiac leads for pacing and / or defibrillation or tissue ablation). This open lumen can alternatively be used for local delivery of drug formulations, cell or gene therapies.

[0058] The dynamic reference frame 154 of the electromagnetic tracking system 144 is also coupled to the navigation probe interface 150 to forward information to the coil array controller 148. The dynamic reference frame 154 is a small magnetic field detector designed to be fixed to the patient 114, adjacent to the area being navigated, such that any movement of the patient 114 is detected as relative motion between the transmitting coil array 146 and the dynamic reference frame 154. This relative motion is forwarded to the coil array controller 148, which updates the registration correlation and maintains accurate navigation. The dynamic reference frame 154 can be configured as a pair of orthogonally oriented coils, each with the same center; or it can be configured to use any other non-coaxial coil configuration. The dynamic reference frame 154 can be externally fixed to the patient 114, adjacent to the navigation area, such as on the patient's chest (e.g., on the chest). Figure 5 (As shown) or on the patient's back. The dynamic reference frame 154 can be secured to the patient's skin with an adhesive patch. The dynamic reference frame 154 can also be removably attached to a reference marker 160, which is also positioned on the patient's body.

[0059] Alternatively, the dynamic reference frame 154 can use a temporary lead directly attached to the heart, or internally attached to a wall, such as the patient's heart or other soft tissue. This provides greater accuracy because the lead will track regional movements of the heart. Gating will also improve the navigation accuracy of the system 100. It should be further noted that multiple dynamic reference frames 154 can also be employed. For example, an external dynamic reference frame 154 can be attached to the chest of the patient 114 and the back of the patient 114. Since some areas of the body may move more than others due to the movement of the heart or respiratory system, each dynamic reference frame 154 can be appropriately weighted to further improve accuracy. In this respect, the weight of the dynamic reference frame 154 attached to the back can be higher than the weight of the dynamic reference frame 154 attached to the chest, because the dynamic reference frame 154 attached to the back is relatively stationary during movement.

[0060] In addition to electromagnetic tracking, navigation system 100 may also include optical tracking system 170, which may include any suitable camera 172 or camera group, such as one or more infrared cameras (e.g., bifocal cameras), capable of identifying, for example, active and passive tracking markers 174 located within a given measurement volume and visible from the viewpoint of camera 172. Camera 172 may scan the given measurement volume and detect light from markers 174 to identify and determine the position of markers 174 in three dimensions. For example, active markers 174 may include infrared emitting markers activated by electrical signals (e.g., infrared light-emitting diodes (LEDs)), and passive markers 174 may include retroreflective markers that reflect infrared light, for example, emitted by an illuminator on camera 172 or other suitable device (e.g., these retroreflective markers reflect incident IR radiation in the direction of the incident light). Markers 174 may be placed on imaging devices 112 and 166 to allow for their registration and positioning.

[0061] Figure 6A left handle controller 38b, a mirror image of the right handle controller 38a, is shown. This left handle controller can be used to control (i.e., move, activate, etc.) tools, such as instruments 50, laparoscopic cameras 51, and ultrasound probes 166, coupled to the IDU 52 of the robotic arm 40. Each of the handle controllers 38a and 38b includes a handle 701 and a paddle 708, one end (e.g., proximal end) of which is pivotally coupled to the handle 701. The paddle 708 is configured to actuate the function of the instrument 50. The paddle 708 may include a finger sensor (e.g., a touch sensor, capacitive sensor, optical sensor, etc.) configured to detect the presence or movement of a finger. In an embodiment, a finger sensor 704 may be located on any part of the handle controllers 38a and 38b. Each of the handle controllers 38a and 38b may also include a trigger 705a and one or more buttons 705b for activating various functions of the instrument 50. Additionally, each of the handle controllers 38a and 38b may include a gimbal assembly 706 that allows the handle controllers 38a and 38b to move and rotate within the coordinate system of the handle controller 38a. The coordinate system is represented by 3D axis symbols 600, including X-axis 600x, Y-axis 600y, and Z-axis 600z. The gimbal assembly includes multiple frames 710a, 710b, and 710c, interconnected by rotatable joints 711 between each frame 710a, 710b, 710c, the handle 701, and the support frame 712. These joints include encoders or other sensors adapted to measure rotation, the measurements of which are then used as input to control the movement of the device 50 (e.g., pitch, roll, yaw, etc.). In an embodiment, the handle controller 38b may be any other directional input device, such as an analog joystick, directional keys, a touchpad, a trackball, a mouse, etc.

[0062] refer to Figure 7 The display screen 136 of workstation 134 is used to display images obtained from the first imaging device 112 and / or the second imaging device 166. Each display screen (i.e., the first display screen 136a and the second display screen 136b) can display an image from one of the image modalities (i.e., from the first imaging device 112 and the second imaging device 166, respectively). In embodiments, images can be displayed on a single display in an independent window, region, or picture-in-picture format.

[0063] The first display screen 136a displays an image 180a from the first imaging modality (i.e., the first imaging device 112, which may be a fluorometer as described above). The display screen 136a also shows the coordinate system of the first imaging modality, depicted using 3D axis symbols 182, including an X-axis 182x, a Y-axis 182y, and a Z-axis 182z. The orientation of axes 182x, y, and z is obtained using the aforementioned electromagnetic and / or optical tracking system. The 3D axis symbols 182 can be superimposed on the image 180a.

[0064] The second display screen 136b displays an image 180b from the second imaging modality (i.e., the second imaging device 166, which may be an ultrasound probe as described above). The display screen 136b also shows the coordinate system of the second imaging modality, depicted using 3D axis symbols 184, including an X-axis 184x, a Y-axis 184y, and a Z-axis 184z. The orientation of axes 184x, y, and z is obtained using the aforementioned electromagnetic and / or optical tracking system. The 3D axis symbols 184 can be superimposed on the image 180b.

[0065] like Figure 7 As shown, the corresponding coordinate systems of the first and second imaging modes are misaligned due to the different orientations (i.e., misalignment) of the corresponding imaging devices 112 and 166. Furthermore, the coordinate systems of the manual controllers 38a and 38b are also misaligned with either the first or second imaging mode. Consequently, a movement input in one direction at the handle controllers 38a and 38b can cause the conduit 152 to move in completely different directions in the first and second images due to the misalignment.

[0066] Figure 9 A method for using a single imaging modality of navigation system 100 with surgical robot system 10 is illustrated. This method can be embodied in software instructions stored in memory and executable by a processor of workstation 134, surgeon console 30, control tower 20, etc. At step 200, surgical robot system 10 is initialized, which includes attaching instrument 50 to IDU 52 of robotic arm 40. Instrument 50 is controlled by user input via handle controller 38a. In an embodiment, pivoting handle 701 about Y-axis 600y controls yaw motion (e.g., movement between left and right), and pivoting handle 701 about X-axis 600x controls pitch motion (e.g., movement between up and down). Similarly, rotation of Z-axis 600z provides roll control of the instrument. In an embodiment, only some movements of handle controller 38a can be mapped to instrument 50. In other words, instrument 50 can have fewer degrees of freedom than handle controller 38a.

[0067] At step 202, the instrument 50 is advanced to the surgical site, which can be accomplished by moving the retainer 46 or other actuation mechanism of the instrument 50. The first imaging device 112 is also activated to image the surgical site and the instrument as the instrument 50 is advanced. The instrument 50 is advanced until it is within the imaging field of view of the first imaging device 112. At this point, the instrument 50 can move according to the original coordinate system of the handle controller 38a described above with respect to step 200.

[0068] At step 204, the instrument 50 (and extends to the handle controller 38a) is registered to the coordinate system of the first imaging device 112. As described above, various tracking systems (e.g., electromagnetic tracking system 144, optical tracking system 170, etc.) can be used to perform the registration to identify the position and orientation of the first imaging device 112.

[0069] The registration status can be displayed using 3D axis symbols, such as... Figure 7 As shown. The coordinate system of the first imaging device 112 can be displayed as a first 3D axis symbol 194, which can be superimposed on the image 190 from the first imaging device 112. In an embodiment, the first 3D axis symbol 194 can be displayed above the end effector or distal portion of the instrument 50. The image 190 also includes a second 3D axis symbol 196, which represents the coordinate system of the handle controller 38a. The second 3D axis symbol 196 can be displayed in any part of the image 190 or the display screen 136 (e.g., the upper corner). Figure 8 As shown in the table, the coordinate systems of the first imaging device 112 and the handle controller 38a were not aligned before registration. Once registration is complete, at step 206, the coordinate system of the handle controller 38a is remapped to align with the coordinate system of the first imaging device 112. Registration can be represented by rotating the second 3D axis symbol 196 to match the first 3D axis symbol 194. The timing of axis alignment can occur simultaneously with the alignment, or within the time period during which alignment occurs, for example, with a delay of 0.5 seconds or longer.

[0070] The registration process can be performed automatically by the surgical robot system 10, for example, based on the determination that the instrument 50 has reached the surgical site. In an embodiment, the registration process can be initiated manually in response to a user command. Once the registration process is complete, the robot system 10 can output a notification indicating that the instrument 50 can now be controlled by the registered handle controller 38a at step 208.

[0071] To further assist in the movement of the device 50, the GUI can be displayed as an overlay on the image 190. The GUI can show the expected movement of the device 50 before its actual movement. In response to movement input via the handle controller 38a, the system 10 can display an indicator, which can be a virtual representation 50' of the device 50 (indicating where the device 50 will move) and / or an arrow 198 corresponding to that movement. In an embodiment, the GUI can display a confirmation prompt along with the movement indicator, requiring the user to confirm that the indicated movement corresponds to the expected movement.

[0072] Figure 10 Methods for using various imaging modalities with navigation system 100 are illustrated. These methods can also be embodied in software instructions stored in memory and executable by processors of workstation 134, surgeon console 30, control tower 20, etc. At step 300, surgical robot system 10 is initialized, which includes attaching instrument 50 to robotic arm 40 via IDU 52. Instrument 50 is controlled by user input via handle controller 38a, as described above regarding... Figure 9 The method described.

[0073] At step 302, the instrument 50 is advanced to the surgical site, which can be accomplished by moving the retainer 46 or other actuation mechanisms of the instrument 50. The first imaging device 112 and the second imaging device 166 are also activated to image the surgical site and the instrument as the instrument 50 is advanced, providing bimodal imaging (e.g., fluoroscopy and ultrasound). The instrument 50 is advanced until it is within the imaging field of view of the first imaging device 112 and the second imaging device 166. At this point, the instrument 50 can move according to the original coordinate system of the handle controller 38a described above with respect to step 300.

[0074] At step 304, the instrument 50 is registered to the coordinate system of each of the first imaging device 112 and the second imaging device 166. As described above, various tracking systems (e.g., electromagnetic tracking system 144, optical tracking system 170, etc.) can be used to perform the registration to identify the position and orientation of the instrument 50 relative to the first imaging device 112 and the second imaging device 166.

[0075] The registration status can be displayed using 3D axis symbols, such as... Figure 8 As shown. A first image 180a from the first imaging device 112 illustrates the coordinate system of the first imaging mode, which is depicted as an overlay on image 180a using 3D axis symbols 182. A second image 180b from the second imaging device 166 illustrates the coordinate system of the second imaging mode, which is depicted using 3D axis symbols 184. The orientation of the coordinate systems of the first and second modes is obtained using the aforementioned electromagnetic and / or optical tracking system.

[0076] At step 306, the coordinate systems of the first and second image modes are aligned. This can be achieved by moving one or both of the imaging devices 112 and 166 until the coordinate systems are aligned. The movement of imaging devices 112 and 166 is accomplished by robotic arms 116 or 40, respectively. The alignment of the coordinate systems can be indicated by aligning the 3D axis symbols 182 and 184 at step 308. After aligning the two imaging modes, the following can be performed: Figure 9 The method is to align the motion control system (i.e., the handle controller 38a) with the two imaging modalities described above.

[0077] Figure 9 and Figure 10 The method is applicable during cardiac procedures, where navigation and guidance of flexible devices are particularly challenging. To further assist in such procedures, navigation system 100 is also configured to compensate for the effects of respiration and cardiac activity, which typically complicate device navigation within and around the heart. Various patient parameters can be monitored, such as blood pressure using a pressure monitor, blood oxygen saturation using a pulse oximeter, and cardiac activity using an electrocardiogram monitor. Imaging devices 112 and 166 can operate based on the measured patient parameters.

[0078] The imaging process can be synchronized with the patient's physiological activity, allowing imaging devices 112 and / or 166 to capture images at this desired synchronization time. For example, a physiological change could be a heartbeat as identified by an ECG. This allows image acquisition at the end of diastole. Diastole is the time between two contractions of the atria or ventricles, during which blood flows from the systemic circulation and lungs into the relaxed chambers. Diastole is typically measured as blood pressure at the moment of maximum cardiac relaxation. This makes this time period suitable for imaging. In another embodiment, image compensation can be achieved by displaying images acquired during a specific time period (e.g., diastole). Thus, imaging can be performed continuously, but only image frames captured during certain time periods can be displayed.

[0079] It should be understood that various modifications can be made to the embodiments disclosed herein. Therefore, the above description should not be construed as limiting, but rather as illustrative of the various embodiments only. Other modifications will be contemplated by those skilled in the art within the scope and spirit of the appended claims.

[0080] The following examples illustrate the techniques described in this article.

[0081] Example 1. A surgical robot system comprising: a first robotic arm including a first instrument drive unit and an instrument coupled to and actuable by the first instrument drive unit; a surgeon's console including a handle controller for receiving user input to move the instrument in a first coordinate system; a second robotic arm including a second instrument drive unit and an imaging device coupled to and actuable by the second instrument drive unit, the imaging device being configured to capture an image of the instrument in a second coordinate system different from the first coordinate system using an imaging modality; a tracking system for acquiring tracking data of the instrument and the imaging device; and a processor configured to: register the instrument in the second coordinate system based on the tracking data; align the first coordinate system with the second coordinate system; and receive input for controlling the instrument in the second coordinate system.

[0082] Example 2. The surgical robot system according to Example 1, wherein the imaging device is selected from the group consisting of an ultrasound probe and a fluorescence fluoroscopy device, and the instrument is selected from the group consisting of a catheter and a flexible endoscope.

[0083] Example 3. The surgical robot system according to Example 1, wherein the handle controller includes multiple directional inputs.

[0084] Example 4. The surgical robot system according to Example 3, wherein the processor is configured to map the plurality of directional inputs from the first coordinate system to the second coordinate system.

[0085] Example 5. The surgical robot system according to Example 1, wherein the surgeon's console further includes a display screen for displaying images captured by the imaging device.

[0086] Example 6. The surgical robot system according to Example 5, wherein the processor is configured to display a first 3D axis symbol corresponding to the first coordinate system and a second 3D axis symbol corresponding to the second coordinate system.

[0087] Example 7. The surgical robot system according to Example 6, wherein the processor is configured to align the second 3D axis symbol with the first 3D axis symbol when the first coordinate system is aligned with the second coordinate system.

[0088] Example 8. The surgical robot system according to Example 1, wherein the processor is configured to display a movement indicator in response to these inputs, indicating where the instrument will be moved.

[0089] Example 9. The surgical robot system according to Example 8, wherein the movement indicator is at least one of a virtual representation or an arrow.

[0090] Example 10. A method for controlling an imaging system using a surgical robot system, the method comprising: receiving user input at a handle controller for moving an instrument in a first coordinate system, wherein the instrument is coupled to a first instrument drive unit of a first robotic arm; capturing an image of the instrument in a second coordinate system using an imaging device with an imaging modality, wherein the imaging device is coupled to a second instrument drive unit of a second robotic arm; obtaining tracking data of the instrument and the imaging device using a tracking system; registering the instrument in the second coordinate system based on the tracking data; aligning the first coordinate system with the second coordinate system; and receiving input for controlling the instrument in the second coordinate system.

[0091] Example 11. The method according to Example 10, wherein the handle controller includes multiple directional inputs.

[0092] Example 12. The method described in Example 11 further includes: mapping the plurality of directional inputs from the first coordinate system to the second coordinate system.

[0093] Example 13. The method according to Example 10 further includes: displaying an image captured by the imaging device on a display screen.

[0094] Example 14. The method according to Example 13 further includes: displaying a first 3D axis symbol corresponding to the first coordinate system and a second 3D axis symbol corresponding to the second coordinate system.

[0095] Example 15. The method according to Example 14 further includes: aligning the second 3D axis symbol with the first 3D axis symbol when the first coordinate system is aligned with the second coordinate system.

[0096] Example 16. A method for controlling a medical device in multiple different imaging modalities, the method comprising: capturing a first image of the device in a first coordinate system using a first imaging device in a first imaging modal, wherein the first imaging device is coupled to a first device drive unit of a first robotic arm; capturing a second image of the device in a second coordinate system using a second imaging device in a second imaging modal, wherein the second imaging device is coupled to a second device drive unit of a second robotic arm, the second imaging modal being different from the first imaging modal, and the second coordinate system being different from the first coordinate system; obtaining tracking data of the device, the first imaging device, and the second imaging device using a tracking system; registering the device in the first coordinate system and the second coordinate system based on the tracking data; and aligning the first coordinate system with the second coordinate system.

[0097] Example 17. The method according to Example 16 further includes: displaying the first image captured by the first imaging device and the second image captured by the second imaging device side by side.

[0098] Example 18. The method according to Example 17 further includes: displaying a first 3D axis symbol corresponding to the first coordinate system and a second 3D axis symbol corresponding to the second coordinate system.

[0099] Example 19. The method according to Example 18 further includes: aligning the second 3D axis symbol with the first 3D axis symbol when the first coordinate system of the first imaging device is aligned with the second coordinate system of the second imaging device.

[0100] Example 20. The method according to Example 16, wherein aligning the first coordinate system with the second coordinate system includes: moving at least one of the first robotic arm or the second robotic arm.

Claims

1. A surgical robot system (10), the surgical robot system comprising: A first robotic arm (40) includes a first instrument drive unit (52) and an instrument (50) connected to and actuated by the first instrument drive unit; A surgical console (30) includes a handle controller (38a, 38b) for receiving user input to move the instrument in a first coordinate system; The second robotic arm (40, 116) includes a second instrument drive unit (52) and an imaging device (112, 166), the imaging device being coupled to and actuated by the second instrument drive unit, and the imaging device being configured to capture an image of the instrument in a second coordinate system different from the first coordinate system using an imaging mode. Tracking system (144), the tracking system being used to acquire tracking data of the instrument and the imaging device; and Processor (21a), which is configured to: Based on the tracking data, the instrument is registered in the second coordinate system; Align the first coordinate system with the second coordinate system; as well as Receive the input that controls the device in the second coordinate system.

2. The surgical robot system according to claim 1, wherein, The imaging device is selected from the group consisting of an ultrasound probe and a fluorescence fluoroscopy instrument, and the instrument is selected from the group consisting of a catheter and a flexible endoscope.

3. The surgical robot system according to any of the preceding claims, wherein, The handle controller includes multiple directional inputs.

4. The surgical robot system according to claim 3, wherein, The processor is configured to map the multiple directional inputs from the first coordinate system to the second coordinate system.

5. The surgical robot system according to any of the preceding claims, wherein, The surgeon's console further includes displays (32, 34) for displaying images captured by the imaging device.

6. The surgical robot system according to claim 5, wherein, The processor is configured to display a first 3D axis symbol (194) corresponding to the first coordinate system and a second 3D axis symbol (196) corresponding to the second coordinate system.

7. The surgical robot system according to claim 6, wherein, The processor is configured to align the second 3D axis symbol with the first 3D axis symbol when the first coordinate system is aligned with the second coordinate system.

8. The surgical robot system according to any of the preceding claims, wherein, The processor is configured to display a movement indicator (50', 198) in response to these inputs, indicating where the instrument will move.

9. The surgical robot system according to claim 8, wherein, The movement indicator is at least one of a virtual symbol (50') or an arrow (198).

10. A method for controlling an imaging system using a surgical robot system, the method comprising: User input for moving the instrument (50) in a first coordinate system is received at the handle controller (38a, 38b), wherein the instrument is coupled to the first instrument drive unit (52) of the first robotic arm (40). The imaging device (112, 166) captures an image of the instrument in the second coordinate system using an imaging mode, wherein the imaging device is connected to the second instrument drive unit (52) of the second robotic arm (40, 116). The tracking system is used to obtain tracking data for the instrument and the imaging device; Based on the tracking data, the instrument is registered in the second coordinate system; Align the first coordinate system with the second coordinate system; and Receive the input that controls the device in the second coordinate system.

11. The method according to claim 10, wherein, The handle controller includes multiple directional inputs.

12. The method of claim 11, further comprising mapping the plurality of directional inputs from the first coordinate system to the second coordinate system.

13. The method according to any of the preceding claims further includes displaying an image captured by the imaging device on a display screen (32, 34).

14. The method according to claim 13, further comprising displaying a first 3D axis symbol (194) corresponding to the first coordinate system and a second 3D axis symbol (196) corresponding to the second coordinate system.

15. The method of claim 14, further comprising aligning the second 3D axis symbol with the first 3D axis symbol when the first coordinate system is aligned with the second coordinate system.

Images