Combined multi-port and endoluminal robotic surgical procedure with joint coordinate system for registration, planning, and navigation

Abstract

A robotic surgical system and method including a navigation catheter including a sensor and for endoluminal navigation, a robotic arm having a laparoscopic surgical instrument and a second sensor, and a computing system including a processor and a memory, the memory storing instructions that when executed by the processor cause the processor to detect a position and orientation of the catheter as it is navigated into a luminal network towards a target, register the luminal network to a three-dimensional (3D) model of the luminal network and a pathway plan, display on a user interface a representation of a distal portion of the catheter, determine that the catheter is in proximity to the target, detect a position and orientation of the second sensor; and display on the user interface a representation of the laparoscopic surgical instrument's position and orientation relative to the position and orientation of the catheter.

Description

A0012629W001COMBINED MULTI-PORT AND ENDOLUMINAL ROBOTIC SURGICAL PROCEDURE WITH JOINT COORDINATE SYSTEM FOR REGISTRATION, PLANNING, AND NAVIGATIONCROSS-REFERENCE TO RELATED APPLICATION

[0001] This application claims the benefit of U.S. Provisional Patent Application Serial No. 63 / 691,787, filed September 6, 2024, the entire content of which is incorporated herein by reference.BACKGROUNDTechnical Field

[0002] This disclosure relates to the field of navigating medical devices within a patient, and in particular, generating and utilizing endoluminal robotics and laparoscopic robotics in a joint coordinate system to enable planning and performance of procedures.Description of Related Art

[0003] There are several commonly applied medical methods, such as endoscopic procedures or minimally invasive procedures, for treating various maladies affecting organs including the liver, brain, heart, lungs, gall bladder, kidneys, and bones. Often, one or more imaging modalities, such as magnetic resonance imaging (MRI), ultrasound imaging, computed tomography (CT), cone-beam computed tomography (CBCT), or fluoroscopy (including 3D fluoroscopy) are employed by clinicians to identify and navigate to areas of interest within a patient and ultimately a target for biopsy or treatment. In some procedures, pre-operative scans are utilized for target identification and intraoperative guidance. For example, in an endoscopic approach, navigating to areas of interest within a patient is enabled with the use of previously acquired MRI or CT image data that may be used to generate a three-dimensional (3D) rendering, model, or volume of the particular organ, such as the lungs. The 3D model provides guidance including pathway plans that are displayed in the 3D model to navigate the luminal network of the organ and arrive at an identified target that is also displayed in the 3D model.

[0004] To assist in the performance of these procedures a tracking system (e.g., electromagnetic, shape sensing, or image-based guidance, etc.) can be employed. Utilizing one of these tracking systems, an endoluminal catheter is driven (either manually or robotically)A0012629W001 within the boundaries of the luminal network, and by comparing sensor data the location of the catheter within the patient can be registered to the pre-procedure models and images and its location in the patient updated as it is moved along a desired path to a target.

[0005] In contrast with endoluminal systems (e.g., endoluminal robots) where the catheterbased system is typically constrained within the luminal network (e.g. airways, blood vessels, etc.) laparoscopic robotic systems typically are not so constrained. Instead, the laparoscopic robotic system employs one or more robotic arms each of which is configured to insert and manipulate an end effector within the patient through a laparoscopic port placed in an incision in the patient. The incisions may be in the abdomen or in the thoracic cavity depending on the location of the surgery.

[0006] While pre-procedural images are regularly acquired to assist in planning the laparoscopic surgical approach, these plans are typically high level identifying the location of a tumor or lesion or other feature of the patient that appears in the images. However, due to the differences in positioning of a patient during imaging and during a laparoscopic procedure, among other factors, the planning is often of a general nature and the performance of the procedure relies on the experience and expertise of the surgeon to identify the same tissue via a laparoscope as seen in the pre-procedure images. This has been referred to as keyhole surgery not least because the field of view through a laparoscope can be analogized to the field of view when looking through a keyhole. Similarly, the placement of the incisions in which the ports are placed relies on experience and a generalized understanding of desirable placements for gaining access to the desired tissue.

[0007] Accordingly, improvements enabling more effective and accurate pre-procedure planning and greater refence to the pre-procedure plans of laparoscopic procedures are desirable.SUMMARY

[0008] A system of one or more computers can be configured to perform particular operations or actions by virtue of having software, firmware, hardware, or a combination of them installed on the system that in operation causes or cause the system to perform the actions. One or more computer programs can be configured to perform particular operations or actions by virtue of including instructions that, when executed by data processing apparatus, cause the apparatus to perform the actions.

[0009] One general aspect of the disclosure is directed to a robotic surgical system including a navigation catheter having a first sensor and configured for endoluminalA0012629W001 navigation; a robotic arm having a laparoscopic surgical instrument and a second sensor; and a computing system including a processor and a memory, the memory storing instructions that when executed by the processor cause the processor to: detect a position and orientation of the first sensor as the catheter is navigated into a luminal network towards a target; register the luminal network to a three-dimensional (3D) model of the luminal network and a pathway plan for navigation of the luminal network; display on a user interface a representation of a distal portion of the catheter as the catheter is navigated in the luminal network along the pathway plan; determine that the catheter is in proximity to the target; detect a position and orientation of the second sensor; and display on the user interface a representation of the laparoscopic surgical instrument’s position and orientation relative to the position and orientation of the catheter. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the methods.

[0010] Implementations may include one or more of the following features. The robotic surgical system where the navigation catheter is operably connected to a robotic system for advancement and articulation. The first sensor and second sensor are electromagnetic (EM) sensors. One or more of the EM sensors is a tunnel magnetoresistance (TMR) sensor. The robotic surgical system may include an EM field generator. The instructions when executed by the processor cause the processor to: deform the 3D model based in part on the detected change in position. The instructions when executed by the processor cause the processor to: detect a change in position of a patient after navigation of the catheter in proximity to the target. The instructions when executed by the processor cause the processor to: receive inputs identifying one or more aspects of a laparoscopic surgical procedure. The inputs identify one or more of a fissure, a blood vessel, a staple line, a resection line, a critical structure, a no-go zone, or a margin. The instructions when executed by the processor cause the processor to: generate an alert when the surgical instrument is in proximity to one of the aspects of the laparoscopic surgical procedure. Implementations of the described techniques may include hardware, a method or process, or computer software on a computer-accessible medium.

[0011] A further aspect of the disclosure is directed to a method of performing a combined endoluminal and laparoscopic procedure. The method includes detecting a position and orientation of a catheter as it is navigated into a luminal network towards a target; registering the luminal network with a three-dimensional (3D) model of the luminal. The method also includes network and a pathway plan for navigation of the luminal network;A0012629W001 determining that the catheter is in proximity to the target, detecting a position and orientation of a robotic laparoscopic surgical instrument, and navigating the robotic laparoscopic to the target. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the methods.

[0012] Implementations may include one or more of the following features. The method may include displaying a relative position of the robotic laparoscopic surgical instrument and the catheter in proximity to the target on a user interface. The method may include detecting a position and orientation of the catheter or the laparoscopic surgical instrument within an electromagnetic field. The method may include receiving imaging of the luminal network and generating the 3D model from the imaging. The method may include detecting an electromagnetic field to determine a position and orientation of the robotic laparoscopic surgical instrument or a position and orientation of the catheter. The method may include detecting a change in position of a patient after navigation of the catheter in proximity to the target. The method may include displaying laparoscopic surgical plan data on a deformed 3D model. The method may include depicting a relative position and orientation of the robotic laparoscopic surgical instrument and the position and orientation of the catheter in intraprocedural images. The method may include displaying laparoscopic surgical plan data on intraprocedural images. The laparoscopic plan data includes one or more of a fissure, a blood vessel, a staple line, a resection line, a critical structure, a no-go zone, or a margin. Implementations of the described techniques may include hardware, a method or process, or computer software on a computer-accessible medium.BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Various aspects and embodiments of the disclosure are described hereinbelow with references to the drawings, wherein:

[0014] FIG. 1 A is a schematic view of a robotic surgical system in accordance with the disclosure;

[0015] FIG. IB is a schematic view of a robotic luminal network navigation system in accordance with the disclosure;

[0016] FIG. 2 is a schematic view of a manual luminal navigation system in accordance with the disclosure;

[0017] FIG. 3 is a view of a surgical suite depicting a user interface in accordance with aspects of the disclosure;A0012629W001

[0018] FIG. 4 depicts a user interface during endoluminal navigation in accordance with the disclosure;

[0019] FIG. 5 depicts a user interface during endoluminal navigation in accordance with the disclosure;

[0020] FIG. 6 is a flow chart depicting a method in accordance with the disclosure; and

[0021] FIG. 7 is a schematic view of a computing system in accordance with the disclosure.DETAILED DESCRIPTION

[0022] The disclosure is directed to a system including an endoluminal (e.g., robotic) platform combined with a laparoscopic robotic platform. Both the endoluminal platform and the laparoscopic robotic platform may include sensors (e.g., EM sensors) that allow for a position in space of a catheter, end effector, or tool to be determined. Further, through the use of the endoluminal robotic platform, and its attendant sensors, pre-procedural images of the patient can be registered to the position and orientation of the patient. With the registration of the pre-procedure images and the patient using the sensors of the endoluminal robotic platform, the location and orientation of target tissue in the patient can be determined. The laparoscopic robotic platform can employ the system sensors and operate in the same coordinate system (e.g., an EM coordinate system) to inform the laparoscopic robotic platform of where to make incisions for placement of laparoscopic ports, what tissues are between the location of any of the end effectors associated with one of the laparoscopic arms, and a pathway to arrive at the target tissue, where to perform resection of tissues, and the location and orientation of relevant tissues to avoid during the procedure. In some instances, the location of the target tissue can be identified by the location of the sensor associated with the endoluminal catheter, by dye marking via the endoluminal catheter, or through the use of light emanating from the endoluminal catheter. These and other features of the disclosure are described in greater detail below.

[0023] Laparoscopic Surgical Robotics

[0024] FIG. 1 A depicts a robotic surgical system 10 including a control tower 20, which is connected to all of the components of the robotic surgical system 10 including a surgeon console 30A and one or more mobile carts 60. Each of the mobile carts 60 includes a robotic arm 40 having a surgical instrument 50 removably coupled thereto. The robotic arms 40 also couple to the mobile carts 60. The robotic surgical system 10 may include any number of mobile carts 60 and / or robotic arms 40.A0012629W001

[0025] The surgical instrument 50 is configured for use during minimally invasive surgical procedures (e.g., laparoscopic) or for catheter-based intraluminal diagnostic, therapeutic, and surgical procedures (described in greater detail in connection with FIG. IB). The surgical instrument 50 may include an end effector 49 such as, for example, an electrosurgical forceps configured to seal tissue by compressing tissue between jaw members and applying electrosurgical current thereto, a surgical stapler including a pair of jaws configured to grasp and clamp tissue while deploying a plurality of tissue fasteners, e.g., staples, and cutting stapled tissue, a surgical clip applier including a pair of jaws configured apply a surgical clip onto tissue or other end effectors without departing from the scope of the disclosure. Alternatively, the surgical instrument 50 may include a robotic actuated catheter including an articulation mechanism (e.g., one or more pull-wires, tubes, or tendons) to alter the shape of the catheter. In addition, the robotic arm 40 may be employed to advance or retract the catheter during a laparoscopic or endoluminal (or combined) procedure.

[0026] In accordance with various embodiments of the present disclosure, one of the robotic arms 40 may include a laparoscopic camera 51 configured to capture video of the surgical site. The laparoscopic camera 51 may be, for example, a stereoscopic endoscope configured to capture two side-by-side (i.e., left and right) images of the surgical site to produce a video stream of the surgical scene. The laparoscopic camera 51 is coupled to an image processing device, which may be disposed, for example, within the control tower 20. The image processing device may be any computing device configured to receive the image feed from the laparoscopic camera 51 and output the processed images or video stream.

[0027] The surgeon console 30A includes a first screen 32, which displays a video feed of the surgical site provided by camera 51 of the surgical instrument 50 disposed on the robotic arm 40, and a second screen 34, which displays a user interface for controlling the robotic surgical system 10. The first screen 32 and second screen 34 may be touchscreens allowing for displaying various graphical user inputs.

[0028] The surgeon console 30A also includes a plurality of user interface devices, such as, for example, foot pedals 36 and a pair of hand controllers 38a and 38b which a user may use to remotely control robotic arms 40 and / or surgical instruments 50. The surgeon console 30A further includes an armrest 33 used to support clinician's arms while operating the hand controllers 38a and 38b. As an alternative, or in addition to surgeon console 30 A, as depicted in FIG. IB, the robotic arms 40, and other elements of the system, including the surgical instrument 50 may be controlled via surgeon console 30B, including in one example, via aA0012629W001 handheld controller 38c, which may be connected to the console 30B in a wired or wireless manner.

[0029] The control tower 20 includes a screen 23, which may be a touchscreen, and outputs various graphical user interfaces (GUIs). In accordance with various aspects of the disclosure, the control tower 20 also acts as an interface between the surgeon console 30 and one or more robotic arms 40. In particular, the control tower 20 is configured to control the robotic arms 40, such as to move the robotic arms 40 and the corresponding surgical instrument 50, based on a set of programmable instructions and / or input commands from the surgeon console 30, in such a way that robotic arms 40 and the surgical instrument 50 execute a desired movement sequence in response to input from the surgeon console 30A. For example, in accordance with one embodiment, surgeon console 30Amay include the foot pedals 36 and the hand controllers 38a and 38b. The foot pedals 36 may be used to enable and lock the hand controllers 38a and 38b, repositioning camera movement and surgical instrument activation / deactivation. In particular, the foot pedals 36 may be used to perform a clutching action on the hand controllers 38a and 38b. Clutching may be initiated by pressing one of the foot pedals 36, which disconnects (i.e., prevents movement inputs) the hand controllers 38a and / or 38b from the robotic arm 40 and corresponding surgical instrument 50 or camera 51 attached thereto. This allows the user to reposition the hand controllers 38a and 38b without moving the robotic arm(s) 40 and the instrument 50 and / or camera 51. This is useful when reaching control boundaries of the surgical space. In accordance with another embodiment of the present disclosure, the surgeon console 30A or 30B may communicate with and control one or more robotic arms 40 and surgical instruments 50 directly, without using the control tower 20 as an interface.

[0030] Each of the control tower 20, the surgeon console 30, and the robotic arm 40 includes a respective computer 21, 31, 41. The computers 21, 31, 41 are interconnected to each other using any suitable communication network based on wired or wireless communication protocols. The term “network,” whether plural or singular, as used herein, denotes a data network, including, but not limited to, the Internet, Intranet, a wide area network, or a local area network, and without limitation as to the full scope of the definition of communication networks as encompassed by the present disclosure. Suitable protocols include, but are not limited to, transmission control protocol / internet protocol (TCP / IP), datagram protocol / intemet protocol (UDP / IP), and / or datagram congestion control protocol (DC). Wireless communication may be achieved via one or more wireless configurations, such as, e.g., radio frequency, optical, Wi-Fi, Bluetooth (an open wireless protocol for exchanging data over shortA0012629W001 distances, using short length radio waves, from fixed and mobile devices, creating personal area networks (PANs), ZigBee® (a specification for a suite of high level communication protocols using small, low-power digital radios based on the IEEE 122.15.4-1203 standard for wireless personal area networks (WPANs)).

[0031] The computers 21, 31, 41 may include any suitable processor (not shown) operably connected to a memory (not shown), which may include one or more of volatile, non-volatile, magnetic, optical, or electrical media, such as read-only memory (ROM), random access memory (RAM), electrically-erasable programmable ROM (EEPROM), non-volatile RAM (NVRAM), or flash memory. The processor may be any suitable processor (e.g., control circuit) adapted to perform the operations, calculations, and / or set of instructions described in the present disclosure including, but not limited to, a hardware processor, a field programmable gate array (FPGA), a digital signal processor (DSP), a central processing unit (CPU), a microprocessor, and combinations thereof. Those skilled in the art will appreciate that the processor may be substituted with any logic processor (e.g., control circuit) adapted to execute algorithms, calculations, and / or set of instructions described herein.

[0032] Endoluminal Navigation

[0033] FIG. IB is a perspective view of an exemplary system for facilitating robotic endoluminal navigation of a medical device, e.g., a catheter, to a soft-tissue target via airways of the lungs. System 100 may be further configured to construct fluoroscopic based three-dimensional volumetric data of the target area from 2D fluoroscopic images to confirm navigation to a desired location. Other intraprocedural imaging modalities may also be employed including CBCT, ultrasound, laparoscopic cameras, and others. System 100 may be further configured to facilitate the approach of a medical device to the target area by using, for example, Electromagnetic Navigation (EMN) and for determining the location of a medical device with respect to the target. The EMN system may employ a variety of sensor technologies including without limitation air-coil sensors, tunnel magnetoresistance (TMR), and others without departing from the scope of the disclosure. Though described in connection with EMN, other systems for intraluminal and lung navigation are considered within the scope of the disclosure including shape sensing technology (e.g., Fiber-bragg gratings) which detect the shape of the distal portion of the catheter and match that shape to the shape of the luminal network in a 3D model.

[0034] With respect to both FIGS. 1 Aand IB each of the various components of the system may be connected via either a wired or wireless connection, or combinations of both, withoutA0012629W001 departing from the scope of the disclosure. In this manner the computers 21, 31, 41 or console 30B can control the robotic arms 40 and the surgical instruments 50 to perform the desired procedure whether endoluminal, laparoscopic, or a combination of both.

[0035] In FIG. IB, the surgical instrument 50 includes a catheter 102. The catheter 102 is inserted into the luminal network of the patient P (e.g., via the nose or mouth). The catheter 102 includes one or more sensors 104 used for determining a position and orientation of distal portion of the catheter 102 within the patient. In one example, this position and orientation may be determined with reference to a coordinate system defined by an electromagnetic (EM) field generated by a magnetic field generator 106. The magnetic field generator 106 is a boardshaped device including two or more magnetic field generating antennae that is typically placed beneath the patient. In one embodiment the catheter 102 includes imaging capabilities. The catheter 102 may also be configured to receive a locatable guide (LG), camera, biopsy, or therapy tool (not specifically shown). In one example, the locatable guide is a second catheter that may include a sensor or a camera and may be inserted into catheter 102 and locked into position. System 100 generally includes an operating table 108 configured to support a patient P for insertion of the catheter 102 through patient P’s mouth and into patient P’s airways. The camera on either the catheter 102 or the locatable guide may be operably connected to a display 110 on the console 30B to display live video and images in one or more user interfaces 112. Also connected to the console 30B are the magnetic field generator 106 and a plurality of reference sensors 114. The magnetic field generator 106 may include a plurality of fiducial markers that are observed in fluoroscopic or cone beam CT images of the patient. The console 30B includes a computing device 116 including software and / or hardware used to facilitate identification of a target, pathway planning to the target, navigation of a medical device to the target, and / or confirmation and / or determination of placement of catheter 102, or a suitable device therethrough, relative to the target. In addition, the computing device generates signals causing the magnetic field generator 106 to generate magnetic fields as well as other functions.

[0036] In accordance with aspects of the disclosure, the visualization of intra-body navigation of a medical device (e.g., a catheter, LG, biopsy, or therapy tool), towards a tumor or lesion, may be a portion of a larger workflow of a navigation system. An imaging device 118 capable of acquiring images or video of the patient P is also included in this particular aspect of system 100. The images, sequence of images, or video captured by imaging device 118 may be stored within imaging device 118 or transmitted to computing device 116 for storage, processing, and display. Additionally, imaging device 118 may move relative to theA0012629W001 patient P so that images may be acquired from different angles or perspectives relative to patient P to create a sequence of images, such as a fluoroscopic video. The pose of imaging device 118 relative to patient P while capturing the images may be estimated via the fiducial markers incorporated within the magnetic field generator 106. The markers are positioned under patient P, between patient P and operating table 108 and between patient P and a radiation source or a sensing unit of imaging device 118. The markers incorporated with the magnetic field generator 106 may be two separate elements which may be coupled in a fixed manner or alternatively may be manufactured as a single unit. Imaging device 118 may include a single imaging device or more than one imaging device. The imaging device 118 may be for example, a fluoroscopic imaging, an ultrasound imaging device, an intraprocedural CBCT imaging device, or an intraprocedural PET imaging device.

[0037] Computing device 116 may be any suitable computing device including a processor and storage medium, wherein the processor is capable of executing instructions stored on the storage medium. Computing device 116 may further include a database configured to store patient data, image data sets including CT images, CBCT images, fluoroscopic images and video, fluoroscopic 3D reconstruction, navigation plans, and other such image data. Although not explicitly illustrated, computing device 116 may include inputs, or may otherwise be configured to receive, image data sets and other data described herein. Additionally, computing device 116 includes a display configured to display images, 3D models, and other data in one or more graphical user interfaces. Computing device 116 may be connected to one or more networks through which one or more databases (e.g., image databases) may be accessed.

[0038] For use in a navigation phase, a suitable system for determining position and orientation of a distal portion of the catheter 102 (e.g., magnetic field generator 106 and software on the computing device 116), is utilized for performing registration of the images and the pathway for navigation with the patient’s luminal network. In one aspect of the disclosure, magnetic field generator 106 is positioned beneath patient P. The magnetic field generator 106 generates an electromagnetic field around at least a portion of the patient P within which the position of the plurality of reference sensors 114 and the sensor 104 can be determined by an application running on the computing device 116. Registration is generally performed to coordinate locations of the three-dimensional model and two-dimensional images, with the patient P’s airways as observed in the images captured by the camera (either on the catheter 102 or locatable guide) and allow for the navigation phase to be undertaken with knowledge of the location of the sensor 104 within the body of the patient.A0012629W001

[0039] As depicted in FIG. IB, the catheter 102 may be navigated using functionality from the robotic arm 40 and mobile cart 60 as well as functionality in the surgical instrument 50 (e.g., rotation, advancement, and articulation). As noted above, the robotic arms 40, surgical instrument 50, and catheter 102 may be controlled via surgeon console 30B, including in one example, via a handheld controller 38c, which may be connected to the console 30B in a wired or wireless manner. Alternatively, the surgeon console 30B may include controls (e.g., track ball, buttons, etc.) to control the robotic arms 40, surgical instrument 50, and catheter 102. Still further, as noted above, the catheter 102 may include an endoluminal camera. The endoluminal camera captures images of the endoluminal pathway as the catheter 102 is advanced towards a target. The endoluminal camera may be a permanent feature of the catheter 102, or a removable camera (e.g., on the locatable guide) that is advanced into a working channel of the catheter 102. One or more fiber-optic filaments or light pipes may be employed to carry light to the end of the catheter 102 from a light source that remains outside of the patient. An active pixel sensor may be located on a distal portion of the catheter (e.g., a CMOS or CCD image sensor) for the formation of live images from captured light reflected by the patient’s tissues. Alternatively, both the light source and image forming components may also be positioned at a distal portion of the catheter 102, or both may remain outside of the patient with the fiberoptic filaments carrying light both from the light source and to the image forming components.

[0040] Registration of the patient P’s location on the magnetic field generator 106 may be performed by moving sensor 104 through the airways of the patient P. More specifically, data pertaining to locations of sensor 104 and the reference sensors 114 in the magnetic field generated by the magnetic field generator 106 is recorded using an application stored in the memory of the computing device 116. A shape resulting from this location data is compared to an interior geometry of passages of a 3D model generated during a planning phase of the procedure (e.g., from pre-procedure CT images), and a location correlation between the shape and the 3D model based on the comparison is determined, e.g., utilizing the software on computing device 116. In addition, the software identifies non-tissue space (e.g., air filled cavities) in the three-dimensional model. The software aligns, or registers, an image representing a location of sensor 104 with the 3D model and / or two-dimensional images generated from the 3D model, which are based on the recorded location data and an assumption that catheter 102 remains located in non-tissue space in patient P’s airways. Alternatively, a manual registration technique may be employed by navigating the catheter 102 with the sensorA0012629W001104 to pre-specified locations in the lungs of the patient P and manually correlating the detected locations (and / or images from the camera) to the 3D model.

[0041] Though described herein with respect to EMN systems using EM sensors, including, e.g., TMR sensors, the instant disclosure is not so limited and may be used in conjunction with flexible sensor, ultrasonic sensors, or without sensors. Fiber-bragg gratings may also be used. Additionally, the methods described herein may be used in conjunction with manual systems including, for example, a bronchoscope configured to receive the catheter 102 and manually inserted and navigated within the patient to a location near the target.

[0042] One aspect of the system 100 is a software component stored or accessible from (e.g., cloud) a computing device 116 and configured for processing computed image data. Though many aspects of image data processing have previously been performed at least partially manually, aspects of this disclosure are directed to automated image processing techniques and systems (e.g., using neural network algorithms). These aspects of the disclosure are described in greater detail below in connection with target identification and pathway planning. Some of the aspects described herein, particularly features displayed on a user interface may be presented during the planning phase or during the navigation phase of a procedure where a medical device, such as a biopsy tool or treatment tool, may be inserted into catheter 102 to obtain a tissue sample from or to treat the target.

[0043] In accordance with the disclosure, a 3D model of a luminal network (e.g., the patient’s lungs) or another suitable portion of the anatomy, may be generated from previously acquired scans, such as CT, CBCT, PET or MRI scans. Tumors and lesions within the scan data are detected and pathways through the 3D model, to arrive at the tumors or lesions, are generated.

[0044] Once the pathway plan is generated and accepted by a clinician, that pathway plan may be utilized by a navigation system to drive a catheter or catheter-like device along the pathway plan through the anatomy and particularly the luminal network (e.g., airways) to reach the tumor or lesion. The driving of the catheter along the pathway plan may be manual or it may be robotic, or a combination of both. In a single procedure planning, registration of the pathway plan to the patient, and navigation are performed to enable a medical device, e.g., catheter 102 to be navigated along the planned path to reach the target or lesion, so that a biopsy or treatment of the target can be completed.

[0045] FIG. 2 is a perspective view of an exemplary manual endoluminal navigation system for facilitating navigation of a medical device (e.g., a catheter) to a target within or nearA0012629W001 a luminal network of the body of a patient (e.g., airways of the lungs). As shown in FIG. 2, catheter 102 is part of a catheter guide assembly 120. In one embodiment, catheter 102 is inserted into the patient P for navigation through a patient’s luminal network (e.g., the airways). The catheter 102 may itself include imaging capabilities via an integrated camera or optics component (not shown), a separate bronchoscope 124 may also be employed but is not required. A locatable guide (LG) 126 (a second catheter), including a sensor 104 may be optionally inserted into catheter 102 and locked into position such that sensor 104 extends a desired distance beyond the distal tip of catheter 102. The position and orientation of sensor 104 relative to a reference coordinate system, and thus the distal portion of catheter 102, within an electromagnetic field can be derived. Catheter guide assemblies 120 are currently marketed and sold by Medtronic PLC under the brand names SUPERDIMENSION® Procedure Kits, or EDGE™ Procedure Kits, and are contemplated as useable with the disclosure. Though not shown here, the catheter guide assembly 120 and catheter 102 may be mounted on, advanced, and articulated via one of the robotic arms 40 without departing from the scope of the disclosure.

[0046] As shown an operating table 106 is configured to support a patient P for the endoluminal navigation procedure and the same or a similar operating table is employed to support a patient P undergoing a robotic laparoscopic surgical procedure employing system 10 (FIG. IB). Monitoring equipment is coupled to catheter 102 (e.g., a video display 128, for displaying the video images received from a video imaging system of the catheter 102); a locating or tracking system 1305 including a locating module 132, a plurality of reference sensors 114 and a transmitter mat 106 including a plurality of incorporated markers (not shown). A computing device 116 includes software and / or hardware used to facilitate identification of a target, pathway planning to the target, navigation of a medical device to the target, and / or confirmation and / or determination of placement of catheter 102, or a suitable medical tool or surgical device, relative to the target.

[0047] Computing device 116 may be any suitable computing device including a processor and storage medium, wherein the processor is capable of executing instructions stored on the storage medium. Computing device 116 may further include a database configured to store patient data, CT data sets including CT images, fluoroscopic data sets including images and video, 3D reconstruction, navigation plans, and any other such data. Although not explicitly illustrated, computing device 116 may include inputs, or may otherwise be configured to receive, CT data sets, fluoroscopic images / video and other data described herein. Additionally,A0012629W001 computing device 116 includes a display configured to display graphical user interfaces. Computing device 116 may be connected to one or more networks through which one or more databases may be accessed.

[0048] Pre-procedural and intra-procedural images of the patient (e.g. computed tomography (CT) or magnetic resonance (MR) images) of the patient are acquired and may be saved on or accessed by the computing device 122. A planning application stored on the computing device 116 enables the review of the images, identification of one or more targets (e.g., tumors or lesions) within the images. The planning application is configured to generate a 3D model from the images. Thus, for example, a 3D model of the lungs can be generated from images of the thoracic cavity. Aspects of the planning procedure generate a 3D model depicting the airways from the trachea through multiple bifurcations, which can be displayed in a user interface on the computing device 116. With the targets identified in the images and the 3D model generated, the planning application further enables the generation of an endoluminal path through a natural orifice of the patient (e.g., the mouth) which the catheter 102 can be navigated to acquire biopsy samples of the target, or to apply therapy to the target (e.g., microwave, radio-frequency, chemical, or cryogenic ablation, or another therapy). The pathway plan may be developed prior to or immediately preceding the navigation procedure and transmitted to or accessed by a navigation application. The navigation application may also be stored on the computing device 116 and employed to guide the navigation of the catheter guide assembly 120 and the catheter 102 to arrive at the target(s) as described further below.

[0049] Catheter guide assembly 120 can be navigated within the patient and the tracking system 130 (e.g., an electromagnetic (EM) tracking system, fiber-Bragg grating flex sensor tracking system, inertial tracking, wireless ranging, optical tracking, or other suitable system for determining position and orientation of a distal portion of the catheter 102) is utilized, to detect a position of the sensor 104 within the patient. By detecting the location of the locatable guide 120 or sensor 104 on the catheter 102 a representation of a portion the catheter 102 can be displayed in the 3D model.

[0050] When employing an EM tracking system, a transmitter mat 106 is positioned beneath patient P. Transmitter mat 106 generates an electromagnetic field around at least a portion of the patient P within which the position of a plurality of reference sensors 114 and the sensor 104 can be determined with use of a locating module 132. A second (and optionally a third) electromagnetic sensor 122may also be incorporated into the end of the catheter 102.A0012629W001The second electromagnetic sensor 122may be a five degree-of-freedom sensor or a six degree- of-freedom sensor. These sensors 122can be employed to determine the location and orientation of a distal portion of the catheter 102 even when the LG 126 with sensor 104 has been removed from the catheter. Further, the sensor 122enables use of the catheter guide assembly 120 without requiring the LG 126 and sensor 104. In addition, one or more reference sensors 114 are attached to the chest of the patient P.

[0051] To enable accurate display of the representation of the catheter 102 in the 3D model a, as with the robotic system describe above registration method must be performed. Registration matches the coordinate locations of the three-dimensional model and two- dimensional images from the procedure planning phase, in which a pathway and targets for biopsy or therapy were defined, with the position and orientation of patient P’s airways as they are on the operating table 108. Registration thus allows for the navigation of the organ, and its luminal network to be undertaken with confidence and with accurate depiction of changes in location of the catheter 102 being updated in the 3D model of the luminal network.

[0052] In one embodiment of the disclosure, registration of the patient P’s location on the transmitter mat 106 may be performed by moving sensor 104 or 122 through the airways of the patient P. More specifically, data pertaining to locations of sensor 104 or 122 (e.g., within an electromagnetic field generated by the transmitter mat 106) as the sensor 104 or 122 is moving through the airways, is recorded using the tracking system 115 and optionally with reference to the locations of the reference sensors 114. A shape resulting from this location data is compared to an interior geometry of passages of a 3D model, and a location correlation between the shape and the 3D model based on the comparison is determined, e.g., utilizing the navigation application on computing device 116. The software aligns, or registers, an image representing a location of sensor 104 within the 3D model and / or two-dimensional images based in part on the recorded location data and an assumption that sensors 104 or 122 remain located in non-tissue space in patient P’s airways. Alternatively, a manual registration technique may be employed by navigating the sensors 104 or 122 to pre-specified locations in the lungs of the patient P, and manually correlating the images from the bronchoscope to the model data of the three-dimensional model.

[0053] Though described herein with respect to EMN systems using EM sensors, the instant disclosure is not so limited and may be used in conjunction with flexible sensors such as fiber-Bragg grating sensors, inertial measurement unit (IMU), ultrasonic sensors, or without sensors. Additionally, as described elsewhere herein, the methods described herein may be usedA0012629W001 in conjunction with motorized or robotic systems such that robotic actuators drive and articulate the catheter 102 within the patient to the desired target.

[0054] Further though generally described herein with respect to endoluminal navigation, the disclosure is not so limited and can be utilized in intraluminal navigation. In intraluminal navigation an opening is created in the natural lumen (e.g., airways) with a piercing device and the catheter 102 is advanced beyond the lumen wall. This approach can be employed with seeking to biopsy or treat tumors and tissue beyond the airway wall or access other adjacent structures including lymph nodes and other structures outside the luminal wall.

[0055] In accordance with aspects of the disclosure, acquisition of intraprocedural images enables the visualization of intra-body navigation of a medical device (e.g., a biopsy tool or a therapy tool), towards a target (e.g., a lesion) may be a portion of a larger workflow of a navigation system. An imaging device 118 (e.g., a fluoroscope or a CT or cone beam CT imaging device such as the Medtronic 0-arm™ surgical imaging system) capable of acquiring 2D and 3D images or video of patient P is also included in this particular aspect of system 100. The images, sequence of images, or video captured by imaging device 118 may be stored within the imaging device 1184 or transmitted to computing device 116 for storage, processing, and display. Additionally, imaging device 124 may move relative to the patient P so that images may be acquired from different angles or perspectives relative to patient P to create a sequence of images, such as a fluoroscopic video. The pose of imaging device 118 relative to patient P while capturing the images may be estimated via markers incorporated with the transmitter mat 106. The markers are positioned under patient P, between patient P and operating table 108 and between patient P and a radiation source or a sensing unit of imaging device 124. The markers incorporated with the transmitter mat 106 may be two separate elements which may be coupled in a fixed manner or alternatively may be manufactured as a single unit. Imaging device 118 may include a single imaging device or more than one imaging device.

[0056] Regardless of whether employing the robotic system of FIG. IB or the manual system of FIG. 3, while endoluminal navigation of a catheter 102 to conduct a procedure from within the luminal network avoids the need to pierce the patient and insert ports to gain access to the targets, not every procedure can be performed with an endoluminal approach. An example of a procedure that cannot be performed with an endoluminal approach is a lung segmentectomy. In a lung segmentectomy, a portion of the lung (a segment) is severed from the remaining lung and the line along which the segment is severed is sealed. The severing of the segment removes separates the diseased tissue (e.g., containing tumor or lesion) from theA0012629W001 health tissue limiting the spread of the disease. The severed segment is then removed from the patient via one of the ports.

[0057] As noted above, pre-procedural images are useful in identifying a segment for removal, and to generally allow for planning of a procedure. Often a 3D model is not generated, and even if a 3D model is generated due to the differences in positioning of the patient required for imaging (usually supine) and many procedures (lateral side-lying), the changes to softer- tissue organs (e.g., the lungs), and their locations within the patient, make the 3D models difficult to reconcile with the views of the organ or other tissues during through a laparoscopic camera 51. Further, registration of the pre-procedure images or 3D models is near to impossible, in part due to this same softness of the tissues and the changes in positioning of the patient from imaging to procedure. As a result, the laparoscopic approach relies on the surgeon’s abilities to visualize the tissue for treatment, and the ability to distinguish that tissue from other healthy tissues.

[0058] To provide for pre-procedural planning and improved intra-procedure guidance for robotic laparoscopic procedures, the disclosure contemplates a combined endoluminal and laparoscopic approach. In accordance with this combined approach the endoluminal approach, as described above in connection with FIG. 2 including the receipt of pre-procedural images or intra-procedural images, the identification of targets, generation of a 3D model, and generation of the endoluminal pathway through the luminal network and 3D model to the identified target. Using the pathway plan and following registration of the pathway plan and to the patient a catheter 102 is navigated to an identified target and tracked through the patient with a representation of the catheter 102 in the 3D model.

[0059] In accordance with one aspect of the disclosure, in addition to identifying the target in the pre-procedure or intraprocedural images, the planning may include the identification a desired margin around the target to ensure that the tumor or lesion is completely removed. In addition, important or critical structures (e.g., blood vessels, fissures, etc.) can be identified and their relative positions to the target can be saved in the planning. The identification of such structures, as described below, can be utilized to define a no-fly zone which the surgical instruments 50 should avoid or at least location of which the surgeon need be aware of during the procedure. The identification of these structures can be tied to a variety of cautions or warnings that may be displayed in the screen 23 of the console 30 or on a display of computing device 116 of console 30B.A0012629W001

[0060] Each surgical instrument 50 and / or robotic arm 40 includes a sensor 53 (e.g., an EM sensor). The sensor 53 allows the tracking system 115 to detect a position and orientation of robotic arms 40 or the surgical instruments 50. As a result, and based on the registration process of the catheter 102 and the sensor 104 / 126, the laparoscopic surgical system 10, and particularly the surgical instruments 50 and the robotic arms 40 employing sensors 53, is registered to the patient, the 3D model, and the pre-procedure or intra-procedure images from which the 3D model is generated. Representations of each surgical instrument 50 can be depicted on the 3D model.

[0061] With the catheter 102 in place with sensor 104 / 122 detecting its position within the patient, and registration complete the relative positions of the surgical instruments 50 and the catheter 102 can be determined. This relative position is based at least on the detected positions of the sensors 104 / 122 and 53. A representation of their relative positions of the catheter 102 and the surgical instruments 50 can be displayed in the 3D model on a display 110 of the computing device 122 or on the screen 23 of the control tower 20. In some instances, displaying the relative positions may require a zooming out of the display of the 3D model, or the addition of features and aspects of the 3D model to the display (e.g., a 3D model of the thoracic cavity as opposed to just the airways of the lungs for endoluminal navigation).

[0062] As a result of the registration and display of the relative positions of the detected sensors 53 to the 3D model, locations for incisions in the patient P in which to place ports can be assessed to identify paths to the target least encumbered by other tissues and where ribs and other hard tissues least interfere with the articulation and manipulation of the robotic arms 40 and the surgical instruments 50 inserted into the patient. With the incision points identified, the placement of the mobile carts 60 including the robotic arms 40 and the surgical instruments 50 relative to the patient to can be adjusted to access the ports and therewith the target within the patient.

[0063] In accordance with aspects of the disclosure, during a procedure, the initial navigation of the of the catheter 102 within the luminal network of the patient occurs with the patient in the supine position. With registration complete the patient can be moved so that they are on in a lateral side-laying position where laparoscopic procedures, particularly of the thoracic cavity are undertaken. As noted above, this movement causes the organs of the thoracic cavity to shift due to gravity. The shift of the organs within the thoracic cavity provides space within the thoracic cavity for the insertion of the surgical instruments 50 to navigate and articulate within the thoracic cavity. Further, with particular reference to the lungs, where aA0012629W001 procedure such as a segmentectomy is undertaken, the lung (right or left) or a relevant lung lobe is isolated from ventilation and deflated. This deflation increases the volume of the thoracic cavity for the surgical instruments 50, and also improves the ability of the surgical instruments 50 and their end effectors to manipulate and act on the tissue of the deflated lung or lobe.

[0064] As will be appreciated, change in position of the patient and the deflation of lung or lobe limits the utility of the 3D model generated from the pre-procedure or intraprocedural images. However, despite the 3D model having less utility, the relative position and orientation of the sensor 104 / 122 on the catheter 102 and the sensors 53 on surgical instruments 50 or the robotic arms 40 are still detected. Since the sensor 104 / 122 on the catheter 102 has been navigated to the target, the position and orientation of the sensor 104 / 122 acts as a beacon to which the surgical instruments 50 are to be navigated to act on the tissue of the target.

[0065] The relative position of the sensors 104 / 122 on the catheter 102 can be displayed on a user interface on the screen 32 of consoles 30A or on a display 110 of the console 30B. The relative positions may be depicted on live images from the camera 51, or on intraprocedural images acquired from another source (e.g., a fluoroscope or CBCT imaging device) within the operating room. To assist in identifying the location of the target visually the catheter 102 may be employed to deploy dye to the area of the target. Alternatively, a light source may be connected to a distal end of the catheter 102 (e.g., via a fiber optic light pipe). The light transmits through the tissue and can be seen via the camera 51 enabling the surgeon to immediately identify the location of the catheter 102 within the luminal network of the patient.

[0066] As an example, FIG. 3 depicts a view from a surgeon’s perspective during a procedure performed using the robotic surgical system 10. A live laparoscopic image 150 captured by a laparoscopic camera 51 connected to one of the robotic arms 40 in the surgical suite is displayed on screen 32 of a robotic console 30. The laparoscopic image 150 captures the end effectors 49 within the field of view. In addition, overlaid on the laparoscopic images 150 is a representation 152 of the catheter 102 that has been navigated using, for example, the robotic luminal navigation system of FIG. IB. With the representation 152 overlaid on the live image, the surgeon has greater confidence of their location within the patient when performing the segmentectomy or other surgical procedures.

[0067] Similarly, FIG. 4 depicts a user interface 160 that is part of the endoluminal navigation system (either the robotic system of FIG. IB or manual system of FIG. 2). The user interface 160 can include a variety of views including a live endoscopic 162, which mayA0012629W001 selectively be replaced by a virtual bronchoscopy view (e.g., an internal view of the 3D model 164. On the live endoscopic view 162 a pathway 166 to a target 168 can be depicted. A marker 170 may be depicted in the live endoscopic view 162 denoting the relative location of the target 168 (e.g., a tumor or lesion) from the current location of the catheter 102 within the patient. The target 168 is also depicted in the 3D model 164 as is the pathway 166. As the catheter 102 is navigated toward the target 168, the user interface updates and eventually the target 168 is in view either in the endoscopic view 162. An indicator 172 of the planned pathway is depicted at the bottom of the user interface 160 providing a visual indicator of both the distance already traveled and the distance still to be navigated to arrive at the target 168. In addition, a numerical distance indicator 174 may present the measured distance still to travel to arrive at the target 168.

[0068] FIG. 5 depicts a user interface 160 depicting the virtual bronchoscopy view 176 Selectively, in a fashion similar to FIG. 3, an indicator 178 or representations of the location of the end effectors 49 of the surgical instruments 50 may be depicted on the user interface 176. By displaying these representations 178 of the end effectors 49, the surgeon is provided a further indication of the proximity of the end effectors 49 from the current location of the catheter 102, but from the internal perspective, to assist in confirming their placement for the performance of tasks associated with the procedure. As will be appreciated the user interface 160 may be depicted in display 110 associated with computing device 116 and console 30B or screen 32 of console 30A or screen 23 or 34 as desired by the surgeon.

[0069] In addition to the relative position, a change in position of the sensor 104 / 122 caused by the change in position and the deflation of the lung or lobe can be determined. One or more deformation algorithms may be stored in a memory (e.g., associated with the computing devices of consoles 30A or 30B). Based on the determined change in position, the deformation algorithm may be employed to estimate a deformation of the luminal network (e.g., the lungs) as a result of the move. The estimate of deformation can be applied to the 3D model 164, and a deformed 3D model 164 can be displayed to assist the surgeon in understanding the tissue being viewed through the laparoscope and to provide general guidance in how to manipulate the surgical instruments 50 or robotic arms 40. To improve the deformation modeling, multiple sensors 104 / 122 may be located on the catheter 102 along its length such that the deformation of the portion of the luminal network that has been navigated can be more accurately modeled.

[0070] The deformed 3D model 164 is necessarily registered with the patient by virtue of the catheter 102 having been navigated to the target and remaining within the patient duringA0012629W001 the change in orientation. As such, features from the 3D model may be overlaid on laparoscopic or other images acquired of the patient during the procedure. For example, the locations of major blood vessels and fissures may be displayed in the live images. The presentation of these features can be translucent such that the surgeon is able to see through them to the underlying tissue. In addition, the location for the presentation on the images need not be exact, but rather can be generally in the locations of the expected deformation to provide reminders or guidance to the surgeon to be on the lookout for the features during the procedure. This may be particularly relevant when the tissue is on the opposite side of the tissue from that being imaged via the laparoscope. In some instances, these tissues on the opposite side of the tissue may include an indicator of depth from the surface of the tissue being viewed via the laparoscope in an effort to prevent the surgeon from advancing the surgical instruments 50 beyond a desired depth.

[0071] In accordance with a further aspect of the disclosure once the relative positions of the sensors 53 on surgical instruments 50 and the sensor 104 / 122 on the catheter 102 are determined, the robotic laparoscopic surgery continues relying largely on the surgeon’s experience and skill. However, even this is informed by the now known location of the target as identified by the catheter 102 reducing the cognitive burden on the surgeon in understanding their environment within the patient.

[0072] Reference is made to FIG. 5 which presents a method 200 for performing a combined endoluminal and laparoscopic surgery in accordance with the disclosure. As outlined above, at step 202 images (e.g., pre-procedural or intraprocedural CT or MRI images) are acquired and reviewed in an application (e.g., running on computing devices associated with consoles 30A or 30B to identify one or more targets 168 within the patient. At step 204 a planning application generates a 3D model 164 from the images. The 3D model 164 may be of just a relevant organ (e.g., the lungs or liver) or of an entire region (e.g., the thoracic cavity) and the clinician can adjust the level of zoom for each portion of the procedure (endoluminal vs laparoscopic). At step 206, with the target(s) identified in the images and the 3D model 164 generated, a pathway 166 for endoluminal navigation of a catheter 102 to the target 168 can be generated by the planning application. As described above, the pathway 166 may be, for example, from a natural opening such as the mouth or nose through the trachea and airways to arrive at or near the target.

[0073] At step 208, via a user interface the planning application displays aspects of the images or the 3D model 164 and enables the user to identify aspects of the anatomy relevant toA0012629W001 the laparoscopic procedure. These aspects may include identification of the lobe or segment and isolation from the remainder of the lung, locations of blood vessels in need of ligation, fissures separating the lung segments, critical structures to be avoided (no fly zones), margins around the segment to ensure complete removal, locations of staple lines or resection lines, and others without departing from the scope of the disclosure.

[0074] In accordance with further aspects of the disclosure, the laparoscopic surgical plan may further include identification of locations on the patient to place laparoscopic ports. As described above, the 3D model 164 may be zoomed out from the intraluminal 3D model 164 to allow for a view of exterior surfaces of the patient. The exterior surfaces may be presented in a translucent or semi -translucent manner such that the organs of the patient are visible through the exterior surface. In accordance with the disclosure, the 3D model 164 of the organ or luminal network in question may be deformed to present a view on the user interface as the patient and their organs will likely be arranged when undertaking the laparoscopic portions of the procedure. Further, where the luminal network in question is the airways of the lungs the 3D model 164 can be further adjusted to model the effects of deflating the relevant portion of the lungs. With the view of the patient changed and the organs deformed as expected to occur as a result of the movement of the patient and other activities, the locations on the patient for incision and placement of the laparoscopic ports may be identified. These locations are locations on the patient’s skin that allow laparoscopic tools to be inserted into the patient and to arrive at the target without interfering with the patient’s ribs or unnecessary or difficult interactions with portions of patient’s organs. With the internal or external portions of the laparoscopic surgery planned the combination of these aspects forms a laparoscopic surgical plan.

[0075] As will be appreciated, some of these aspects of the laparoscopic surgical plan may be automatically generated by the application utilizing various image processing applications to segment and analyze the images and the 3D model 164. With the pathway plan for endoluminal navigation and the laparoscopic surgical plan developed, they are both saved in a memory accessible to both the planning application and a surgical application.

[0076] At step 212, after placement of the patient P on the operating table 108, the 3D model and the pathway plan can be displayed in the user interface of the surgical application, for example on the display of computing device 116. At step 214, by navigating the catheter 102 into the luminal network of the patient, the 3D model 164 is registered to the actual luminal network of the patient (e.g., the airways). In one aspect of the disclosure, the registrationA0012629W001 process is performed utilizing one or more EM sensors on the catheter 102 detecting an EM field generated by transmitter matt 106. During the advancement, the locations of travel of the catheter 102 are mapped and compared to the 3D model 164 of the airways. Once sufficient datapoints are collected and the registration complete, the location of the catheter 102 within the luminal network (e.g., the airways) can be displayed in the 3D model 164 on the user interface at step 216 and the catheter 102 can be navigated to the target at step 218 (e.g., robotically advanced and articulated following the pathway plan).

[0077] With the catheter 102 at or near the target, a biopsy may be acquired from, or therapy may be optionally applied to the target at step 220. With the catheter 102 at or near the target 168, the position of the patient’s position is adjusted to facilitate laparoscopic surgical approaches and that change in position is detected at step 222. This change in positioning of the patient may be made in combination with other steps including switching a single lumen endotracheal tube for a dual lumen endotracheal tube, deflating lobes of one lung or an entire lung, segments of the lungs, rolling from a supine to a lateral side-lying, and others. This detection is primarily a determination based on a change in position of the sensor 104 / 122 of the catheter 102 after having been navigated to the target during the endoluminal navigation phase of the procedure.

[0078] At step 224, based at least in part on the changes in location of the sensor 104 / 122 on the catheter 102 caused by the adjustment in position of the patient, the 3D model 164 may be optionally deformed, and the deformed 3D model 164 depicted in the user interface 160. The deformed 3D model depicts a representation of the detected location of the sensor 104 / 122 on the catheter 102 depicted in the deformed 3D model 164 (e.g., within the luminal network).

[0079] At step 226 the surgical instruments 50 on the robotic arms 40 are navigated to a position where the location of the end effectors 49 relative to the detected location of the sensor 104 / 122 of the catheter is detected. In one example, the surgical instruments 50 or robotic arms 40 include an EM sensor 53, for example a tunnel magnetoresistance (TMR) sensor. These EM sensors 53 may be on the surgical instruments 50 or robotic arms 40. Additionally, or alternatively the robotic arms 40 or surgical instruments may include optical sensors (not shown) whose relative position can be detected. An optical sensor on the operating table can be used as a basis for converting a coordinate system of the optical tracking system to a common coordinate system with the tracking system of the catheter 102 (e.g., an EM tracking system).A0012629W001

[0080] Still further, to detect the position of the surgical instruments 50, intraprocedural images (e.g., fluoroscopic or CT images) can be captured and a relative position of the catheter 102 and the target it has been navigated to with respect to the surgical instruments 50 can be determined. This determination of relative position and orientation can be made either directly from the intraprocedural images or via generation of a 3D model 164 depicting the relevant tissue, the surgical instruments 50, and the catheter 102.

[0081] Once the position and orientation of the surgical instruments 50 with respect to the catheter 102, and particularly the target to which it has been navigated, is known a variety of optional steps may be undertaken. At step 228 the surgical instruments 50 may be depicted in the 3D model 164 (e.g., zoomed out to include the skin surface as described above). At step 230 the relative positions of the surgical instruments 50 and the catheter 102 may be depicted in the 3D model 164 or in step 232 in live or intraprocedural images of the patient. Further at step 234 any aspect of the laparoscopic surgical plan (step 208) can be displayed either on portions of the 3D model 164 or on the intraprocedural images.

[0082] To perform, for example a segmentectomy, with the relative position of the surgical instruments 50 to the catheter 102 determined and potentially displayed, the surgeon can proceed to conduct the laparoscopic portion of the surgery substantially as they have previously done relying primarily on the images provided by the laparoscope. However, this is supplemented by the relative location data of the sensor 53 associated with surgical instruments 50 and the sensor 104 / 122 on the catheter 102 as the position of the laparoscopic instruments are being updated on the user interface while the surgical instruments 50 are navigated to the target. Further, the application, upon detecting a proximity to one or more of the critical structures or a no-go zone may provide an audible or visual indicator to alert the surgeon.

[0083] Reference is now made to FIG. 6, which is a schematic diagram of a computing system 300 configured for use with the methods of the disclosure including the method 200 of FIG. 3. The computing system may be found in any of computing device 116 and computers 21, 31, 41. Further, aspects of the computing system 300 may be connected to one or more but not necessarily all of the computing device 116 or computer 21, 31, 41 (e.g., a distributed computing system) without departing from the scope of the disclosure. System 300 may include a workstation 301, and optionally an imaging device 315 (e.g., a fluoroscope, CT imaging device, or an ultrasound imaging device). In some embodiments, workstation 301 may be coupled with imaging device 315, directly or indirectly, e.g., by wireless communication. Workstation 301 may include a memory 302, a processor 304, a display 306 and an input deviceA0012629W001310. Processor or hardware processor 304 may include one or more hardware processors. Workstation 301 may optionally include an output module 312 and a network interface 308. Memory 302 may store an application 318 and image data 314. Application 318 may include instructions executable by processor 304 for executing the methods of the disclosure including the method of FIG. 5.

[0084] Application 318 may further include a user interface 316. Image data 314 may include the CT scans, the generated fluoroscopic 3D reconstructions of the target area and / or any other fluoroscopic image data and / or the generated one or more slices of the 3D reconstruction. Processor 304 may be coupled with memory 302, display 306, input device 310, output module 312, network interface 308 and imaging device 315. Workstation 301 may be a stationary computing device, such as a personal computer e.g., computing device 116, or a portable computing device such as a tablet computer, or further be implemented on the console 30, further, workstation 301 may embed a plurality of computer devices.

[0085] Memory 302 may include any non-transitory computer-readable storage media for storing data and / or software including instructions that are executable by processor 304 and which control the operation of workstation 301 and, in some embodiments, may also control the operation of imaging device 315. Imaging device 315 may be used to capture a sequence of fluoroscopic images based on which the fluoroscopic 3D reconstruction is generated and to capture a live 2D fluoroscopic view according to this disclosure. In an embodiment, memory 302 may include one or more storage devices such as solid-state storage devices, e.g., flash memory chips. Alternatively, or in addition to the one or more solid-state storage devices, memory 302 may include one or more mass storage devices connected to the processor 304 through a mass storage controller (not shown) and a communications bus (not shown).

[0086] Although the description of computer-readable media contained herein refers to solid-state storage, it should be appreciated by those skilled in the art that computer-readable storage media can be any available media that can be accessed by the processor 304. That is, computer readable storage media may include non-transitory, volatile, and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data. For example, computer-readable storage media may include RAM, ROM, EPROM, EEPROM, flash memory or other solid-state memory technology, CD-ROM, DVD, Blu-Ray or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or otherA0012629W001 magnetic storage devices, or any other medium which may be used to store the desired information, and which may be accessed by workstation 301.

[0087] Application 318 may, when executed by processor 304, cause display 306 to present user interface 316. User interface 316 may be configured to present to the user a single screen including a three-dimensional (3D) view of a 3D model, a live two-dimensional (2D) fluoroscopic view showing the medical device, and a target view, which corresponds to the 3D model of the target, overlaid on the live images (e.g., CT of fluoroscopic). User interface 316 may be further configured to display the target in different colors depending on whether the medical device tip is aligned with the target in three dimensions. Still further, in connection with the above descriptions the user interface 316 can be configured to depict any of the aspects of the endoluminal pathway or the laparoscopic surgical plan in the 3D model, deformed 3D model, or the intraprocedural images.

[0088] Network interface 308 may be configured to connect to a network such as a local area network (LAN) consisting of a wired network and / or a wireless network, a wide area network (WAN), a wireless mobile network, a Bluetooth network, and / or the Internet. Network interface 708 may be used to connect between workstation 301 and imaging device 315. Network interface 708 may also be used to receive image data 314. Input device 310 may be any device by which a user may interact with workstation 301, such as, for example, a mouse, keyboard, foot pedal, touch screen, and / or voice interface. Output module 312 may include any connectivity port or bus, such as, for example, parallel ports, serial ports, universal serial busses (USB), or any other similar connectivity port known to those skilled in the art. From the foregoing and with reference to the various figures, those skilled in the art will appreciate that certain modifications can be made to the disclosure without departing from the scope of the disclosure.EXAMPLES

[0089] Example 1 - A robotic surgical system including a navigation catheter including a first sensor and configured for endoluminal navigation, a robotic arm having a laparoscopic surgical instrument and a second sensor, and a computing system including a processor and a memory, the memory storing instructions that when executed by the processor cause the processor to: detect a position and orientation of the first sensor as the catheter is navigated into a luminal network towards a target, register the luminal network to a three-dimensional (3D) model of the luminal network and a pathway plan for navigation of the luminal network, display on a user interface a representation of a distal portion of the catheter as the catheter isA0012629W001 navigated in the luminal network along the pathway plan, determine that the catheter is in proximity to the target, detect a position and orientation of the second sensor; and display on the user interface a representation of the laparoscopic surgical instrument’s position and orientation relative to the position and orientation of the catheter.

[0090] Example 2 - The robotic surgical system of Example 1, wherein the navigation catheter is operably connected to a robotic system for advancement and articulation.

[0091] Example 3 - The robotic surgical system of any of Examples 1 and 2, wherein the first sensor and second sensor are electromagnetic (EM) sensors.

[0092] Example 4 - The robotic surgical system of Example 3, wherein one or more of the EM sensors is a tunnel magnetoresistance (TMR) sensor.

[0093] Example 5 - The robotic surgical system of Example 3, further comprising an EM field generator.

[0094] Example 6 - The robotic surgical system of any of Examples 1-4, wherein the instructions when executed by the processor cause the processor to detect a change in position of a patient after navigation of the catheter in proximity to the target.

[0095] Example 7 - The robotic surgical system of Example 5, wherein the instructions when executed by the processor cause the processor to deform the 3D model based in part on the detected change in position.

[0096] Example 8 - The robotic surgical system of any of Examples 1-7, wherein the instructions when executed by the processor cause the processor to receive inputs identifying one or more aspects of a laparoscopic surgical procedure.

[0097] Example 9 - The robotic surgical system of Example 8, wherein the inputs identify one or more of a fissure, a blood vessel, a staple line, a resection line, a critical structure, a no- go zone, or a margin.

[0098] Example 10 - The robotic surgical system of any of Examples 8 or 9, wherein the instructions when executed by the processor cause the processor to generate an alert when the surgical instrument is in proximity to one of the aspects of the laparoscopic surgical procedure.

[0099] Example 11 - A method of performing a combined endoluminal and laparoscopic procedure including detecting a position and orientation of a catheter as it is navigated into a luminal network towards a target, registering the luminal network with a three-dimensional (3D) model of the luminal network and a pathway plan for navigation of the luminal network;, determining that the catheter is in proximity to the target, detecting a position and orientationA0012629W001 of a robotic laparoscopic surgical instrument, and navigating the robotic laparoscopic surgical instrument to the target.

[0100] Example 12 - The method of Example 11, further comprising displaying a relative position of the robotic laparoscopic surgical instrument and the catheter in proximity to the target on a user interface.

[0101] Example 13 - The method of any of Examples 11 to 12, further comprising detecting a position and orientation of the catheter or the laparoscopic surgical instrument within an electromagnetic field.

[0102] Example 14 - The method of any of Examples 11 to 13, further comprising receiving imaging of the luminal network and generating the 3D model from the imaging.

[0103] Example 15 - The method of any of Examples 11 to 14, further comprising detecting an electromagnetic field to determine a position and orientation of the robotic laparoscopic surgical instrument or a position and orientation of the catheter.

[0104] Example 16 - The method of any of Examples 11 to 15, further comprising detecting a change in position of a patient after navigation of the catheter in proximity to the target.

[0105] Example 17 - The method of any of Examples 11 to 16, further comprising displaying laparoscopic surgical plan data on a deformed 3D model.

[0106] Example 18 - The method of any of Examples 11 to 17, further comprising depicting a relative position and orientation of the robotic laparoscopic surgical instrument and the position and orientation of the catheter in intraprocedural images.

[0107] Example 19 - The method of any of Examples 11 to 18, further comprising displaying laparoscopic surgical plan data on intraprocedural images.

[0108] Example 20 - The method of Example 19, wherein the laparoscopic plan data includes one or more of a fissure, a blood vessel, a staple line, a resection line, a critical structure, a no-go zone, or a margin.

[0109] Although embodiments have been described in detail with reference to the accompanying drawings for the purpose of illustration and description, it is to be understood that the inventive processes and apparatus are not to be construed as limited. It will be apparent to those of ordinary skill in the art that various modifications to the foregoing embodiments may be made without departing from the scope of the disclosure.

Claims

A0012629W001We claim:

1. A robotic surgical system comprising: a navigation catheter including a first sensor and configured for endoluminal navigation; a robotic arm having a laparoscopic surgical instrument and a second sensor; a computing system including a processor and a memory, the memory storing instructions that when executed by the processor cause the processor to: detect a position and orientation of the first sensor as the catheter is navigated into a luminal network towards a target; register the luminal network to a three-dimensional (3D) model of the luminal network and a pathway plan for navigation of the luminal network; display on a user interface a representation of a distal portion of the catheter as the catheter is navigated in the luminal network along the pathway plan; determine that the catheter is in proximity to the target; detect a position and orientation of the second sensor; and display on the user interface a representation of the laparoscopic surgical instrument’s position and orientation relative to the position and orientation of the catheter.

2. The robotic surgical system of claim 1, wherein the navigation catheter is operably connected to a robotic system for advancement and articulation.

3. The robotic surgical system of any of claims 1 and 2, wherein the first sensor and second sensor are electromagnetic (EM) sensors.

4. The robotic surgical system of claim 3, wherein one or more of the EM sensors is a tunnel magnetoresistance (TMR) sensor.

5. The robotic surgical system of claim 3, further comprising an EM field generator.

6. The robotic surgical system of any of claims 1-4, wherein the instructions when executed by the processor cause the processor to: detect a change in position of a patient after navigation of the catheter in proximity to the target.A0012629W0017. The robotic surgical system of claim 5, wherein the instructions when executed by the processor cause the processor to: deform the 3D model based in part on the detected change in position.

8. The robotic surgical system of any of claims 1-7, wherein the instructions when executed by the processor cause the processor to: receive inputs identifying one or more aspects of a laparoscopic surgical procedure.

9. The robotic surgical system of claim 8, wherein the inputs identify one or more of a fissure, a blood vessel, a staple line, a resection line, a critical structure, a no-go zone, or a margin.

10. The robotic surgical system of any of claims 8 or 9, wherein the instructions when executed by the processor cause the processor to: generate an alert when the surgical instrument is in proximity to one of the aspects of the laparoscopic surgical procedure.

11. A method of performing a combined endoluminal and laparoscopic procedure comprising: detect a position and orientation of a catheter as it is navigated into a luminal network towards a target; registering the luminal network with a three-dimensional (3D) model of the luminal network and a pathway plan for navigation of the luminal network; determine that the catheter is in proximity to the target; detect a position and orientation of a robotic laparoscopic surgical instrument; and navigate the robotic laparoscopic to the target.

12. The method of claim 11, further comprising displaying a relative position of the robotic laparoscopic surgical instrument and the catheter in proximity to the target on a user interface.

13. The method of any of claims 11 to 12, further comprising detecting a position and orientation of the catheter or the laparoscopic surgical instrument within an electromagnetic field.

14. The method of any of claims 11 to 13, further comprising receiving imaging of the luminal network and generating the 3D model from the imaging.A0012629W00115. The method of any of claims 11 to 14, further comprising detecting an electromagnetic field to determine a position and orientation of the robotic laparoscopic surgical instrument or a position and orientation of the catheter.

Images