Depicting hidden target anatomy on live imaging

Abstract

A robotic surgical system and method, the system including a navigation catheter having a first sensor and configured for endoluminal navigation, a robotic arm having a laparoscopic camera and a second sensor, and a computing system including a processor and a memory, the memory storing instructions that when executed by the processor cause the processor to detect a position and orientation of the first sensor as the catheter is navigated into a luminal network towards a target, register the luminal network to a three-dimensional (3D) model of the luminal network and a pathway plan for navigation of the luminal network, detect the second sensor, determine a position and orientation of the laparoscopic camera, determine when the first sensor is within a field of view of the laparoscopic camera, and display a representation of a location of the first sensor on a laparoscopic image captured by the laparoscopic camera.

Description

PATENT APPLICATIONA0013105W001 (00017-01309PCT00)DEPICTING HIDDEN TARGET ANATOMY ON LIVE IMAGINGCROSS-REFERENCE TO RELATED APPLICATION

[0001] This application claims the benefit of U.S. Provisional Patent Application Serial No. 63 / 709,065, filed October 18, 2024, the entire contents of which is incorporated herein by reference.BACKGROUNDTechnical Field

[0001] This disclosure relates to systems and methods for identifying a location of target tissue and depicting an indicator of the location of the target tissue and other clinically relevant data that would otherwise be unseen on live laparoscopic images to aid in access and application of therapy.Background

[0002] There are several commonly applied medical methods, such as endoscopic procedures or minimally invasive procedures, for treating various maladies affecting organs including the liver, brain, heart, lungs, gall bladder, kidneys, and bones. Often, one or more imaging modalities, such as magnetic resonance imaging (MRI), ultrasound imaging, computed tomography (CT), cone-beam computed tomography (CBCT), or fluoroscopy (including 3D fluoroscopy) are employed by clinicians to identify and navigate to areas of interest within a patient and ultimately arrive at a target for biopsy or treatment. In some procedures, pre-operative and intra-operative scans are utilized for target identification and guidance. In an exemplary endoscopic approach, navigating to areas of interest within a patient is enabled though the use of MRI or CT image data that is used to generate a three-dimensional (3D) rendering, 3D model, or 3D volume of the particular organ, such as the lungs. The 3D model provides guidance for navigation of a catheter including pathway plans that are displayed in the 3D model to navigate the luminal network of the organ and arrive at an identified target that is also displayed in the 3D model.

[0003] To assist in the performance of these procedures a tracking system (e.g., electromagnetic, shape sensing, or image-based guidance, etc.) can be employed. Utilizing one of these tracking systems, an endoluminal catheter is advanced (either manually or robotically) within the boundaries of the luminal network, and by comparing sensor data toPATENT APPLICATIONA0013105W001 (00017-01309PCT00) the shape of the lumens of the 3D model the location of the catheter within the patient can be registered to the pre-procedure models and images and the location of the catheter in the patient is displayed in the 3D model and is updated as the catheter is moved along a desired path to a target.

[0004] In contrast with endoluminal systems (e.g., endoluminal robots) where the catheter-based system is typically constrained within the luminal network (e.g. airways, blood vessels, etc.) laparoscopic robotic systems typically are not so constrained. Instead, the laparoscopic robotic system employs one or more robotic arms each of which is configured to insert and manipulate an end effector within the patient through a laparoscopic port placed in an incision in the patient. The incisions may be in the abdomen or in the thoracic cavity depending on the location of the surgery.

[0005] While pre-procedural images are regularly acquired to assist in planning the laparoscopic surgical approach, these plans are typically high level identifying the location of a tumor or lesion or other feature of the patient that appears in the images or a 3D model generated from the images. However, due to the differences in positioning of a patient during imaging and during a laparoscopic procedure, among other factors, the planning is often of a general nature and the performance of the procedure relies on the experience and expertise of the surgeon to identify the same tissue via a laparoscope as seen in the preprocedure images and 3D models.

[0006] Laparoscopic surgery has been referred to as keyhole surgery, not least because the field of view (“FOV”) through a laparoscope can be analogized to the field of view when looking through a keyhole. The field of view of a laparoscopic procedure (the “keyhole”)has vastly improved over many years because of advances in imaging equipment. However, it remains difficult to reconcile the location of the area of interest depicted in preoperative scans and 3D models with the live camera views from a laparoscope.

[0007] This disclosure is directed to systems and methods of improving visibility of targets and clinically relevant data during laparoscopic procedures.SUMMARY

[0008] A system of one or more computers can be configured to perform particular operations or actions by virtue of having software, firmware, hardware, or a combination of them installed on the system that in operation causes or cause the system to perform the actions. One or more computer programs can be configured to perform particular operationsPATENT APPLICATIONA0013105W001 (00017-01309PCT00) or actions by virtue of including instructions that, when executed by data processing apparatus, cause the apparatus to perform the actions. One general aspect of the disclosure is directed to a robotic surgical system including navigation catheter having a first sensor and configured for endoluminal navigation. The robotic surgical system also includes a robotic arm having a laparoscopic camera and a second sensor; and a computing system including a processor and a memory, the memory storing instructions that when executed by the processor cause the processor to: detect a position and orientation of the first sensor as the catheter is navigated into a luminal network towards a target; register the luminal network to a three-dimensional (3D) model of the luminal network and a pathway plan for navigation of the luminal network; detect the second sensor; determine a position and orientation of the laparoscopic camera; determine when the first sensor is within a field of view of the laparoscopic camera; and display a representation of a location of the first sensor on a laparoscopic image captured by the laparoscopic camera. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the methods.

[0009] Implementations may include one or more of the following features. The robotic surgical system where the navigation catheter is operably connected to a second robotic arm for advancement and articulation. The first sensor and second sensor are electromagnetic (EM) sensors. One or more of the EM sensors is a tunnel magnetoresistance (TMR) sensor. The instructions when executed by the processor cause the processor to: receive inputs identifying one or more aspects of a laparoscopic surgical procedure including targets, segment or lobe identifications, or clinically relevant data. The clinically relevant data includes one or more of a resection line, suture line, blood vessel, margin, fissure, a no-go zone, or critical structure. The instructions when executed by the processor cause the processor to: deform the 3D model and the locations of the clinically relevant data. The one or more aspects of the laparoscopic procedure and clinically relevant data are depicted on the laparoscopic image. The instructions when executed by the processor cause the processor to: depict a representation of the laparoscopic camera at the determined location and orientation in the deformed 3D model. The robotic surgical system may include an EM field generator. Implementations of the described techniques may include hardware, a method or process, or computer software on a computer-accessible medium.PATENT APPLICATIONA0013105W001 (00017-01309PCT00)

[0010] A second general aspect of the disclosure is directed to a method of performing a combined endoluminal and laparoscopic procedure. The method includes detecting a position and orientation of a catheter as it is navigated into a luminal network towards a target; registering the luminal network with a three-dimensional (3D) model of the luminal. The method also includes network and a pathway plan for navigation of the luminal network; determining that a sensor on the catheter is in proximity to the target, detecting a position and orientation of a laparoscopic camera, determining when the sensor is within a field of view of the laparoscopic camera, and displaying a representation of a location of the sensor on a laparoscopic image captured by the laparoscopic camera. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the methods.

[0011] Implementations may include one or more of the following features. The method where one or more of the laparoscopic camera and catheter are robotically driven. The method may include detecting a position and orientation of the catheter or the laparoscopic camera within an electromagnetic field. The method may include receiving imaging of the luminal network and generating the 3D model from the imaging. The method of any may include detecting a location and orientation of a robotic surgical instrument. The method may include displaying a representation of the robotic surgical instrument in a user interface displaying the 3D model. The method of may include generating a surgical plan including one or more of a pathway for endoluminal navigation of the catheter to the target, a resection line, a suture line, blood vessels, margin, fissures, a no-go zone, critical structures The method of any may include detecting a change in position of a patient after navigation of the catheter in proximity to the target. The method may include deforming the 3D model and the surgical plan. The method may include the deformed surgical plan on the laparoscopic image captured by the laparoscopic camera. Implementations of the described techniques may include hardware, a method or process, or computer software on a computer-accessible medium.BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Various aspects and embodiments of the disclosure are described hereinbelow with references to the drawings, wherein:PATENT APPLICATIONA0013105W001 (00017-01309PCT00)

[0013] FIG. l is a schematic view of a laparoscopic surgical system in accordance with the disclosure;

[0014] FIG. 2 is a schematic view of an endoluminal navigation system in accordance with the disclosure;

[0015] FIG. 3 is a flow chart depicting a method in accordance with the disclosure;

[0016] FIG. 4 depicts a laparoscopic image on a display and a 3D model in a user interface on a second display in accordance with the disclosure;

[0017] FIG. 5 depicts a laparoscopic image with an indication of a location of a target in accordance with the disclosure;

[0018] FIG. 6 depicts a laparoscopic image with an indication of a location of a target and other clinically relevant data in accordance with the disclosure.

[0019] FIG. 17 is a schematic representation of an endoluminal navigation system in accordance with the disclosure.DETAILED DESCRIPTION

[0020] In accordance with aspects of the disclosure, described herein are systema and methods of identifying an area of interest within an organ in pre-operative images and then presenting an indication of the location of that area of interest on live laparoscopic images. Particularly where the area of interest (e.g., tumor or lesion) is within the organ, and cannot be seen in laparoscopic images. In this manner, the disconnect between the pre-operative scans and 3D models to the laparoscopic images can be bridged.

[0021] In accordance with the disclosure a pre-operative scan is undertaken of a relevant organ or portion of the patient and an area of interest or target (e.g., tumor, lesion, or organ segment) is identified in the images of an organ (e.g., the lungs, liver, pancreas, etc.). An application (e.g., planning and navigation software) is employed to generate an endoluminal pathway to or near the area of interest. The combination of the area of interest and the endoluminal pathway to arrive at the area of interest is a navigation plan for navigation of a catheter to the area of interest. Other clinically relevant data may also be included such as margins, resection lines, the location of blood vessels, etc. resulting in a surgical plan.

[0022] With the navigation plan established, a catheter is navigated to the area of interest. With the catheter navigated to the area of interest, the tracking system determines coordinates of a distal portion of the catheter (e.g., a sensor) in the coordinates of the tracking system. The laparoscopic tools (e.g., laparoscope) can be tracked by the trackingPATENT APPLICATIONA0013105W001 (00017-01309PCT00) system, or the coordinates of their locations and orientations can be converted to a common coordinate system for use by the tracking system. As a result, when the field of view of the laparoscope is manipulated so that the location of the catheter (unseen within the organ) is within the field of view, and indicator is displayed on the live images. The provision of this indicator on the images allows the surgeon to orient themselves with regard to the location of the relevant tissue for the procedure and then to proceed with confidence that they are acting on the correct tissue, with knowledge of surrounding tissues (e.g., fissures, segments, blood vessels, etc.) to quickly perform the procedure. The clinically relevant data may also be displayed on the live images. In some instances, a deformation model may be applied to the 3D model of the surgical plan to account for deformation of the organ as a result of changing positions of the patient or other factors to accurately display the clinically relevant data on the live fluoroscopic images.Laparoscopic Surgical Robotics

[0023] FIG. 1 depicts a laparoscopic surgical robotic system 10 including a control tower 20, which is connected to all of the components of the laparoscopic surgical robotic system 10 including a surgeon console 30 and one or more mobile carts 60. Each of the mobile carts 60 includes a robotic arm 40 having a surgical instrument 50 removably coupled thereto. The robotic arms 40 also couple to the mobile carts 60. The robotic system 10 may include any number of mobile carts 60 and / or robotic arms 40.

[0024] The surgical instrument 50 is configured for use during minimally invasive surgical procedures (e.g., laparoscopic). The surgical instrument 50 include an end effector 49 such as an electrosurgical forceps configured to seal tissue by compressing tissue between jaw members and applying electrosurgical current thereto, a surgical stapler including a pair of jaws configured to grasp and clamp tissue while deploying a plurality of tissue fasteners, e.g., staples, and cutting stapled tissue, a surgical clip applier including a pair of jaws configured apply a surgical clip onto tissue or other end effectors without departing from the scope of the disclosure.

[0025] Though generally described herein in connection robotic surgical approaches the term “laparoscopic” can be interpreted to refer to either or both manually manipulated tools or robotic tools configured for laparoscopic approach.

[0026] One of the robotic arms 40 may include a laparoscopic camera 51 configured to capture video of the surgical site. The laparoscopic camera 51 may be a stereoscopicPATENT APPLICATIONA0013105W001 (00017-01309PCT00) endoscope configured to capture two side-by-side (i.e., left and right) images of the surgical site and produce a video stream of the surgical scene. The laparoscopic camera 51 is coupled to an image processing device, which may be disposed within the control tower 20. The image processing device may be any computing device configured to receive the image feed from the laparoscopic camera 51 and output the processed images or video stream.

[0027] The surgeon console 30 includes a first screen 32, which displays a video feed of the surgical site provided by laparoscopic camera 51 of the surgical instrument 50 disposed on the robotic arm 40, and a second screen 34, which displays a user interface for controlling the surgical robotic system 10. The first screen 32 and second screen 34 may be touchscreens allowing for displaying various graphical user inputs.

[0028] The surgeon console 30 also includes a plurality of user interface devices, such as foot pedals 36 and a pair of hand controllers 38a and 38b which are used by a user to remotely control robotic arms 40. The surgeon console further includes an armrest 33 used to support clinician's arms while operating the hand controllers 38a and 38b.

[0029] The control tower 20 includes a screen 23, which may be a touchscreen, and outputs on the graphical user interfaces (GUIs). The control tower 20 also acts as an interface between the surgeon console 30 and one or more robotic arms 40. In particular, the control tower 20 is configured to control the robotic arms 40, so as to move the robotic arms 40 and the corresponding surgical instrument 50, based on a set of applications or programmable instructions and / or input commands from the surgeon console 30, in such a way that robotic arms 40 and the surgical instrument 50 execute a desired movement sequence in response to input from the foot pedals 36 and the hand controllers 38a and 38b. The foot pedals 36 may be used to enable and lock the hand controllers 38a and 38b, that are employed to repositioning the laparoscopic camera 51 and surgical instruments 50 as well as trigger electrosurgical activation / deactivation and other features of the surgical instruments 50. In particular, the foot pedals 36 may be used to perform a clutching action on the hand controllers 38a and 38b. Clutching is initiated by pressing one of the foot pedals 36, which disconnects (i.e., prevents movement inputs) the hand controllers 38a and / or 38b from the robotic arm 40 and corresponding surgical instrument 50 or laparoscopic camera 51 attached thereto. This allows the user to reposition the hand controllers 38a and 38b without moving the robotic arm(s) 40 and the surgical instrument 50 and / or laparoscopic camera 51. This is useful when reaching control boundaries of the surgical space.PATENT APPLICATIONA0013105W001 (00017-01309PCT00)

[0030] Each of the control tower 20, the surgeon console 30, and the robotic arm 40 includes a respective computer 21, 31, 41. The computers 21, 31, 41 are interconnected to each other using any suitable communication network based on wired or wireless communication protocols. The term “network,” whether plural or singular, as used herein, denotes a data network, including, but not limited to, the Internet, Intranet, a wide area network, or a local area network, and without limitation as to the full scope of the definition of communication networks as encompassed by the present disclosure. Suitable protocols include, but are not limited to, transmission control protocol / internet protocol (TCP / IP), datagram protocol / internet protocol (UDP / IP), and / or datagram congestion control protocol (DC). Wireless communication may be achieved via one or more wireless configurations, e.g., radio frequency, optical, Wi-Fi, Bluetooth (an open wireless protocol for exchanging data over short distances, using short length radio waves, from fixed and mobile devices, creating personal area networks (PANs), ZigBee® (a specification for a suite of high level communication protocols using small, low-power digital radios based on the IEEE 122.15.4-1203 standard for wireless personal area networks (WPANs)).

[0031] The computers 21, 31, 41 may include any suitable processor (not shown) operably connected to a memory (not shown), which may include one or more of volatile, non-volatile, magnetic, optical, or electrical media, such as read-only memory (ROM), random access memory (RAM), electrically-erasable programmable ROM (EEPROM), non-volatile RAM (NVRAM), or flash memory. The processor may be any suitable processor (e.g., control circuit) adapted to perform the operations, calculations, and / or set of instructions described in the present disclosure including, but not limited to, a hardware processor, a field programmable gate array (FPGA), a digital signal processor (DSP), a central processing unit (CPU), a microprocessor, and combinations thereof. Those skilled in the art will appreciate that the processor may be substituted for by using any logic processor (e.g., control circuit) adapted to execute algorithms, calculations, and / or set of instructions described herein.Endoluminal Navigation

[0032] FIG. 2 is a perspective view of an exemplary endoluminal navigation system for facilitating navigation of a medical device (e.g., a catheter) to a target within or near a luminal network of the body of a patient (e.g., airways of the lungs). As shown in FIG. 2, catheter 102 is part of a catheter guide assembly 106. The catheter guide assembly 106 mayPATENT APPLICATIONA0013105W001 (00017-01309PCT00) be mounted to one of the robotic arms 40, or a separate driving mechanism (not shown) for advancement and articulation of the catheter 102. In one embodiment, catheter 102 is inserted into the patient P for navigation through a patient’s luminal network (e.g., the airways). The catheter 102 may itself include imaging capabilities via an integrated camera or optics component 109, a separate bronchoscope 108 may also be employed but is not required. A locatable guide (LG) 110 (a second catheter), including a sensor 104 may be optionally inserted into catheter 102 and locked into position such that sensor 104 extends a desired distance beyond the distal tip of catheter 102. The position and orientation of sensor 104 relative to a reference coordinate system, and thus the distal portion of catheter 102, within an electromagnetic field can be derived. Catheter guide assemblies 106 are currently marketed and sold by Medtronic PLC under the brand names SUPERDIMENSION® Procedure Kits, or EDGE™ Procedure Kits, and are contemplated as useable with the disclosure. Though not shown here, the catheter guide assembly 106 and catheter 102 may be mounted on, advanced, and articulated via one of the robotic arms 40 without departing from the scope of the disclosure.

[0033] As shown an operating table 112 is configured to support a patient P for the endoluminal navigation procedure and the same or a similar operating table is employed to support a patient P undergoing a robotic laparoscopic surgical procedure employing system 10 (FIG. 1). Monitoring equipment is coupled to catheter 102 (e.g., a video display 114, for displaying the video images received from a video imaging system of the catheter 102); a locating or tracking system 115 including a locating module 116, a plurality of reference sensors 118 and a transmitter mat 120 including a plurality of incorporated markers (not shown). A computing device 122 includes software and / or hardware used to facilitate identification of a target, pathway planning to the target, navigation of a medical device to the target, and / or confirmation and / or determination of placement of catheter 102, or a suitable medical tool or surgical device, relative to the target.

[0034] Computing device 122 may be any suitable computing device including a processor and storage medium, wherein the processor is capable of executing instructions (e.g., applications) stored on the storage medium. Computing device 122 may further include a database configured to store patient data, CT data sets including CT images, fluoroscopic data sets including images and video, 3D reconstruction, navigation plans, and any other such data. Although not explicitly illustrated, computing device 122 may includePATENT APPLICATIONA0013105W001 (00017-01309PCT00) inputs, or may otherwise be configured to receive, CT data sets, fluoroscopic images / video and other data described herein. Additionally, computing device 122 includes a display configured to display graphical user interfaces. Computing device 122 may be connected to one or more networks through which one or more databases may be accessed.

[0035] Pre-procedural and intra-procedural images of the patient (e.g. computed tomography (CT) or magnetic resonance (MR) images) of the patient are acquired and may be saved on or accessed by the computing device 122. A planning application stored on the computing device 122 enables the review of the images, identification of one or more targets (e.g., tumors or lesions) within the images. The planning application is configured to generate a 3D model from the images of the relevant organ including the target (e.g., tumor or lesion). Thus, for example, a 3D model of the lungs can be generated from images of the thoracic cavity. Aspects of the planning procedure generate a 3D model depicting the airways from the trachea through multiple bifurcations, which can be displayed in a user interface on the computing device 122. With the targets identified in the images and the 3D model generated, the planning application further enables the generation of an endoluminal path through a natural orifice of the patient (e.g., the mouth) which the catheter 102 can be navigated to acquire biopsy samples of the target, or to apply therapy to the target (e.g., microwave, radio-frequency, chemical, or cryogenic ablation, or another therapy). The pathway plan may be developed prior to or immediately preceding the navigation procedure and transmitted to or accessed by a navigation application. The navigation application may also be stored on the computing device 122 and employed to guide the navigation of the catheter guide assembly 106 and the catheter 102 to arrive at the target(s) as described further below.

[0036] Catheter 102 can be navigated within the patient and the tracking system 115 (e.g., an electromagnetic (EM) tracking system, fiber-Bragg grating flex sensor tracking system, inertial tracking, wireless ranging, optical tracking, or other suitable system for determining position and orientation of a distal portion of the catheter 102) is utilized, to detect a position of the sensor 104 within the patient. By detecting the location of the locatable guide 110 or sensor 104 on the catheter 102 a representation of a portion the catheter 102 can be displayed in the 3D model.

[0037] When employing an EM tracking system, a transmitter mat 120 is positioned beneath patient P. Transmitter mat 120 generates an electromagnetic field around at least aPATENT APPLICATIONA0013105W001 (00017-01309PCT00) portion of the patient P within which the position of a plurality of reference sensors 118 and the sensor 104 can be determined with use of a tracking module 116. A second (and optionally a third) electromagnetic sensor 126 may also be incorporated into the end of the catheter 102. The second electromagnetic sensor 126 may be a five degree-of-freedom sensor or a six degree-of-freedom sensor. These sensors 126 can be employed to determine the location and orientation of a distal portion of the catheter 102 even when the LG 110 with sensor 104 has been removed from the catheter. Further, the sensors 126 enable use of the catheter guide assembly 106 without requiring the LG 110 and sensor 104. In addition, one or more reference sensors 118 are attached to the chest of the patient P.

[0038] To enable accurate display of the representation of the catheter 102 in the 3D model a registration method must be performed. Registration matches the coordinate locations of the three-dimensional model and two-dimensional images from the procedure planning phase, in which a pathway and targets for biopsy or therapy were defined, with the position and orientation of patient P’s airways as they are on the operating table 112. Registration thus allows for the navigation of the organ, and its luminal network to be undertaken with confidence and with accurate depiction of changes in location of the catheter 102 being updated in the 3D model of the luminal network.

[0039] In one embodiment of the disclosure, registration of the patient P’s location on the transmitter mat 120 may be performed by moving sensor 104 or 126 through the airways of the patient P. More specifically, data pertaining to locations of sensor 104 or 126 (e.g., within an electromagnetic field generated by the transmitter mat 120) as the sensor 104 or 126 is moving through the airways, is recorded using the tracking system 115 and optionally with reference to the locations of the reference sensors 118. A shape resulting from this location data is compared to an interior geometry of passages of a 3D model, and a location correlation between the shape and the 3D model based on the comparison is determined, e.g., utilizing the navigation application on computing device 122. The software aligns, or registers, an image representing a location of sensor 104 within the 3D model and / or two- dimensional images based in part on the recorded location data and an assumption that sensors 104 or 126 remain located in non-tissue space in patient P's airways. Alternatively, a manual registration technique may be employed by navigating the sensors 104 or 126 to pre-specified locations in the lungs of the patient P, and manually correlating the images from the bronchoscope to the model data of the three-dimensional model.PATENT APPLICATIONA0013105W001 (00017-01309PCT00)

[0040] Though described herein with respect to EMN systems using EM sensors (e.g., an air coil sensor, a tunnel magneto resistive sensor, or others), the instant disclosure is not so limited and may be used in conjunction with flexible sensors such as fiber-Bragg grating sensors, inertial measurement unit (IMU), ultrasonic sensors, or without sensors. Additionally, as described elsewhere herein, the methods described herein may be used in conjunction with manual systems such as depicted in FIG. 2 or motorized or robotic systems where the catheter 102 or catheter guide assembly 106 are integrated with a robotic arm 40 (FIG. 1) such that robotic actuators drive and articulate the catheter 102 within the patient to the desired target.

[0041] Further though generally described herein with respect to endoluminal navigation, the disclosure is not so limited and can be utilized in intraluminal navigation. In intraluminal navigation an opening is created in the natural lumen (e.g., airways) with a piercing device and the catheter 102 is advanced beyond the lumen wall. This approach can be employed with seeking to biopsy or treat tumors and tissue beyond the airway wall or access other adjacent structures including lymph nodes and other structures outside the luminal wall.

[0042] Still further, systems and methods described herein the endoluminal catheter 102 may be employed both single-channel endoluminal robotics (“SC ELR”) applications and multi-channel endoluminal robotics (“MC ELR”) applications. SC ELR applications may include, for example, thoracic and pancreatic procedures. MC ELR applications may include, for example, procedures in the colon, including procedures that involve a combination of endoluminal and laparoscopic or surgical robotic workflows. These and other endoluminal applications considered within the scope of this disclosure.

[0043] In accordance with aspects of the disclosure, acquisition of intraprocedural images enables the visualization of intra-body navigation of a medical device (e.g., a biopsy tool or a therapy tool), towards a target (e.g., a lesion) may be a portion of a larger workflow of a navigation system. An imaging device 124 (e.g., a fluoroscope or a CT or cone beam CT imaging device such as the Medtronic 0-arm™ surgical imaging system) capable of acquiring 2D and 3D images or video of patient P is also included in this particular aspect of system 100. The images, sequence of images, or video captured by imaging device 124 may be stored within the imaging device 124 or transmitted to computing device 122 for storage, processing, and display. Additionally, imaging device 124 may move relative to thePATENT APPLICATIONA0013105W001 (00017-01309PCT00) patient P so that images may be acquired from different angles or perspectives relative to patient P to create a sequence of images, such as a fluoroscopic video. The pose of imaging device 124 relative to patient P while capturing the images may be estimated via markers incorporated with the transmitter mat 120. The markers are positioned under patient P, between patient P and operating table 112 and between patient P and a radiation source or a sensing unit of imaging device 124. The markers incorporated with the transmitter mat 120 may be two separate elements which may be coupled in a fixed manner or alternatively may be manufactured as a single unit. Imaging device 124 may include a single imaging device or more than one imaging device.

[0044] While endoluminal navigation of a catheter 102 to conduct a procedure from within the luminal network avoids the need to pierce the patient and insert ports to gain access to the targets, not every procedure can be performed with an endoluminal approach. An example of a procedure that cannot be performed with an endoluminal approach is a lung segmentectomy. In a lung segmentectomy, a portion of the lung (a segment) is severed from the remaining lung and the line along which the segment is severed is sealed, often simultaneously. The severing of the segment removes the diseased tissue (e.g., containing tumor or lesion) from the health tissue limiting the spread of the disease. The severed segment is then removed from the patient via one of the ports.

[0045] As noted above, pre-operative images are useful in identifying a segment for removal, and to generally allow for planning of a procedure. Often a 3D model is not generated, and even if a 3D model is generated due to the differences in positioning of the patient required for imaging (usually supine) and many procedures (lateral side-lying), the changes to softer-tissue organs (e.g., the lungs), and their locations within the patient, make the 3D models difficult to reconcile with the views of the organ or other tissues during through a laparoscopic camera 51. Further, registration of the pre-procedure images or 3D models can be very challenging, in part due to this same softness of the tissues and the changes in positioning of the patient from imaging to procedure. As a result, the laparoscopic approach, whether manual or robotic, relies on the surgeon’s abilities to visualize the tissue for treatment, and the ability to distinguish that tissue from other healthy tissues.

[0046] To provide for pre-procedural planning and improved intra-procedure guidance for robotic laparoscopic procedures, the disclosure contemplates a combined endoluminalPATENT APPLICATIONA0013105W001 (00017-01309PCT00) and laparoscopic approach. In accordance with this combined approach the endoluminal approach, as described above in connection with FIG. 2 including the receipt of preprocedural images or intra-procedural images, the identification of targets, generation of a 3D model, and generation of the endoluminal pathway through the luminal network and 3D model to the identified target. Using the pathway plan and following registration of the pathway plan and to the patient a catheter 102 is navigated to an identified target and tracked through the patient with a representation of the catheter 102 in the 3D model.

[0047] In accordance with one aspect of the disclosure, in addition to identifying the target in the pre-procedure or intraprocedural images, the planning may include the identification a desired margin around the target to ensure that the tumor or lesion is completely removed. In addition, other clinically relevant data including critical structures (e.g., blood vessels, fissures, etc.) can be identified and their relative positions to the target can be saved in the planning. The identification of such structures, as described below, can be utilized to define a no-fly zone which the surgical instruments 50 should avoid or at least location of which the surgeon need be aware of during the procedure. The identification of these structures can be tied to a variety of cautions or warnings that may be displayed in the screen 23 of the console 30 or on a display of computing device 122.

[0048] Each surgical instrument 50 and / or robotic arm 40 includes a sensor 53 (e.g., an EM sensor). The sensor 53 allows the tracking system 115 to detect a position and orientation of robotic arms 40 or the surgical instruments 50 or laparoscopic camera 51. As a result, and based on the registration process of the catheter 102 and the sensor 104 / 126, the laparoscopic surgical system 10, and particularly the surgical instruments 50, laparoscopic camera 51, and the robotic arms 40 employing sensors 53, is registered to the patient, the 3D model, and the pre-procedure or intra-procedure images from which the 3D model is generated. Representations of each surgical instrument 50 can be depicted on the 3D model.

[0049] With the catheter 102 in place with sensor 104 / 126 detecting its position within the patient, and registration complete the relative positions of the surgical instruments 50 and the catheter 102 can be determined by the tracking system 115. This relative position is based at least on the detected positions of the sensors 104 / 126 and 53. A representation of their relative positions of the catheter 102 and the surgical instruments 50 can be displayed in the 3D model on a display of the computing device 122 or on the screen 23 ofPATENT APPLICATIONA0013105W001 (00017-01309PCT00) the control tower 20. In some instances, displaying the relative positions may require a zooming out of the display of the 3D model, or the addition of features and aspects of the 3D model to the display (e.g., a 3D model of the thoracic cavity as opposed to just the airways of the lungs for endoluminal navigation).

[0050] In accordance with aspects of the disclosure, during a procedure, the initial navigation of the of the catheter 102 within the luminal network of the patient occurs with the patient in the supine position. With registration complete the patient can be moved so that they are on in a lateral side-laying position where laparoscopic procedures, particularly of the thoracic cavity are undertaken. As noted above, this movement causes the organs of the thoracic cavity to shift due to gravity. The shift of the organs within the thoracic cavity provides space within the thoracic cavity for the insertion of the surgical instruments 50 to navigate and articulate within the thoracic cavity. Further, with particular reference to the lungs, where a procedure such as a segmentectomy is undertaken, the lung (right or left) or a relevant lung lobe is isolated from ventilation and deflated. This deflation increases the volume of the thoracic cavity for the surgical instruments 50, and also improves the ability of the surgical instruments 50 and their end effectors to manipulate and act on the tissue of the deflated lung or lobe. Though not subject to deflation, changes in position have similar effects in movement of soft organs within the abdomen.

[0051] As will be appreciated, change in position of the patient and the deflation of lung or lobe limits the utility of the 3D model generated from the pre-procedure or intraprocedural images. However, despite the 3D model having potentially less utility, the tracking system 115 identifies the relative position and orientation of the sensor 104 / 126 on the catheter 102 and the sensors 53 on surgical instruments 50, laparoscopic camera 51, or the robotic arms 40. Since the sensor 104 / 126 on the catheter 102 has been navigated to the target, the position and orientation of the sensor 104 / 126 acts as a beacon to which the surgical instruments 50 and laparoscopic camera 51 are to be navigated to act on and image the tissue of the target.

[0052] With regards to the images captured by the laparoscopic camera 51, as the field of view of view of the laparoscopic camera 51 is manipulated, its position and orientation can be detected. As a result, the application can determine when the orientation of the field of view of the laparoscopic camera 51 is directed at the sensor 104 / 126 on the catheter 102PATENT APPLICATIONA0013105W001 (00017-01309PCT00) that has been navigated to a location within the patient but is not visible as it is within an organ or duct.

[0053] When the laparoscopic camera 51 is oriented such that the sensor 104 / 126 (though not visible as it is within the organ) is within the field of view of the laparoscopic camera 51, an indicator of the location of the sensor 104 / 126 is depicted in the live images captured by the laparoscopic camera 51 displayed in one or more of the screens 23, 32. This indicator may optionally include other clinically relevant data. For example, locations of fissures, blood vessels, segments, and other data that can be beneficial to the surgeon. In this way, once the laparoscopic camera 51 directed at the sensor 104 / 126 on the catheter 102 that has been navigated to or near the target, the surgeon can quickly manipulate the laparoscopic tools 50 to perform the necessary procedure (e.g., a segmentectomy) with confidence that they are considering the target tissue and ensuring that the target tissue is acted on.

[0054] In accordance with aspects of the disclosure, one or more deformation algorithms may be stored in the memory of the computing devices 122, 21, 31, 41. Based on the determined change in position of the sensor 104 / 126 (e.g., when moved from supine to lateral side lying and / or deflation, the deformation algorithm may be employed to estimate a deformation of the relevant organ (e.g., the lungs) as a result of the move. The estimate of deformation can be applied to the 3D model. Data from the deformed 3D model (e.g., critical structures, pathways, targets, margins, and others can then be displayed on the live images captured by the laparoscopic camera 51. To improve the deformation modeling, multiple sensors 104 / 126 may be located on the catheter 102 along its length such that the deformation of the portion of the luminal network that has been navigated can be more accurately modeled.

[0055] The deformed 3D model is necessarily registered with the patient by virtue of the catheter 102 having been navigated to the target and remaining within the patient during the change in orientation. The presentation of features such as the locations of major blood vessels and fissures may be a semi-lucent or translucent overlay such that the surgeon is able to see through them to the underlying tissue. In addition, the location for the presentation on the images need not be exact, but rather can be generally in the locations of the expected deformation to provide reminders or guidance to the surgeon to be on the lookout for the features during the procedure. This may be particularly relevant when thePATENT APPLICATIONA0013105W001 (00017-01309PCT00) tissue is on the opposite side of the tissue from that being imaged via the laparoscope. In some instances, these tissues on the opposite side of the tissue may include an indicator of depth from the surface of the tissue being viewed via the laparoscope in an effort to prevent the surgeon from advancing the surgical instruments 50 beyond a desired depth.

[0056] As will be appreciated though the relative positions of the sensors 53 on surgical instruments 50 and laparoscopic camera 51, and the sensor 104 / 126 on the catheter 102 are determined and displayed, the laparoscopic surgery continues relying largely on the surgeon’s experience and skill. However, the surgeon’s experience is further informed by the now known location of the target, as identified by the location of the sensor 104 / 126 on the catheter 102, depicted in the images captured laparoscopic camera 51 thus greatly reducing the cognitive burden on the surgeon in understanding their environment within the patient. Further, other clinically relevant data, as described above, may also be displayed on the live laparoscopic images, to further assist the surgeon in carrying out the procedure.

[0057] FIG. 3 depicts a method 200 in accordance with the disclosure. At step 202 images from a pre-operative scan are received by an application, for example, running on computing devices 122, 21, 31, 41, or another computing device such as a laptop or tablet. Reviewing the images, one or more targets within the images are identified at step 204. The targets may be automatically identified or manually identified, or a combination of both. In addition to the targets, other aspects of the laparoscopic procedure that may be identified either manually or automatically include identification of a segment or lobe (e.g., lung or liver) for resection, laparoscopic port placement locations, or other clinically relevant data. With the targets identified, a 3D model of the organ or portion of the patient is generated at step 206. The 3D model may, where appropriate, including a pathway from, for example, a natural or artificial opening through a luminal network to the target within the organ or portion of the patient. As described above, this pathway is an endoluminal pathway along which a catheter can be navigated to arrive at the target. In addition to identifying the target and generating the endoluminal pathway to arrive at the target, the application may enable manual identification of or automatically identification of clinically relevant data regarding the organ or portion of the patient near the target. The clinically relevant data may include blood vessels, lung segments, fissures, lung or liver lobes, the head of pancreas, margins around tumors, critical structures to be avoided such as nerve ganglia, gall and bile ducts, the ureter, and others. In addition, the application may provide functionality for the clinicianPATENT APPLICATIONA0013105W001 (00017-01309PCT00) to depict resection lines, suture lines, staple lines, no-go zones, etc. Thus, beyond a navigation plan, the application enables the clinician to generate a complete surgical plan for the procedure. As will be appreciated, the application can present images and views including 3D modeling views for both the endoluminal and laparoscopic aspects of the procedure to provide a complete surgical plan. The surgical plan, with all of the clinically relevant data, whether automatically generated or input by the surgeon is saved to a recording media (e.g., local memory, server or cloud storage, etc.).

[0058] At step 208, the surgical plan is loaded into a navigation application running on a computing device 122, 21, 31, 41 in the operating room. With the surgical plan loaded, and the patient placed on the table 112 and the tracking system 115 enabled, the catheter 102 is inserted into the luminal network of the patient (e.g., into the airways). In one aspect of the disclosure employing EM navigation, once inserted into the patient the sensor 104 / 126 on the catheter 102 detects the EM fields generated by the transmitter mat 120. At step 210, by collecting data via the tracking system 115 as the catheter 102 is inserted into the luminal network a cloud of data points are collected. By matching this cloud of data points to the 3D model of the luminal network, the surgical plan is registered to the patient at step 212. With the surgical plan registered to the patient, at step 212 the endoluminal pathway through the luminal network can be followed and the catheter 102 advanced further into the patient. Changes in the location of the sensors 104 / 126 on the catheter 102, as it is advanced into the luminal network, are depicted in the 3D model. The catheter 102 is advanced along the endoluminal pathway until a distal portion of the catheter 102 and particularly the sensor 104 / 126 is at or near the target (e.g., tumor or segment).

[0059] As noted above, in accordance with one aspect of the disclosure each robotic arm 40 of the laparoscopic robotic system 10 includes a sensor 53 that is detectable by the tracking system 115. The detection position and orientation of the sensor 53 provides an indication of the location of the robotic arm 40 relative to the sensors 104 / 126. Because the geometry of the laparoscopic camera 51 connected to the robotic arm 40 is known, the relative position of the sensor 53 to the end of the laparoscopic camera 51 is known even as the laparoscopic camera 51 is moved by the robotic arm 40. Thus, at step 214 once the sensor 53 is detected by the tracking system 115, the robotic arm 40 is registered to the patient and the surgical plan. Accordingly, as a result of the registration, any movement of the robotic arm 40 and the laparoscopic camera 51, or surgical instruments 50 on anotherPATENT APPLICATIONA0013105W001 (00017-01309PCT00) robotic arm 40 is detected by the tracking system 115 and a representation of the surgical instruments 50 can be displayed with reference to the 3D model on screen 23, a live laparoscopic images depicting the surgical instrument can simultaneously be displayed in screen 32, as depicted in FIG. 4. Those of skill in the art will recognize that the display of the 3D model and images may be in any or all of the screens and may be presented together in a single screen to provide visual information to the surgeon without departing from the scope of the disclosure.

[0060] In some instances, the 3D model, as shown in FIG. 4, is depicted from the viewpoint of the laparoscopic camera 51, through other orientation may be depicted or selected by the surgeon as desired. In a further aspect, a home button may be presented on the user interface in the display, which by selection will return the view to the viewpoint of the laparoscopic camera. As described above, the viewpoint of the laparoscopic camera 51 is determined via the tracking system 115 detecting the position and orientation of sensor 53 and the known geometry of the robotic arm 40.

[0061] As described herein, the tracking system 115 is an EM tracking system and the sensor 53 is an EM sensor. Utilization of a single tracking system 115 can reduce the complexity of the overall system but is not strictly required. In an alternative arrangement, the sensors 53 may be optical sensors, encoders, or others. These sensors 53, may be detected relative to one another to determine the locations of each robotic arm 40 in the operating room. During a procedure each robotic arm 40 may be driven to a particular point, e.g., each of the three reference sensors 118 to the place a distal end of a surgical instrument 50 or laparoscopic camera 51 at a known location in in the EM field generated by the transmitter mat. By observing subsequent movements of the robotic arm 40, the position and orientation of the surgical instrument 50 or laparoscopic camera 51 detected by the other sensor 53 arrangement can be translated to in coordinate system of the EM field. While the translation from one sensor coordinate system to a second coordinate system is potentially more complex it is another method of determining the relative position of a laparoscopic camera 51 and surgical instrument 50 connected to the robotic arms to the sensors 104 / 126 of the catheter 102.

[0062] Regardless of the type of sensor system employed, the navigation application receives input from the tracking system 115, and / or the other sensors and at step 216 can determine whether the sensor 104 / 126 at or near the end of the catheter 102 is within thePATENT APPLICATIONA0013105W001 (00017-01309PCT00) field of view (“FOV”) of the laparoscopic camera 51. If the sensor 104 / 126 is within the field of view of the laparoscopic camera 51 then an indicator of the location of the sensor 104 / 126, which is indicative of the location of the target tissue, is displayed in the live laparoscopic images at step 220 as depicted in FIG. 5. If not, the method returns to step 214 where the position and orientation of the laparoscopic camera 51 continues to be displayed relative to the 3D model.

[0063] As noted above, in addition to the indication of the location of the sensor 104 / 126 being displayed on the live laparoscopic images a variety of additional data may be displayed. The data may include an indication of a distance and direction of the surgical instrument 50 to the target tissue, margins, resection lines, the location of blood vessels or other anatomical structures, no-go zones, critical structures to be avoided, and others without departing from the scope of the disclosure. With regards the distance and direction of the surgical instrument 50 to the target this data may be presented via the user interface by a ruler or other virtual measuring overlay, this may be displayed between the distal tip of the surgical instrument 50 and the target or in a side panel, or other data section of the user interface.

[0064] Those of skill in the art will recognize that, for some procedures, for example lung lobe or segment resections (segmentectomies), the lung as observed via the laparoscopic camera 51 will appear very differently from the 3D model which is formed from the pre-operative or intraoperative scans. The pre-operative and intraoperative scans are typically captured at full breath hold with the patient in the supine position, while the resection surgery is typically undertaken in the lateral side lying position with the lung being resected in a deflated state. As a result, any features which have been identified in the 3D model or pre-operative images, will have moved, sometimes considerably. As described herein, the magnitude of the movement may be measured by the change in position and orientation experienced by the sensor 104 / 126 after navigation to the target location. The change in position and orientation data in combination with one or more deformation algorithms can be employed by the navigation application to generate a deformed 3D model at optional step 218 prior to display of the location of the sensor 104 / 126 in the live images captured by the laparoscopic camera 51. The utilization of the deformation algorithm allows for more accurate depiction of the other data from the surgical plan (e.g., resection lines, blood vessels, margins, etc.) on the images from laparoscopic camera as depicted in FIG. 6.PATENT APPLICATIONA0013105W001 (00017-01309PCT00)

[0065] Reference is now made to FIG. 7, which is a schematic diagram of a computing system 300 configured for use with the methods of the disclosure including the method 200 of FIG. 3. The computing system may be found in any of computing device 122 and computers 21, 31, 41. Further, aspects of the computing system 300 may be connected to one or more but not necessarily all of the computing device 122 or computer 21, 31, 41 (e.g., a distributed computing system) without departing from the scope of the disclosure. System 300 may include a workstation 301, and optionally an imaging device 315 (e.g., a fluoroscope, CT imaging device, or an ultrasound imaging device). In some embodiments, workstation 301 may be coupled with imaging device 315, directly or indirectly, e.g., by wireless communication. Workstation 301 may include a memory 302, a processor 304, a display 306 and an input device 310. Processor or hardware processor 304 may include one or more hardware processors. Workstation 301 may optionally include an output module 312 and a network interface 308. Memory 302 may store an application 318 and image data 314. Application 318 may include instructions executable by processor 304 for executing the methods of the disclosure including the method of FIG. 3.

[0066] Application 318 may further include a user interface 316. Image data 314 may include the CT scans, the generated fluoroscopic 3D reconstructions of the target area and / or any other fluoroscopic image data and / or the generated one or more slices of the 3D reconstruction. Processor 304 may be coupled with memory 302, display 306, input device 310, output module 312, network interface 308 and imaging device 315. Workstation 301 may be a stationary computing device, such as a personal computer e.g., computing device 122, or a portable computing device such as a tablet computer, or further be implemented on the console 30, further, workstation 301 may embed a plurality of computer devices.

[0067] Memory 302 may include any non-transitory computer-readable storage media for storing data and / or software including instructions that are executable by processor 304 and which control the operation of workstation 301 and, in some embodiments, may also control the operation of imaging device 315. Imaging device 315 may be used to capture a sequence of fluoroscopic images based on which the fluoroscopic 3D reconstruction is generated and to capture a live 2D fluoroscopic view according to this disclosure. In an embodiment, memory 302 may include one or more storage devices such as solid-state storage devices, e.g., flash memory chips. Alternatively, or in addition to the one or more solid-state storage devices, memory 302 may include one or more mass storage devicesPATENT APPLICATIONA0013105W001 (00017-01309PCT00) connected to the processor 304 through a mass storage controller (not shown) and a communications bus (not shown).

[0068] Although the description of computer-readable media contained herein refers to solid-state storage, it should be appreciated by those skilled in the art that computer-readable storage media can be any available media that can be accessed by the processor 304. That is, computer readable storage media may include non-transitory, volatile, and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data. For example, computer-readable storage media may include RAM, ROM, EPROM, EEPROM, flash memory or other solid-state memory technology, CD-ROM, DVD, Blu-Ray or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which may be used to store the desired information, and which may be accessed by workstation 301.

[0069] Application 318 may, when executed by processor 304, cause display 306 to present user interface 316. User interface 316 may be configured to present to the user a single screen including a three-dimensional (3D) view of a 3D model, a live two- dimensional (2D) fluoroscopic view showing the medical device, and a target view, which corresponds to the 3D model of the target, overlaid on the live images (e.g., CT of fluoroscopic). User interface 316 may be further configured to display the target in different colors depending on whether the medical device tip is aligned with the target in three dimensions. Still further, in connection with the above descriptions the user interface 316 can be configured to depict any of the aspects of the endoluminal pathway or the laparoscopic surgical plan in the 3D model, deformed 3D model, or the intraprocedural images.

[0070] Network interface 308 may be configured to connect to a network such as a local area network (LAN) consisting of a wired network and / or a wireless network, a wide area network (WAN), a wireless mobile network, a Bluetooth network, and / or the Internet. Network interface 708 may be used to connect between workstation 301 and imaging device 315. Network interface 708 may also be used to receive image data 314. Input device 310 may be any device by which a user may interact with workstation 301, such as, for example, a mouse, keyboard, foot pedal, touch screen, and / or voice interface. Output module 312 may include any connectivity port or bus, such as, for example, parallel ports, serial ports,PATENT APPLICATIONA0013105W001 (00017-01309PCT00) universal serial busses (USB), or any other similar connectivity port known to those skilled in the art.

[0071] From the foregoing and with reference to the various figures, those skilled in the art will appreciate that certain modifications can be made to the disclosure without departing from the scope of the disclosure.EXAMPLES - the disclosure may be further considered in connection with the following examples in which:Example 1 - A robotic surgical system including a navigation catheter including a first sensor and configured for endoluminal navigation, a robotic arm having a laparoscopic camera and a second sensor, and a computing system including a processor and a memory, the memory storing instructions that when executed by the processor cause the processor to detect a position and orientation of the first sensor as the catheter is navigated into a luminal network towards a target, register the luminal network to a three-dimensional (3D) model of the luminal network and a pathway plan for navigation of the luminal network, detect the second sensor, determine a position and orientation of the laparoscopic camera, determine when the first sensor is within a field of view of the laparoscopic camera, and display a representation of a location of the first sensor on a laparoscopic image captured by the laparoscopic camera.Example 2 - The robotic surgical system of example 1, wherein the navigation catheter is operably connected to a second robotic arm for advancement and articulation.Example 3 - The robotic surgical system of examples 1 or 2, wherein the first sensor and second sensor are electromagnetic (EM) sensors.Example 4 - The robotic surgical system of example 3, wherein one or more of the EM sensors is a tunnel magnetoresistance (TMR) sensor.Example 5 - The robotic surgical system of example 3, further comprising an EM field generator.Example 6 - The robotic surgical system of any of examples 1-5, wherein the instructions when executed by the processor cause the processor to: receive inputs identifying one or more aspects of a laparoscopic surgical procedure including targets, segment or lobe identifications, or clinically relevant data.PATENT APPLICATIONA0013105W001 (00017-01309PCT00)Example 7 - The robotic surgical system of example 6, wherein the clinically relevant data includes one or more of a resection line, suture line, blood vessel, margin, fissure, a no-go zone, or critical structure.Example 8 - The robotic surgical system of example 7, wherein the instructions when executed by the processor cause the processor to: deform the 3D model and the locations of the clinically relevant data. Example 9 - The robotic surgical system of example 8, wherein the one or more aspects of the laparoscopic procedure and clinically relevant data are depicted on the laparoscopic image.Example 10 - The robotic surgical system of example 7, wherein the instructions when executed by the processor cause the processor to: depict a representation of the laparoscopic camera at the determined location and orientation in the deformed 3D model.Example 11 - A method of performing a combined endoluminal and laparoscopic procedure including detecting a position and orientation of a catheter as it is navigated into a luminal network towards a target, registering the luminal network with a three-dimensional (3D) model of the luminal network and a pathway plan for navigation of the luminal network, determining that a sensor on the catheter is in proximity to the target, detecting a position and orientation of a laparoscopic camera, determining when the sensor is within a field of view of the laparoscopic camera, and displaying a representation of a location of the sensor on a laparoscopic image captured by the laparoscopic camera.Example 12 - The method of example 11, wherein one or more of the laparoscopic camera and catheter are robotically driven.Example 13 - The method of examples 11 or 12, further comprising detecting a position and orientation of the catheter or the laparoscopic camera within an electromagnetic field.Example 14 - The method of any of examples 11-13, further comprising receiving imaging of the luminal network and generating the 3D model from the imaging.Example 15 - The method of any of examples 14, further comprising detecting a location and orientation of a robotic surgical instrument.Example 16 - The method of example 15, further comprising displaying a representation of the robotic surgical instrument in a user interface displaying the 3D model.PATENT APPLICATIONA0013105W001 (00017-01309PCT00)Example 17 - The method of any of examples 11-16, further comprising generating a surgical plan including one or more of a pathway for endoluminal navigation of the catheter to the target, a resection line, a suture line, blood vessels, margin, fissures, a no-go zone, critical structuresExample 18 - The method of example 17, further comprising detecting a change in position of a patient after navigation of the catheter in proximity to the target.Example 19 - The method of example 18, further comprising deforming the 3D model and the surgical plan.Example 20 - The method of example 19, further comprising the deformed surgical plan on the laparoscopic image captured by the laparoscopic camera.

Claims

PATENT APPLICATIONA0013105W001 (00017-01309PCT00)We claim:

1. A robotic surgical system comprising: a navigation catheter including a first sensor and configured for endoluminal navigation; a robotic arm having a laparoscopic camera and a second sensor; and a computing system including a processor and a memory, the memory storing instructions that when executed by the processor cause the processor to: detect a position and orientation of the first sensor as the catheter is navigated into a luminal network towards a target; register the luminal network to a three-dimensional (3D) model of the luminal network and a pathway plan for navigation of the luminal network; detect the second sensor; determine a position and orientation of the laparoscopic camera; determine when the first sensor is within a field of view of the laparoscopic camera; and display a representation of a location of the first sensor on a laparoscopic image captured by the laparoscopic camera.

2. The robotic surgical system of claim 1, wherein the navigation catheter is operably connected to a second robotic arm for advancement and articulation.

3. The robotic surgical system of claims 1 or 2, wherein the first sensor and second sensor are electromagnetic (EM) sensors.

4. The robotic surgical system of claim 3, wherein one or more of the EM sensors is a tunnel magnetoresistance (TMR) sensor.

5. The robotic surgical system of claim 3, further comprising an EM field generator.

6. The robotic surgical system of any of claims 1-5, wherein the instructions when executed by the processor cause the processor to: receive inputs identifying one or more aspects of a laparoscopic surgical procedure including targets, segment or lobe identifications, or clinically relevant data.PATENT APPLICATIONA0013105W001 (00017-01309PCT00)7. The robotic surgical system of claim 6, wherein the clinically relevant data includes one or more of a resection line, suture line, blood vessel, margin, fissure, a no-go zone, or critical structure.

8. The robotic surgical system of claim 7, wherein the instructions when executed by the processor cause the processor to: deform the 3D model and the locations of the clinically relevant data.

9. The robotic surgical system of claim 8, wherein the one or more aspects of the laparoscopic procedure and clinically relevant data are depicted on the laparoscopic image.

10. The robotic surgical system of claim 7, wherein the instructions when executed by the processor cause the processor to: depict a representation of the laparoscopic camera at the determined location and orientation in the deformed 3D model.

11. A method of performing a combined endoluminal and laparoscopic procedure comprising: detecting a position and orientation of a catheter as it is navigated into a luminal network towards a target; registering the luminal network with a three-dimensional (3D) model of the luminal network and a pathway plan for navigation of the luminal network; determining that a sensor on the catheter is in proximity to the target; detecting a position and orientation of a laparoscopic camera; determining when the sensor is within a field of view of the laparoscopic camera; and displaying a representation of a location of the sensor on a laparoscopic image captured by the laparoscopic camera.

12. The method of claim 11, wherein one or more of the laparoscopic camera and catheter are robotically driven.PATENT APPLICATIONA0013105W001 (00017-01309PCT00)13. The method of claims 11 or 12, further comprising detecting a position and orientation of the catheter or the laparoscopic camera within an electromagnetic field.

14. The method of any of claims 11-13, further comprising receiving imaging of the luminal network and generating the 3D model from the imaging.

15. The method of any of claims 14, further comprising detecting a location and orientation of a robotic surgical instrument.

Images