Visualized views for lung biopsy and therapeutic procedures

Abstract

A system and method for planning lung navigation paths, the system comprising: a computing device including a processor and a memory storing an application that, when executed by the processor, accesses image data stored in the memory, analyzes the image data to identify the pleura, tumor, and airway, determines a path through the airway to the tumor, generates a 3D model of the lung including representations of the pleura, airway, tumor, and the path through the airway to the tumor, wherein the identification of the pleura, tumor, and airway, and the determination of the path, are performed by one or more neural network algorithms associated with the application, and the 3D model and path are presented in a user interface on a display communicating with the computing device.

Description

Background Technology Technical Field

[0001] This disclosure pertains to the field of intraluminal navigation, specifically to the visualization of navigation pathways and targets within a patient's lungs. Related technical specifications

[0002] Several commonly used medical methods, such as endoscopic or minimally invasive surgery, are employed to treat a variety of diseases affecting organs including the liver, brain, heart, lungs, gallbladder, kidneys, and bones. Clinicians typically use one or more imaging modalities, such as magnetic resonance imaging (MRI), positron emission tomography (PET), ultrasound imaging (PET), computed tomography (CT), cone-beam computed tomography (CBCT), or fluoroscopy, to identify and navigate to regions of interest within the patient and ultimately to biopsy or therapeutic targets. In some procedures, preoperative scans can be used for target identification and intraoperative guidance. Real-time imaging can also be used to obtain more accurate and current images of the target area. Furthermore, real-time image data displaying the current position of a medical device relative to the target and its surrounding environment can be used to safely and accurately navigate the device to the target, e.g., without causing damage to other organs or tissues.

[0003] For example, endoscopic methods can be useful when navigating to regions of interest within a patient's body, particularly within the body's luminal network, such as the lungs, blood vessels, colorectal lumen, and renal ducts. To enable endoscopic methods, navigation systems can use previously acquired MRI or CT image data to generate three-dimensional (3D) renderings, models, or volumes of specific body parts.

[0004] Navigation planning can be created using volumes generated from MRI or CT scans to facilitate the advancement of a navigation catheter (or other suitable medical device) through a bronchoscope and the patient's bronchial branches to the region of interest. Positioning or tracking systems, such as electromagnetic (EM) tracking systems, can be combined with, for example, CT data to facilitate the guidance of the navigation catheter through bronchial branches to the region of interest. In some cases, the navigation catheter can be positioned within a branch network of luminal spaces, adjacent to the region of interest, or within an airway located within the region of interest to provide access to one or more medical devices.

[0005] As is understood, accurate placement of catheters and medical devices is crucial for ensuring successful treatment. Improvements to current navigation catheter systems are desired. Summary of the Invention

[0006] A system having one or more computers can be configured to perform specific operations or actions by installing software, firmware, hardware, or combinations thereof on the system, which, when operated, cause the system to perform these actions. One or more computer programs can be configured to perform specific operations or actions by including instructions that, when executed by a data processing device, cause that device to perform these actions. One general aspect of this disclosure includes a system for planning lung navigation paths, a computing device including a processor and memory, the memory storing an application program that, when executed by the processor,: accesses image data stored in the memory; analyzes the image data to identify the pleura, tumor, and airway, and determines a path through the airway to the tumor; generates a 3D model of the lung, the 3D model including representations of the pleura, the airway, the tumor, and the path through the airway to the tumor, wherein the identification of the pleura, tumor, and airway, and the determination of the path, are performed by one or more neural network algorithms associated with the application program; and presents the 3D model and the path in a user interface on a display communicating with the computing device. Other embodiments of the aspects include corresponding computer systems, apparatuses, and computer programs, each configured to perform the actions of these methods, recorded on one or more computer storage devices.

[0007] The implementation may include one or more of the following features. The system, wherein, when executed by the processor, the application identifies the geometric center of the tumor as a first biopsy location and displays an indicator of the first biopsy location in the tumor in the user interface, wherein the path is [not specified]. When executed by the processor, the application receives a second biopsy location, determines a path through the airway to the tumor to reach the second biopsy location, and displays an indicator of the second biopsy location and the path in the user interface. The application determines multiple paths to each biopsy location and presents a score for each path in the user interface. The level of detail of the 3D model is adjusted based on the zoom level. When executed by the processor, the application displays the determined paths on a live intraluminal image. When executed by the processor, the application generates a timeline view of the lung navigation path. The timeline view includes a representation of each bifurcation of the airway along the path. The timeline view depicts an indicator of the position of the catheter being navigated along the path. The timeline view depicts a representation of the tumor and the biopsy location within the tumor. A portion of the timeline changes color as navigation proceeds. The system may include a timer indicating the duration of the procedure. Implementations of the described technology may include hardware, methods or processes, or computer software on a computer-accessible medium.

[0008] Another aspect of this disclosure includes a system comprising: a catheter configured for navigation within a patient's lungs; a tracking system configured to detect the position of the catheter; and a computing device including a processor and memory, the memory storing an application program that, when executed by the processor,: accesses a set of images stored in the memory; analyzes the image data to identify the pleura, tumor, and airway, and determines a path through the airway to the tumor; generates a 3D model of the lung, the 3D model including a representation of the pleura, a representation of the airway, a representation of the tumor, and a representation of the path through the airway to the tumor; and presents the 3D model and the path in a user interface on a display communicating with the computing device, wherein the identification of the pleura, the identification of the tumor, the identification of the airway, and the determination of the path are performed by one or more neural network algorithms associated with the application program. Other embodiments of this aspect include corresponding computer systems, apparatuses, and computer programs, each configured to perform the actions of these methods, on one or more computer storage devices.

[0009] Implementations may include one or more of the following features. The system, when executed by the processor, depicts a departure angle cone on the tumor. The application, when executed by the processor, displays or removes elements of the 3D model based on zoom levels. The application, when executed by the processor, displays the path on a live bronchoscopic image in the user interface. The application, when executed by the processor, generates a target-aligned view of the 3D model and the live fluorescence fluoroscopy image and presents it in the user interface. The application, when executed by the processor, generates a timeline view of the path, wherein the timeline view includes a representation of each bifurcation of the airway along the path. The application, when executed by the processor, displays tumor size data on the user interface. Implementations of the described technology may include hardware, methods or processes, or computer software on a computer-accessible medium. Attached Figure Description

[0010] Various aspects and embodiments of this disclosure are described below with reference to the accompanying drawings, in which:

[0011] Figure 1A This is a schematic diagram of a robotic surgical system based on the contents of this disclosure.

[0012] Figure 1B This is a schematic diagram of a lumen network navigation system based on the contents of this disclosure;

[0013] Figure 2 It is a 3D model generated based on the contents of this disclosure;

[0014] Figure 3 This is a user interface for planning intraluminal navigation, based on the contents of this disclosure;

[0015] Figure 4 This is a user interface for planning intraluminal navigation to a selected tumor, as disclosed herein;

[0016] Figure 5 It is a user interface that describes the planned path through the 3D model to the tumor, as depicted in this disclosure;

[0017] Figure 6A The user interface shown in this disclosure illustrates the planned path magnified to the area surrounding the tumor.

[0018] Figure 6B Pop-up features of the user interface according to this disclosure are described;

[0019] Figure 7 This is the user interface for intraluminal navigation based on the contents of this disclosure;

[0020] Figure 8 It is a user interface for intraluminal navigation based on the instructions and registration suggestions in this disclosure;

[0021] Figure 9 The merger described in accordance with this disclosure is Figure 7 and Figure 8 The timeline feature on the user interface;

[0022] Figure 10 This refers to the user interface for target alignment based on the contents of this disclosure; and

[0023] Figure 11 This is a schematic depiction of a system for generating and displaying a user interface and implementing intraluminal navigation, based on the present disclosure. Detailed Implementation

[0024] Catheters and catheter-like devices, such as endoscopes, biopsy tools, and therapeutic instruments, are used in a wide range of endovascular medical procedures. These flexible devices are typically used to navigate through the human body's network of lumens, including the vascular system, airways, and digestive system. According to one aspect of this disclosure, neural network algorithms can be used to automatically analyze preoperative or intraoperative images (e.g., CT or PET, MRI or CBCT images) to develop a deep airway tree (a 3D model of the airways). The neural network algorithm performs pixel-by-pixel segmentation, which identifies and distinguishes fluids, air, intraluminal tissues (e.g., airway tissues), bones, and other elements. A deep airway tree refers to a 3D model of the network of lumens (e.g., airways) that is close to the pleural boundary, which contains essentially the entire network of lumens. Many 3D modeling applications that do not employ neural networks tend to output shallower 3D models (e.g., modeling only the larger airways), which can result in poor or incomplete 3D models.

[0025] Another aspect of this disclosure relates to the use of neural network algorithms in identifying tumors or lesions within preoperative or intraoperative images. The neural network algorithms are specifically trained to identify tumors and lesions within preoperative or intraoperative images. The neural network analyzes the preoperative or intraoperative images pixel-by-pixel based on the value of each pixel (e.g., Henle units) to separate tumors and lesions from other tissues or air. Multiple groups of pixels with common or substantially common values ​​are grouped and compared with expected structures (e.g., trachea and carina) and shapes (e.g., tubes of different diameters) within the preoperative or intraoperative images to distinguish air within the airway, the airway itself, healthy tissue, and diseased tissue (e.g., tumors or lesions).

[0026] Another aspect of this disclosure relates to neural network algorithms that generate paths within a lumen network for navigation from an identified tumor or lesion back to a starting location (e.g., the trachea in the lungs). To determine these paths, the neural network can consider various factors, including the proximity of the lumen structure to the tumor or lesion, the angle at which an instrument (e.g., a biopsy tool) leaves the lumen network to reach the target, the radius of the lumen network (e.g., the increasing radius of the path from the target to the starting point), tissue density, and the mechanical properties of the biopsy and treatment tools.

[0027] Another feature of this disclosure involves a user interface that displays zoom-based filtering. Zoom-based filtering changes the level of detail in the displayed 3D model based on where navigation has progressed within the luminal network. When appropriate, the entire 3D model and one or more targets can be displayed on the user interface. As planning or navigation continues, portions of the 3D model are displayed in greater detail and may include depictions of portions of the navigated path, more detail of the targets, and other physical structures of the patient near the targets (e.g., the pleura or vessels). Once near a target, additional detail is depicted in the user interface, including proximity to pleural boundaries, vessels, additional detail of the relative position of the path for catheters or other tools relative to the target, and other details. Changing the detail of the 3D model displayed on the user interface reduces cognitive load, allowing the user to focus only on essential information based on the location of the catheter within the luminal network. These and other aspects of this disclosure are described in more detail below.

[0028] Another aspect of this disclosure relates to user interfaces that enhance visualization features and facilitate efficient surgery. In a first feature of this aspect, a path generated by a neural network algorithm is displayed on a live image of the lumen (e.g., endoscopic or bronchoscopic) acquired during navigation of a catheter or other tool. A second feature of the user interface is a schematic navigation view. As will be appreciated, clinicians may occasionally lose their way in lumen navigation, and assessing the exact location of the catheter within the lumen network is not always readily apparent. The schematic navigation view provides a timeline view that depicts the navigation path to the target (e.g., a tumor or lesion) and the progress of navigation along the path, among other features.

[0029] Another aspect of this disclosure is the incorporation of the above aspects into a robotic surgical system in which a catheter is driven by a robot to one or more targets within the patient.

[0030] Figure 1AA robotic surgical system 10 is depicted, including a control tower 20 connected to all components of the robotic surgical system 10, including a surgeon's console 30 and one or more trolleys 60. Each trolley 60 includes a robotic arm 40 removably coupled to surgical instruments 50. The robotic arm 40 is also coupled to the trolley 60. The robotic surgical system 10 may include any number of trolleys 60 and / or any number of robotic arms 40.

[0031] Surgical instrument 50 is configured for use during minimally invasive surgery (e.g., laparoscopic) or for use in catheter-based luminal procedures (combined with...) Figure 1B (Described in more detail). Surgical instrument 50 may include end effectors 49, such as, without departing from the scope of this disclosure, electrosurgical forceps configured to seal tissue by pressing tissue between jaw members and applying an electrosurgical current thereto; surgical staplers configured to grasp and clamp a pair of jaws of tissue while deploying multiple tissue fasteners (e.g., staples) and cutting the stapled tissue; surgical clamp applicators or other end effectors configured to apply surgical clamps to tissue. Alternatively, surgical instrument 50 may include a robot-actuated catheter including an articulation mechanism (e.g., one or more traction sutures or traction tendons) that effectively alters the shape of the catheter. Furthermore, a robotic arm 40 may be used to advance or retract the catheter during laparoscopic or endoluminal (or combined) procedures.

[0032] One of the robotic arms 40 may include a laparoscopic camera 51 configured to capture video of the surgical site. The laparoscopic camera 51 may be a stereoscopic endoscope configured to capture two side-by-side (i.e., left and right) images of the surgical site to generate a video stream of the surgical scene. The laparoscopic camera 51 is coupled to an image processing device, which may be located within the control tower 20. The image processing device may be any computing device configured to receive video feeds from the laparoscopic camera 51 and output a processed video stream.

[0033] The surgeon's console 30 includes a first screen 32 and a second screen 34. The first screen displays a video feed of the surgical site provided by a camera 51 on a surgical instrument 50 mounted on a robotic arm 40, and the second screen displays a user interface for controlling the robotic surgical system 10. The first screen 32 and the second screen 34 may be touchscreens that allow for the display of various graphical user inputs.

[0034] The surgical console 30 also includes multiple user interface devices, such as a foot pedal 36 and a pair of hand controllers 38a and 38b, which are used by the user to remotely control the robotic arm 40. The surgical console further includes a handrail 33 for supporting the clinician's arm when operating the hand controllers 38a and 38b.

[0035] The control tower 20 includes a screen 23, which may be a touchscreen and output on a graphical user interface (GUI). The control tower 20 also serves as an interface between the surgeon's console 30 and one or more robotic arms 40. Specifically, the control tower 20 is configured to control the robotic arms 40, for example, to move the robotic arms 40 and corresponding surgical instruments 50 based on a set of programmable instructions and / or input commands from the surgeon's console 30, such that the robotic arms 40 and surgical instruments 50 perform a desired sequence of movements in response to inputs from foot pedals 36 and hand controllers 38a and 38b. Foot pedals 36 can be used to enable and lock hand controllers 38a and 38b, reposition camera movement, and activate / deactivate electrosurgical devices. Specifically, foot pedals 36 can be used to perform a clutch action on the hand controllers 38a and 38b. Engaging the clutch by depressing one of the foot pedals 36 disconnects the hand controllers 38a and / or 38b from the robotic arms 40 and their attached corresponding instruments 50 or cameras 51 (i.e., prevents movement input). This allows the user to reposition the hand controllers 38a and 38b without moving the robotic arm 40, instruments 50, and / or camera 51. This is useful when reaching the control boundaries of the surgical space.

[0036] Each of the control tower 20, the surgeon's console 30, and the robotic arm 40 includes a corresponding computer 21, 31, or 41. Computers 21, 31, and 41 are interconnected using any suitable communication network based on wired or wireless communication protocols. As used herein, the term "network," whether plural or singular, means a data network, including but not limited to the Internet, intranet, wide area network, or local area network, and is not limited to the full scope of the definition of communication networks covered by this disclosure. Suitable protocols include, but are not limited to, Transmission Control Protocol / Internet Protocol (TCP / IP), Datagram Protocol / Internet Protocol (UDP / IP), and / or Datagram Congestion Control Protocol (DC). Wireless communication can be achieved via one or more wireless configurations, such as radio frequency, optical, Wi-Fi, Bluetooth (an open wireless protocol for exchanging data from fixed and mobile devices over short distances using short-length radio waves to create personal area networks (PANs), and ZigBee® (a specification for a set of advanced communication protocols using small, low-power digital radios based on the IEEE 122.15.4-1203 standard for wireless personal area networks (WPANs)).

[0037] Computers 21, 31, and 41 may include any suitable processor (not shown) operatively connected to a memory (not shown) that may include one or more of volatile, non-volatile, magnetic, optical, or electrical media, such as read-only memory (ROM), random access memory (RAM), electrically erasable programmable ROM (EEPROM), non-volatile RAM (NVRAM), or flash memory. The processor may be any suitable processor (e.g., control circuitry) adapted to perform the operations, calculations, and / or instruction sets described in this disclosure, including but not limited to hardware processors, field-programmable gate arrays (FPGAs), digital signal processors (DSPs), central processing units (CPUs), microprocessors, and combinations thereof. Those skilled in the art will understand that a processor may be used instead of a logic processor (e.g., control circuitry) adapted to perform the algorithms, calculations, and / or instruction sets described herein.

[0038] Figure 1B This is a three-dimensional diagram of an exemplary system for facilitating intraluminal navigation of a medical device (e.g., a catheter) to a soft tissue target via airways through the lungs. System 100 can be further configured to construct fluorescence-based three-dimensional volumetric data of the target region from 2D fluorescence fluoroscopic images to confirm navigation to the desired location. Other intraoperative imaging modalities, including CBCT, ultrasound, laparoscopic cameras, etc., can also be employed. System 100 can be further configured to facilitate access of the medical device to the target region and to determine the position of the medical device relative to the target, for example, using electromagnetic navigation (EMN). The EMN system can employ a variety of sensor technologies, including, but not limited to, hollow coil sensors, tunneling magnetoresistive (TMR), etc., without departing from the scope of this disclosure. Although described in conjunction with EMN, other systems for luminal and lung navigation are considered within the scope of this disclosure, including shape sensing technologies (e.g., fiber Bragg gratings) that detect the shape of the distal portion of the catheter and match that shape to the shape of the luminal network in a 3D model.

[0039] exist Figure 1BIn this embodiment, catheter 102 is part of catheter guiding assembly 106. In one embodiment, catheter 102 is inserted into bronchoscope 108 to access the luminal network of patient P. Specifically, catheter 102 of catheter guiding assembly 106 can be inserted into the working channel of bronchoscope 108 to navigate through the patient's luminal network. Catheter 102 itself may include imaging capabilities and is not strictly required to be in bronchoscope 108. Positionable guide (LG) 110 (second catheter) including sensor 104 can be inserted into catheter 102 and locked in place such that sensor 104 extends beyond the distal tip of catheter 102 to a desired distance. The position and orientation of sensor 104 relative to a reference coordinate system can be determined, and thus the position and orientation of the distal portion of catheter 102 in an electromagnetic field can be determined. Catheter guiding assembly 106 is currently marketed and sold by Medtronic Corporation under the trade names SUPERDIMENSION® Surgical Kit or EDGE™ Surgical Kit and is believed to be usable in conjunction with this disclosure. Alternatively or concurrently, the conduit 102 may be part of a robotic system, a robot-assisted system, or a robot, and is configured to perform one or more of the methods described herein, as well as to execute algorithms and applications.

[0040] System 100 generally includes: an operating table 112 configured to support a patient P; a bronchoscope 108 configured for insertion into the airway of patient P through the patient P's mouth; monitoring equipment 114 (e.g., a video display for displaying video images received from a video imaging system of the bronchoscope 108 or catheter 102) coupled to the bronchoscope 108 or catheter 102; a positioning or tracking system 115 including a positioning module 116, multiple reference sensors 118, and a launch pad 120 including multiple coupled markers; and a computing device 122 including software and / or hardware for facilitating target identification, path planning to the target, navigation of medical devices to the target, and / or confirmation and / or determination of the placement of the catheter 102 or appropriate devices passing through it relative to the target.

[0041] According to this disclosure, visualization of in vivo navigation of medical devices (e.g., catheters, LG, biopsy, or therapeutic tools) toward a tumor or lesion can be part of a larger workflow of the navigation system. This particular aspect of system 100 also includes an imaging device 124 capable of acquiring images or videos of patient P. Images, image sequences, or videos captured by imaging device 124 can be stored within imaging device 124 or transmitted to computing device 122 for storage, processing, and display. Additionally, imaging device 124 can be moved relative to patient P, allowing images to be acquired from different angles or perspectives relative to patient P to create image sequences such as fluorescence fluoroscopy videos. The orientation of imaging device 124 relative to patient P during image acquisition can be estimated via markers attached to emission pad 120. The markers are positioned below patient P, between patient P and operating table 112, and between patient P and the radiation source or sensing unit of imaging device 124. The markers attached to emission pad 120 can be two separate elements that can be fixedly coupled together, or alternatively, can be manufactured as a single unit. Imaging device 124 may include a single imaging device or more than one imaging device. Imaging device 124 may be, for example, a fluorescence fluoroscopy imaging device, an ultrasound imaging device, or an intraoperative CBCT imaging device.

[0042] The computing device 122 can be any suitable computing device including a processor and a storage medium, wherein the processor is capable of executing instructions stored on the storage medium. The computing device 122 may further include a database configured to store patient data, image datasets (including CT images, CBCT images, fluoroscopic images and videos, fluoroscopic 3D reconstructions, navigation plans), and other such image data. Although not explicitly shown, the computing device 122 may include inputs, or may be configured to receive image datasets and other data described herein. Furthermore, the computing device 122 includes a display configured to display images, 3D models, and other data in one or more graphical user interfaces. The computing device 122 may be connected to one or more networks through which one or more databases (e.g., image databases) can be accessed.

[0043] For use during the navigation phase, registration of images and a path for navigation is performed using a six-degree-of-freedom electromagnetic positioning or tracking system 114 or other suitable system for determining the position and orientation of the distal portion of the catheter 102. The tracking system 114 includes a positioning module 116, multiple reference sensors 118, and a transmitter pad 120 (including markers). The tracking system 114 is configured for use with a positionable guide 110, and in particular, sensors 104. As described above, the positionable guide 110 and sensors 104 are configured for insertion through the catheter 102 into the airway of patient P (with or without a bronchoscope 108) and can be selectively locked relative to each other via a locking mechanism.

[0044] The emitter pad 120 is positioned below the patient P. The emitter pad 120 generates an electromagnetic field around at least a portion of the patient P, within which the tracking module 116 can be used to determine the positions of a plurality of reference sensors 118 and sensors 104. A second electromagnetic sensor 126 may also be incorporated into the end of the catheter 102. The second electromagnetic sensor 126 may be a five-degree-of-freedom sensor or a six-degree-of-freedom sensor. One or more of the reference sensors 118 are attached to the chest of the patient P. Registration is typically performed to coordinate the positions from the three-dimensional model and two-dimensional images from the planning phase with the airway of the patient P as observed through the bronchoscope 108, and to allow for the navigation phase with the positions of the sensors 104 known.

[0045] According to another aspect of this disclosure, catheter 102 can be navigated through the lumen network of patient P relying on sensor 126 without employing a positionable guide 110. In various aspects, the use of [the following is unclear and likely incomplete: "from combined"] Figure 1A The described robotic surgical system 10 functions to navigate catheter 102. Furthermore, catheter 102 may include an intraluminal camera. As catheter 102 advances toward a target, the intraluminal camera captures images of the intraluminal path. The intraluminal camera may be a permanent feature of catheter 102 or a removable camera, similar to the one that guides 110 into the working channel of catheter 102. One or more light tubes may be employed to deliver light from a light source held outside the patient to the end of catheter 102, and to capture light reflected by the patient's tissue and deliver the reflected light to an image-forming component also located outside the patient.

[0046] Registration of the patient P's position on the launch pad 120 can be performed by moving the sensor 104 through the patient P's airway. More specifically, as the positionable guide 110 moves through the airway, data regarding the position of the sensor 104 is recorded using the launch pad 120, reference sensor 118, and tracking system 114. The shape generated from this position data is compared with the internal geometry of the passage in a 3D model generated during the planning phase, and the positional correlation between the shape and the 3D model based on this comparison is determined, for example, using software on computing device 122. Additionally, the software identifies non-organic spaces (e.g., air-filled cavities) in the 3D model. The software aligns or registers an image representing the position of the sensor 104 with the 3D model and / or 2D images generated from the 3D model, based on the recorded position data and the assumption that the positionable guide 110 remains within non-organic spaces in the patient P's airway. Alternatively, a manual registration technique can be employed by navigating the bronchoscope 108 and sensor 104 to a pre-designated location in the lungs of patient P and manually correlating the images from the bronchoscope with model data from the 3D model.

[0047] Although this document describes an EMN system using EM sensors, this disclosure is not limited thereto and may be used in conjunction with flexible sensors, ultrasonic sensors, or without sensors. Shape sensing techniques (e.g., fiber Bragg gratings) can also be used for lumen and lung navigation, detecting the shape of the distal portion of the catheter and matching that shape to the shape of the lumen network in a 3D model.

[0048] Furthermore, the method described herein can be used in conjunction with a robotic system so that a robotic actuator can drive the catheter 102 or bronchoscope 108 to approach the target.

[0049] One aspect of system 100 is a software component that is stored on or accessible from computing device 122 (e.g., the cloud) and configured to process computational image data. While many aspects of image data processing have previously been performed at least in part manually, aspects of this disclosure relate to automated image processing techniques and systems (e.g., using neural network algorithms). These aspects of this disclosure are described in more detail below in conjunction with target recognition and path planning. Some of the aspects described herein, particularly features displayed on the user interface, may be presented during the planning phase or during the navigation phase of the procedure, where medical devices, such as biopsy or therapeutic instruments, may be inserted into catheter 102 to obtain tissue samples from a target or to treat a target.

[0050] According to this disclosure, a 3D model of a network of luminal structures (e.g., a patient's lungs) or another suitable portion of an anatomy can be generated from previously acquired scans (such as CT, CBCT, PET, or MRI scans). Tumors and lesions within the scan data are detected, and pathways through the 3D model to reach the tumor or lesion are generated.

[0051] Once the pathway plan is generated and accepted by the clinician, it can be used by a navigation system to drive a catheter or similar device along the planned path through anatomical structures, particularly through networks of lumens (e.g., airways), to the tumor or lesion. Driving the catheter along the planned path can be manual, robotic, or a combination of both. In a single surgical plan, the pathway plan is registered with the patient and navigated to enable the medical device (e.g., a catheter) to be navigated along the planned route to the target or lesion, thereby enabling the biopsy or treatment of the target.

[0052] As described above, in conjunction with aspects of this disclosure, computing device 122 employs neural network algorithms to automatically analyze image data (e.g., CT, CBCT, PET, or MRI image data). This image data can be preoperative image data (e.g., acquired days or weeks prior to endovascular navigation surgery) or intraoperative image data (e.g., 3D fluorescence fluoroscopy image data or CBCT image data). One or more neural network algorithms analyze this image data for multiple purposes. According to one aspect of this disclosure, a primary purpose of the neural network algorithm is to identify and segment the pleura of the patient's lungs in the image data. As will be appreciated, the location of the pleura is a surgically significant part of the patient's anatomy. Generally, pleural puncture is avoided because it allows air (and blood) to enter the pleural cavity and can lead to pneumothorax or other complications. Furthermore, in instances where pleural puncture is unavoidable, prior knowledge of the pleural location and the probability of puncture during surgery allows clinicians to plan and prepare to mitigate the effects of pleural puncture.

[0053] Neural network algorithms are configured to analyze image data pixel by pixel. This can be based on various factors, including the density of tissue in each pixel of the image data. The harder or denser the tissue, the brighter the pixels of that tissue will appear in the image. For example, very dense bone tissue will appear brighter (e.g., white) in image data than any other tissue, with a higher Heinz value. In contrast, very infrequently dense air typically appears very dark (e.g., black) in image data. Soft tissues have varying density levels and therefore exhibit varying brightness levels, ranging from dense cartilage tissue (e.g., the trachea and central airways) to less dense and elastic tissue (e.g., alveoli), and are depicted in images with a range of brightness. Using segmentation and thresholding techniques, neural network algorithms can define the pixels of the pleura as the outermost part of the lung tissue and separate them from the pixels of the muscles and bones that make up the rib cage.

[0054] The second objective of the neural network algorithm is to identify tumors or lesions within image data. Similar to the pleural boundary, segmentation techniques are employed to define pixel groups that define tumors or lesions. Tumors or lesions will have a higher density than the healthy soft tissue of the lung and will have a shape not associated with the harder tissue of the lung. The harder tissue within the lung is typically a result of cartilaginous tissue associated with the airways themselves. By distinguishing tissue pixel groups based on both density and shape, false positives for tumors or lesions can be avoided.

[0055] A third objective of neural network algorithms is to generate 3D models of the airways and, in some instances, the pulmonary vessels. Airways and vessels have relatively uniform or at least within a certain range of density. Neural network algorithms can detect these tissues and group tissues of similar density to form essentially tubular shapes. These tubular shapes are typically connected to a common source (e.g., trachea, pulmonary artery, pulmonary vein). 3D models of the airways and vessels can be generated by grouping the pixels together in multiple directions (axial, coronal, sagittal) through multiple slices of image data. As described above, by using neural network algorithms and their ability to detect minute differences in pixel brightness, 3D models of the pleura, extending deep into the airways and close to the lungs, can be generated.

[0056] The fourth objective of the neural network algorithm is to identify a path through the 3D model to the target, allowing catheter 102 to be navigated through the airway to reach the target (e.g., a tumor or lesion). This path begins at the target and is defined in the direction of the trachea. In one example, the target center (the center of the tumor) is used as the starting point of the path. The neural network searches for the airway closest to the target center in the 3D model. Proximity to the airway is not the only consideration. Factors considered by the neural network in determining the path include the angle at which an tool (e.g., a biopsy tool) leaves the luminal network to reach the target. As will be appreciated, it is difficult (if not impossible) for most intraluminal tools to leave the airway at an angle close to a right angle to the longitudinal direction of the airway. Most intraluminal tools are straight and somewhat rigid at their distal portions, and therefore cannot be manipulated to make such sharp turns to leave the patient's narrow airway. Therefore, the departure angle of catheter 102 or the tool to the tumor center must be considered when determining the nearest airway. Another factor as part of path generation is the radius of the airway. For example, determining whether the airway closest to the target is large enough to receive catheter 102. If not, a more distant airway might be a better option for catheter navigation, or it could be possible to identify another location where the catheter will have to leave the airway to reach the target—a location where the angle with the center of the target allows the biopsy or treatment tool to move essentially straight. In some instances, multiple paths can be generated using neural network algorithms, and suggested paths can be identified and presented to the user. Furthermore, as the catheter returns from the target to the trachea, the neural network algorithm can analyze potential paths to ensure that the airway being traversed has an increasing radius along that path. Other factors considered include tissue density, the mechanical properties of the biopsy and treatment tools, and other factors can also be involved in path generation.

[0057] According to one aspect of this disclosure, an application on computing device 122 can present a user with a large dataset of images associated with different patients, from which the user can select. The user selects an image dataset for the patient from whom a pathway plan needs to be generated. Once selected via the user interface, the application initiates a set of algorithms, including neural network algorithms, to perform the aforementioned aspects, including pleural boundary segmentation, identification of tumors or lesions (targets) within the image dataset, generation of 3D models of the airway and blood vessels, and generation of a pathway from the trachea to the tumor or lesion. After these steps, a 3D model 202 is displayed in user interface 204, as shown in [the original text]. Figure 2As shown in the diagram, the 3D model 202 includes an airway tree 206, blood vessels 208, pleural boundaries 210, and tumors or lesions 212. Therefore, the 3D model 202 provides a significantly more accurate model of the patient's anatomy because the neural network algorithm is able to generate a 3D model of the airway much deeper than previous systems. Furthermore, the relative positions of the airway 206, blood vessels 208, pleural boundaries 210, and tumors or lesions 212 are much more accurate than previous 3D models used in intraluminal navigation, particularly lung navigation. Surgical planning can occur on any computing device capable of receiving and displaying image data and running a planning application, including computing device 122 or a computing device completely separate from system 100.

[0058] Figure 3 A first planning screen 214 on UI 204 is depicted. Planning screen 214 identifies and labels individual tumors 212. Table 216 is displayed, which includes information about each identified tumor 212, such as its location (e.g., upper right lobe, upper left lobe, etc.), a thumbnail image 218 of the tumor shape, and the tumor size 220 (e.g., its longest dimension). Button 222 allows manipulation of the 3D model 202, including, for example, centering the field of view of the 3D model, zooming in or out on the 3D model, measurement tools, increasing or decreasing the contrast of parts of the 3D model, and rotating the 3D model. Furthermore, image data of the 3D model 202 generated therefrom can be selected via tab 224 (e.g., CT or PET).

[0059] Figure 4 UI 204 is depicted after receiving a selection of tumor 212 (e.g., tumor 1). The selection can be made on 3D model 202 or in table 216. Information about the tumor 212 is highlighted in table 216, and image data of the 3D model generated from it is displayed in side panel 226. Side panel 226 includes images from three perspectives: axial image 228, coronal image 230, and sagittal image 232, each including a scroll bar 234 that allows the image to change according to scrolling (e.g., the next image along that axis). The three scroll bars 234 can be interconnected, such that movement of one causes all three to scroll, or they can be selectively scrolled individually. These images depict the location of tumor 212 identified by the (described above) neural network algorithm found in 3D model 202.

[0060] Upon receiving a selection of the Continue Planning button 236, the user interface 204 proceeds to... Figure 5 The 3D model depicts the planned pathway 238 within the airway 206 to reach the tumor or lesion 212. (As in...) Figure 5As shown, two paths 238 leading to tumor 212 are depicted. Recommended paths (e.g., the paths with the highest scores in Table 216) are presented in a unique style (e.g., highlighted) relative to the other paths. In the planning tab of Table 216, each path 238 is assigned a score by a neural network algorithm, taking into account factors such as the departure angle of the tool (e.g., a biopsy tool) from catheter 102 to the biopsy site 240 within tumor 212. As will be appreciated, the smaller the angle, the more direct the path from the opening at the distal end of catheter 102 to the biopsy site 240. A straighter path from catheter 102 to biopsy site 240 is easier to achieve because many biopsy and treatment tools have rigid end portions. While articulating catheter 102 can point its opening toward target 212, the rigidity of the tool passing through catheter 102 will generally cause the tool and catheter 102 to straighten as the tool leaves catheter 102. Another factor is the distance the tool must travel from the airway (e.g., through the airway walls and parenchyma) to reach the biopsy site 240. A shorter travel distance from the airway to the tumor 212 results in less interaction with intermediate tissues, which may include, for example, blood vessels. Accordingly, a shorter travel distance from the airway 206 is generally (but not always) preferred.

[0061] exist Figure 5 In this context, biopsy site 240 is the geometric center of tumor 212. This can be the default location for a path plan suggested by a neural network algorithm. Additional biopsy sites 240 can be added to tumor 212 using various pointing tools (e.g., mouse, touchscreen, etc.). By adding (or moving) biopsy sites 240, additional or modified paths 238 can be generated by the application's neural network algorithm. Each biopsy site 240 can receive an identifier (e.g., a number), and each biopsy site 240 is listed separately under the heading "Biopsy" in Table 216 and can have its own path 238. Although not explicitly stated in... Figure 5 As explicitly shown, however, neural network algorithms can also determine multiple distinct biopsy sites within tumor 212 and the path to each biopsy site 240. The locations of these biopsy sites 240 can be based on historical patterns used for effective biopsies (or treatments) when considering information about the size, shape, location, and other factors of tumor 212. Alternatively, the distribution of biopsy sites 240 determined by the neural network can be based on empirical algorithms derived from data collected from multiple previous biopsies and treatments, from which the data is preserved and analyzed (e.g., by another neural network algorithm) to identify potentially effective biopsy sites 240 within tumor 212. Furthermore, biopsy sites can be determined based on the tumor's location within the patient, taking into account features such as the pleura, blood vessels, the shape and location of the exit cone, and other factors, to minimize any challenges in collecting biopsies and optimize the chances of obtaining important and useful samples from tumor 212.

[0062] To accurately place each biopsy site 240, zooming in on the tumor 212 using button 222 can produce a display of a 3D model 202 showing the portion of the tumor 212 in the vicinity of the tumor 212, as shown in... Figure 6A As described in [the text]. Figure 6A As can be seen, the initial biopsy location 240 is at the center of tumor 212. A departure angle cone 242 is projected from the end of path 238 and defines the expected potential trajectory of the tool that has passed through this location of catheter 102 within the 3D model 202 (and the patient). This departure angle cone 242 represents the potential volume of tumor 212 from which a biopsy can be collected from the potential volume at this location of catheter 102. Although the departure angle cone 242 includes the biopsy location 240, the exact biopsy location 240 may not be achievable in the patient due to the mechanical properties of catheter 102 and the biopsy or treatment tool. Importantly, in Figure 6A In China (also) Figure 5 (Shown in the image) A representation of a portion of the pleura 210 is shown. As described above, the pleura 210 represents a boundary that is typically (if possible) avoided to prevent pneumothorax. Thus, having identified biopsy sites 240, identified paths 238 to those biopsy sites 240, and shown the departure angle cone 242 relative to the tumor 212 and the pleura 210, a clinician can manually or through a neural network algorithm determine if there is any possibility of potentially damaging the pleura 210. When it is determined that such an event may occur, alternative paths 238 can be selected or identified, or the biopsy sites 240 can be altered to avoid the possibility of puncturing the pleura 210.

[0063] Tumor 212 may have multiple (2, 5, 10, etc.) biopsy sites 240 to ensure sufficient sampling has been achieved. Each biopsy site 240 may have its own path 232 and exit angle cone 242. Once the planning for tumor 212 is satisfactory, the process can be repeated for all tumors 212 in the 3D model 202. After all biopsy sites 240 of all tumors 212 and paths 238 to all biopsy sites 240 have been identified, selection of the examination planning button 244 allows examination of all paths 238 and biopsy sites 240 leading to all tumors 212. If accepted, the plan is stored and can be used immediately or at a later date in surgery. Those skilled in the art will recognize that paths 238 and biopsy sites 240 may be used alternatively or additionally in treatment planning without departing from the scope of this disclosure, and biopsy sites 240 may be therapeutic application sites.

[0064] In one aspect of this disclosure, the examination of the pathway plan provides a virtual bronchoscopic view that simulates navigation of catheter 102 through a network of lumens on UI 204. UI 204 can also depict a 3D model 202, and the scaling level of the 3D model can be changed as the simulated navigation progresses (e.g., Figure 4 (as shown in Figure 6), thus providing clinicians with the necessary information for the examination at the points in the simulated navigation.

[0065] If we get to know each other, in Figure 2 The features of the 3D model 202 depicted in the image are not all present in the subsequent descriptions. Figure 4 As depicted in Figure 6. The reduction in features of the 3D model decreases the cognitive load on clinicians and allows them to focus on relevant aspects of the 3D model and the patient's anatomy. Using button 222, different features among these can be selectively turned on or off by the clinician. For example, upon reaching... Figure 6A As shown in UI 204, a clinician might be interested in the blood vessels 208 surrounding the tumor 212, allowing the path 238 to be adjusted to avoid intersections of the blood vessels 208. Accordingly, relevant blood vessels can be manually or automatically highlighted so that they appear in UI 204. Furthermore, some of these features can be automatically removed or phantomized (e.g., made translucent or transparent) in the planning software to reduce the appearance of intermediate portions of the anatomical structure. By reducing the displayed 3D model to only clinically relevant portions, cognitive load is reduced, and a clearer understanding of the planned surgery is achieved.

[0066] Figure 6B Another aspect of this disclosure is depicted, in which a pop-up window 246 is presented in UI 204. This pop-up window presents clinicians with various options, allowing for the automatic generation of biopsy sites 240 within the tumor 212. A button 248 in the pop-up window allows selection of a central biopsy site, a PET biopsy site, a geometric biopsy site, or a custom biopsy site to be added to the tumor 212.

[0067] While the foregoing aspects of this disclosure focus on pathway planning and biopsy collection (or therapeutic applications), this disclosure is not limited thereto. The following features primarily relate to aspects of UI 204 during endovascular navigation procedures. Figure 7A side-by-side view depicting two visualization features of this disclosure is shown. A live bronchoscopic image 302 is displayed on one side of US 204. This live bronchoscopic image is acquired by a bronchoscope associated with catheter 102 (inserted into its lumen or as a component thereof). Path 238 is overlaid on the live bronchoscopic image 302. This overlay of path 238 provides guidance to clinicians when navigating the patient's airway and provides directional assistance regarding which airway to follow to reach the tumor 212 at the end of path 238, as determined during planning (as described above). Simultaneously with the bronchoscopic image 302, a 3D model 202 is also displayed on the other side. The display of the 3D model is connected to the view in the bronchoscopic image 302. Thus, as navigation of catheter 102 proceeds, the position of catheter 102 along path 238 can be depicted in the 3D model 202. Alternatively or additionally, as catheter 102 is navigated to different parts of the lumen network, the scaling level of 3D model 202 can be adjusted (similar to...). Figure 4 (Up to the level depicted in Figure 6). The 3D model 212 includes at least the airway 206, the tumor 212, and a portion of the navigation-related pleura 210.

[0068] Although described as a completely separate part of the application, the plan can be reset or modified before or as part of the procedure, if necessary or appropriate. Furthermore, the plan can be modified and targets can be added to or removed from the path plan, or new targets can be selected during navigation using methods substantially similar to those described above in this document.

[0069] Figure 8 A pop-up window 303, which may optionally be presented on UI 204, is depicted, alerting clinicians that a local registration process can be performed to reduce or eliminate CT body discrepancies. The local registration process can acquire fluoroscopy or cone-beam computed tomography (CBCT) images. In one local registration process, fluoroscopy images are acquired and the relative positions of duct 102 and the tumor are determined. The determined relative positions are then used to update the relative positions of duct 102 and tumor 212 displayed in the 3D model. Alternatively, if CBCT images are acquired, the CBCT images are processed (e.g., using neural network algorithms) to identify the relative positions of duct 102 and tumor, and the 3D model is updated using the image data from the CBCT. The CBCT image data may not be an image of the entire lung (or another network of lumens), but may be a small portion near duct 102 and tumor 212. A 3D model can be generated from the CBCT image data and parts of the original 3D model can be replaced. Alternatively, biopsy location 240 and path 238 from the original 3D model can be applied to the 3D model from the CBCT data to allow accurate “last mile” navigation to biopsy location 240.

[0070] Another feature of UI 204 is the timeline feature 304 that appears at the bottom of UI 204. A more detailed view of the timeline 304 can be found in... Figure 9 The timeline is a linear representation of path plan 238 (e.g., from the trachea to the tumor) and provides a directional navigation instruction. Timeline 304 provides an abstract representation of path 238. Timeline 304 highlights information needed for navigation, such as bifurcations along the way, and visually shows them in a way that the associated lumens are clearly marked. Furthermore, the timeline of this view represents the distance up to the target and can therefore be used as a simple way to assess progress toward the target. As the catheter 102 is navigated within the network of lumens (e.g., the lungs), indicators 306 are depicted at each bifurcation where a decision must be made about which airway to follow. Each indicator includes the number of airways at the bifurcation, and one of these airways or one side of the bifurcation is highlighted, thus indicating the direction in which navigation should continue. As navigation proceeds along a portion of path 238 and progress of the catheter 102 is detected, the timeline changes color (e.g., from dark blue to light blue). The dark blue of the path 238 already traversed and the gap 308 between it and the tumor 212 represent the distance that the biopsy or treatment tool must travel outside the airway to reach the tumor 212. The tumor 212 is represented by a different color along timeline 304. The biopsy site 240 can be depicted within the tumor 212 portion of timeline 304. Although not in... Figure 8 As shown in the diagram, however, if related to the specific path 238 being navigated, a representation of the pleura 210 can be shown on the timeline 304 to the right of the tumor 212 section. Timer 310 displays the time required to traverse path 238 as part of the surgery. The surgery can be recorded and replayed using button 312 on timeline 304.

[0071] Another feature of this disclosure involves displaying, as in Figure 10 The alignment view 400 is shown on UI 204. Figure 10 In this model, a portion of the live fluorescence fluoroscopy image and the 3D model 202 are depicted on the UI 204. By adjusting the position of the catheter 102 and observing the differences in its position in the fluorescence fluoroscopy image and in the 3D model 202, clinicians can observe the interaction between the biopsy tool and the tumor 212, even if the tumor is not fully visible in the fluorescence fluoroscopy image. The movement shown in the 3D model helps clinicians confirm that the biopsy or treatment tool is aligned with and interacts with the tumor 212 as planned.

[0072] Now for reference Figure 11This figure is a schematic diagram of system 400, configured for use with the features described above. System 400 may include a workstation 401 and optionally an imaging device 415 (e.g., a fluoroscope, CBCT scanner, or ultrasound device). In some embodiments, workstation 401 may be directly or indirectly connected to imaging device 415, for example, via wireless communication. Workstation 401 may include memory 402, processor 404, display 406, and input device 410. Processor 404 may include one or more hardware processors. Workstation 401 may optionally include an output module 412 and a network interface 408. Memory 402 may store application program 418 and image data 414. Application program 418 may include methods executable by processor 404 for performing the methods disclosed herein (e.g., image processing neural network algorithms described herein) and instructions presented on one or more displays (e.g., screens 23, 32, or 34).

[0073] Application 418 may further include user interface 416. Image data 414 may include CT or CBCT scans, 3D models, fluorescence fluoroscopy 3D reconstruction of target areas and / or any other fluorescence fluoroscopy image data, ultrasound image data, magnetic resonance imaging (MRI) data, positron emission tomography (PET) image data, etc. Processor 404 may be coupled to memory 402, display 406, input device 410, output module 412, network interface 408, and imaging device 415. Workstation 401 may be a fixed computing device such as a personal computer, or a portable computing device such as a tablet computer. Workstation 401 may include multiple computer devices. Those skilled in the art will recognize that the methods and systems described herein can be incorporated into and executed on any computing device capable of receiving image data of a lumen network, including computers 21, 31, and 41 of the robotic surgical system 10.

[0074] Memory 402 may include any non-transitory computer-readable storage medium for storing data and / or software including instructions that can be executed by processor 404 and control the operation of workstation 401 and, in some embodiments, the operation of imaging device 415. Memory 402 may include one or more storage devices, such as solid-state storage devices (e.g., flash memory chips). As an alternative to or supplement to one or more solid-state storage devices, memory 402 may include one or more mass storage devices connected to processor 404 via a mass storage controller (not shown) and a communication bus (not shown).

[0075] Although the description of computer-readable media herein refers to solid-state storage devices, those skilled in the art will understand that computer-readable storage media can be any available medium accessible to the processor 404. That is, computer-readable storage media can include non-transitory, volatile and non-volatile, removable and non-removable media implemented using any method or technique for storing information such as computer-readable instructions, data structures, program modules, or other data. For example, computer-readable storage media can include RAM, ROM, EPROM, EEPROM, flash memory or other solid-state memory technologies, CD-ROM, DVD, Blu-ray or other optical storage devices, magnetic tape cassettes, magnetic tape, disk storage devices or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible by workstation 401 (e.g., cloud-based storage devices).

[0076] When executed by processor 404, application 418 can cause display 406 to (e.g., to present user interface 416). Without departing from the scope of this disclosure, user interface 416 may be configured to present to the user different UIs 204 described herein (e.g., on screens 23, 32, or 34) as well as other content.

[0077] Network interface 408 can be configured to connect to a network, such as a local area network (LAN), wide area network (WAN), wireless mobile network, Bluetooth network, and / or the Internet, consisting of wired and / or wireless networks. Network interface 408 can be used to establish a connection between workstation 401 and imaging device 415. Network interface 408 can also be used to receive image data 414. Input device 410 can be any device that a user can use to interact with workstation 401, such as a mouse, keyboard, foot pedal, touchscreen, and / or voice interface. Output module 412 can include any connection port or bus, such as a parallel port, serial port, universal serial bus (USB), or any other similar connection port known to those skilled in the art. Based on the foregoing and with reference to the various accompanying drawings, those skilled in the art will understand that certain modifications can be made to this disclosure without departing from the scope of this disclosure. Example Example 1 - A system for planning lung navigation paths includes: a computing device including a processor and a memory, the memory storing an application that, when executed by the processor, accesses image data stored in the memory, analyzes the image data to identify the pleura, tumor, and airway, determines a path through the airway to the tumor, generates a 3D model of the lung, the 3D model including representations of the pleura, the airway, the tumor, and the path through the airway to the tumor, wherein the identification of the pleura, the tumor, and the airway, and the determination of the path are performed by one or more neural network algorithms associated with the application, and the 3D model and the path are presented in a user interface on a display communicating with the computing device. Example 2 - A system as described in Example 1, wherein the application, when executed by the processor, identifies the geometric center of the tumor as a first biopsy location and displays an indicator of the first biopsy location in the tumor in the user interface, wherein the path. Example 3 - A system as described in Example 2, wherein when the application is executed by the processor, it receives a second biopsy location, determines a path through the airway to the tumor to reach the second biopsy location, and displays an indicator of the second biopsy location and the path in the user interface. Example 4 - A system as described in either Example 2 or 3, wherein the application determines multiple paths to each biopsy location and presents a score for each path in the user interface. Example 5 - A system as described in any of the preceding examples, wherein the level of detail of the 3D model is adjusted based on the scaling level. Example 6 - A system as described in any of the preceding examples, wherein the application displays the determined path on a live intraluminal image when executed by the processor. Example 7 - A system as described in any of the preceding examples, wherein the application generates a timeline view of the lung navigation path when executed by the processor. Example 8 - A system as described in Example 7, wherein the timeline view includes a representation of each bifurcation of the airway along the path. Example 9 - A system as described in either Example 7 or 8, wherein the timeline view depicts an indicator of the location of a duct being navigated along the path. Example 10 - A system as described in any of Examples 7-9, wherein the timeline view depicts a representation of the tumor and biopsy sites within the tumor. Example 11 – A system as described in any of Examples 7-10, wherein a portion of the timeline changes color as navigation proceeds. Example 12 - A system as described in any of Examples 7-11 further includes a timer indicating the duration of the procedure. Example 13 - A system comprising a catheter configured for navigation within a patient's lungs, a tracking system configured for detecting the position of the catheter; and a computing device including a processor and a memory, the memory storing an application that, when executed by the processor,: accesses a set of images stored in the memory, analyzes the image data to identify the pleura, a tumor, and an airway, and determines a path through the airway to the tumor; generates a 3D model of the lung, the 3D model including a representation of the pleura, a representation of the airway, a representation of the tumor, and a representation of the path through the airway to the tumor; and presents the 3D model and the path in a user interface on a display communicating with the computing device, wherein the identification of the pleura, the identification of the tumor, the identification of the airway, and the determination of the path are performed by one or more neural network algorithms associated with the application. Example 14 - A system as described in Example 13, wherein the application, when executed by the processor, identifies the geometric center of the tumor as a first biopsy location and displays an indicator of the first biopsy location in the tumor in the user interface. Example 15 - A system as described in any of Examples 13-14, wherein the application, when executed by the processor, draws a departure angle cone on the tumor. Example 16 - A system as described in any of Examples 13-15, wherein the application, when executed by the processor, displays or removes elements of the 3D model based on a scaling level. Example 17 - A system as described in any of Examples 13-16, wherein the application displays the path on a live bronchoscopic image in the user interface when executed by the processor. Example 18 - A system as described in any of Examples 13-17, wherein the application, when executed by the processor, generates a target-aligned view of the 3D model and a live fluorescence perspective image and presents it in the user interface. Example 19 - A system as described in any of Examples 13-18, wherein the application generates a timeline view of the path when executed by the processor, wherein the timeline view includes a representation of each bifurcation of the airway along the path. Example 20 - A system as described in any of Examples 13-19, wherein the application displays tumor size data on the user interface when executed by the processor.

[0078] While detailed embodiments are disclosed herein, these embodiments are merely examples of the disclosure, which can be embodied in various forms and aspects. Therefore, the specific structural and functional details disclosed herein should not be construed as limiting, but rather as the basis for the claims and as a representative basis for teaching those skilled in the art to employ the disclosure in various ways in virtually any suitably described structure.

Claims

1. A system for planning lung navigation paths, comprising: A computing device including a processor and memory, wherein the memory stores an application program, which, when executed by the processor: Access the image data stored in the memory; The image data is analyzed to identify the pleura, tumor, and airway, and to determine the path through the airway to the tumor. Generate a 3D model of the lung, the 3D model including the pleura, the airway, the tumor, and a representation of the path from the airway to the tumor; as well as The 3D model and the path are presented in a user interface on a display that communicates with the computing device.

2. The system as claimed in claim 1, wherein, When executed by the processor, the application identifies the geometric center of the tumor as the first biopsy location and displays an indicator of the first biopsy location in the tumor in the user interface, wherein the path is...

3. The system as described in claim 2, wherein, When executed by the processor, the application receives a second biopsy location, determines a path through the airway to the tumor to reach the second biopsy location, and displays an indicator of the second biopsy location and the path in the user interface.

4. The system as described in any one of claims 2, wherein, The application identifies multiple paths to each biopsy location and presents a score for each path in the user interface.

5. The system as described in claim 2, wherein, The level of detail in the 3D model is adjusted based on the scaling level.

6. The system as claimed in claim 2, wherein, The identification of the pleura, the tumor, the airway, and the determination of the path are performed by one or more neural network algorithms associated with the application.

7. The system as claimed in claim 2, wherein, When the application is executed by the processor, it generates a timeline view of the lung navigation path.

8. The system of claim 7, wherein, The timeline view includes a representation of each branch of the airway along the path.

9. The system of claim 7, wherein, The timeline view depicts an indicator of the location of the duct being navigated along the path.

10. The system of claim 7, wherein, The timeline view depicts a representation of the tumor and the biopsy sites within the tumor.

11. The system of claim 7, wherein, A portion of the timeline changes color as navigation progresses.

12. The system of claim 7, further comprising a timer indicating the duration of the procedure.

13. A system comprising: A catheter, configured for navigation within a patient's lungs; A tracking system configured to detect the position of the catheter; as well as A computing device including a processor and memory, wherein the memory stores an application program, which, when executed by the processor: Access the image set stored in the memory; The image data is analyzed to identify the pleura, tumor, and airway, and to determine the path through the airway to the tumor. Generate a 3D model of the lung, the 3D model including a representation of the pleura, a representation of the airways, a representation of the tumor, and a representation of the path from the airways to the tumor; and The 3D model and the path are presented in a user interface on a display that communicates with the computing device.

14. The system as described in 13, wherein, When executed by the processor, the application identifies the geometric center of the tumor as the first biopsy location and displays an indicator of the first biopsy location in the tumor in the user interface.

15. The system of claim 13, wherein, When the application is executed by the processor, it draws a departure angle cone on the tumor.

16. The system of claim 13, wherein, When the application is executed by the processor, it displays or removes elements of the 3D model based on the scaling level.

17. The system of claim 13, wherein, When the application is executed by the processor, the path is displayed on the live bronchoscopic image in the user interface.

18. The system of claim 13, wherein, When executed by the processor, the application generates a target-aligned view of the 3D model and the live fluorescence perspective image and presents it in the user interface.

19. The system of claim 13, wherein, The identification of the pleura, the identification of the tumor, the identification of the airway, and the determination of the path are performed by one or more neural network algorithms associated with the application.

20. The system of claim 13, wherein, When the application is executed by the processor, it displays tumor size data on the user interface.

Images