Dynamic intervention 3D model deformation

Abstract

A controller for assisting navigation in an interventional procedure, comprising: a memory storing instructions, and a processor (310) executing the instructions. When executed by the processor (310), the instructions cause the controller to implement a process including the following operations: obtaining (S410) a three-dimensional model generated before the interventional procedure based on a path having multiple branches in an object that segments the interventional procedure. The process also includes determining (S470) during the interventional procedure whether the current position of a tracking device (250) is outside the path in the three-dimensional model. When the current position of the tracking device (250) is outside the path in the three-dimensional model, the process includes deforming (S480) the three-dimensional model to the current position of the tracking device (250).

Description

Background Art

[0001] Optical shape sensing (OSS) can be used to determine the shape of an optical fiber using light along the fiber. The optical fiber can be provided in or on an interventional medical device to determine its shape. Information about the shape of the optical fiber can be used to position and navigate the interventional medical device during surgical intervention. Optical shape sensing is based on the principle that the wavelength of reflected light varies under different circumstances. Therefore, distributed strain measurement in an optical fiber can be used to determine the shape of the optical fiber using characteristic Rayleigh backscattering or controlled grating patterns.

[0002] Examples of interventional medical devices include guidewires, catheters, sheaths, and bronchoscopes, and optical shape sensing can be used to provide real-time position and orientation of such interventional medical devices for guidance during minimally invasive procedures.

[0003] One example of using optical shape sensing with an interventional device is to embed an optical fiber in a guidewire and navigate the guidewire through a channel in the body of the interventional device. FIG1A illustrates a virtual image of an optical fiber 101 embedded in a guidewire 102. The three-dimensional position of the guidewire 102 can then be registered to an anatomical imaging modality, such as X-ray or computed tomography (CT), to provide anatomical context of the guidewire 102 and the shape sensing interventional device (e.g., a bronchoscope) into which the guidewire 102 is inserted or attached.

[0004] Separately from the above, lung lesions can be biopsied using an endobronchial approach in which a bronchoscope is guided down the airway. A camera at the end of the bronchoscope provides an image of the airway, and abnormal portions of the airway can be biopsied using a small tool inserted through the working channel of the bronchoscope. Methods for lung biopsy face challenges, including:

[0005] ○ Bronchoscopes can only be used for large upper airways

[0006] ○ Many lesions are not connected to any airway, large or small, and many others are in or connected to only small airways that cannot be reached by interventional tools.

[0007] ○ Large deformation of the lungs caused by breathing and cardiac motion

[0008] Endobronchial approaches can be limited to lesions in or connected to the upper airway because only the upper airway can fit a bronchoscope. Otherwise, visualization may be lost, resulting in blind navigation of the interventional medicine device inside or outside the airway to take random tissue samples. In addition, when the lesion is not connected to the airway, the interventional medicine device must pierce the airway wall and travel through the lung parenchyma outside the airway to perform a transbronchial biopsy. Transbronchial biopsy can also be performed when the lesion's closest airway is too small to navigate the tool.

[0009] Several approaches have been developed to address these endobronchial challenges and provide physicians with better navigation. In some of these approaches, preoperative three-dimensional images of the lungs are acquired and processed using algorithms to generate a three-dimensional model of the airways and lesions. The physician can then navigate using this three-dimensional model, with or without X-rays and / or a bronchoscope. However, the three-dimensional model is limited to what the airways looked like at the time of preoperative imaging. In addition to imaging, some navigation methods also use tracking technologies, such as electromagnetic navigation, which tracks the three-dimensional position of certain locations of interventional devices. However, with the exception of optical shape sensing, most tracking technologies rely on locating the tip of the interventional medical device, and this is problematic due to respiratory and cardiac motion.

[0010] In addition to the challenges described above regarding lung biopsy, three major challenges are encountered in lung tumor surgery, regardless of the type of lung tumor surgery being performed. The location of the tumor can be initially determined based on a preoperative computed tomography (CT) scan performed with the lung inflated. During surgery, the lung collapses, and therefore the three-dimensional orientation of the lung and the location of the tumor do not match the preoperative images used for planning. Figure 1B illustrates a comparison between an inflated view of the lung and a collapsed view of the lung. This is further complicated by the fact that the lung typically moves and repositions throughout the surgical procedure. This movement can cause the surgeon to lose track of the tumor location and the orientation of the lung lobe relative to the surface of the lung. Second, the lung is very complex, with many blood vessels and airways that must be carefully dissected and processed before the tumor and any feeding airways or blood vessels are removed. Third, small, untouched tumors are very difficult to locate in the lung, particularly using video-assisted thoracoscopic surgery (VATS) or robotic surgery.

[0011] FIG1C illustrates a three-dimensional model of the airway and tumor with a planned pathway to the tumor. FIG1C also shows the planned pathway for the interventional procedure superimposed on a real-time fluoroscopic image of the airway and tumor. The planned pathway 103 from the trachea is shown as a thin line from the top. The current position 106 of the bronchoscope is also seen in the fluoroscopic image to the right of the planned pathway 103. The tumor position 104 is shown at the end of the planned pathway 103. However, the three-dimensional model in FIG1C is static. Therefore, no method for adjusting the three-dimensional model is provided.

[0012] As described herein, dynamic interventional three-dimensional model deformation can be used to enhance positioning and navigation for interventional medical devices. Summary of the Invention

[0013] According to a representative embodiment of the present disclosure, a controller for assisting navigation in an interventional procedure includes a memory storing instructions and a processor executing the instructions. When executed by the processor, the instructions cause the controller to implement a process. The process includes: obtaining a three-dimensional model generated before the interventional procedure based on a path with multiple branches in an object that divides the interventional procedure. The process also includes: determining whether the current position of a tracking device is outside the path in the three-dimensional model during the interventional procedure. When the current position of the tracking device is outside the path in the three-dimensional model, the process includes deforming the three-dimensional model to the current position of the tracking device.

[0014] According to another representative embodiment of the present disclosure, a method for assisting navigation in an interventional procedure includes obtaining a three-dimensional model generated before the interventional procedure based on a path having multiple branches in an object segmenting the interventional procedure. The method also includes determining, during the interventional procedure, via a controller executing instructions using a processor, whether a current position of a tracking device is outside the path in the three-dimensional model. If the current position of the tracking device is outside the path in the three-dimensional model, the method further includes deforming the three-dimensional model to the current position of the tracking device.

[0015] According to another representative embodiment, a system for assisting navigation in an interventional procedure includes an imaging device and a computer. The imaging device generates a computer tomography image of an object of the interventional procedure before the interventional procedure for generating a three-dimensional model based on segmenting a path having multiple branches in the object of the interventional procedure before the interventional procedure. The computer includes: a memory for storing instructions and a processor for executing the instructions. When executed by the processor, the instructions cause the system to execute a process including obtaining a three-dimensional model generated before the interventional procedure. The process executed when the process executes the instructions also includes: determining whether the current position of the tracking device is outside the path in the three-dimensional model. When the current position of the tracking device is outside the path in the three-dimensional model, the process includes deforming the three-dimensional model to the current position of the tracking device. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The exemplary embodiments will be best understood from the following detailed description when read in conjunction with the accompanying drawings. It is emphasized that the various features are not necessarily drawn to scale. In fact, for the clarity of discussion, the dimensions may be arbitrarily increased or decreased. Wherever applicable and practical, like reference numerals refer to like elements.

[0017] FIG. 1A illustrates a virtual image of an optical fiber inserted into a working channel of a bronchoscope.

[0018] FIG. 1B illustrates a comparison between an inflated view of the lung and a collapsed view of the lung.

[0019] Figure 1C illustrates a three-dimensional model of the airway and tumor with a planned approach to the tumor. It also shows an overlay of real-time fluoroscopic images of the anatomy during the interventional procedure, including the position of the interventional device inside the airway.

[0020] Figure 2 A system for dynamically intervening in deformation of a three-dimensional model is illustrated, according to a representative embodiment.

[0021] Figure 3 A general computer system is illustrated on which a method for dynamically intervening in deformation of a three-dimensional model may be implemented according to another representative embodiment.

[0022] Figure 4 A method for dynamically intervening in deformation of a three-dimensional model according to a representative embodiment is illustrated.

[0023] Figure 5A Illustrated are comparative views of a tracking device and a three-dimensional model of a path with multiple branches before applying a dynamic intervention three-dimensional model deformation, according to a representative embodiment.

[0024] Figure 5B Illustrated is progress for dynamic interventional three-dimensional model deformation using optical shape sensing, according to a representative embodiment.

[0025] Figure 6 Another method of dynamically intervening in deformation of a three-dimensional model according to a representative embodiment is illustrated.

[0026] Figure 7 Illustrated is a virtual bronchoscopic view of the airway from the current position of the tracking device determined using dynamic interventional three-dimensional model deformation, according to a representative embodiment.

[0027] Figure 8 Another method of dynamically intervening in deformation of a three-dimensional model according to a representative embodiment is illustrated.

[0028] Figure 9 Another method of dynamically intervening in deformation of a three-dimensional model according to a representative embodiment is illustrated.

[0029] Figure 10 Another method of dynamically intervening in deformation of a three-dimensional model according to a representative embodiment is illustrated.

[0030] Figure 11 Another method of dynamically intervening in deformation of a three-dimensional model according to a representative embodiment is illustrated.

[0031] Figure 12 Another method of dynamically intervening in deformation of a three-dimensional model according to a representative embodiment is illustrated.

[0032] Figure 13 Illustrated is an endoluminal view of an airway from the current position of a tracking device determined using dynamic interventional three-dimensional model deformation, according to a representative embodiment.

[0033] Figure 14 Another method of dynamically intervening in deformation of a three-dimensional model according to a representative embodiment is illustrated. DETAILED DESCRIPTION

[0034] In the following detailed description, representative embodiments of the disclosure specific details are set forth for purposes of explanation and not limitation, so as to provide a thorough understanding of the embodiments according to the present teachings. Descriptions of known systems, devices, materials, operating methods, and manufacturing methods may be omitted to avoid blurring the descriptions of the representative embodiments. Nevertheless, systems, devices, materials, and methods within the scope of the knowledge of those of ordinary skill in the art are within the scope of the present teachings and may be used in accordance with the representative embodiments. It should be understood that the terms used herein are only for the purpose of describing specific embodiments and are not intended to be limiting. The defined terms are a supplement to the technical and scientific meanings of the defined terms as generally understood and accepted in the technical field of this teaching.

[0035] It will be understood that although the terms first, second, third, etc. may be used herein to describe various elements or components, these elements or components should not be limited by these terms. These terms are only used to distinguish one element or component from another element or component. Therefore, without departing from the teachings of the inventive concept, the first element or component discussed below may be referred to as the second element or component.

[0036] The terms used herein are only used for the purpose of describing specific embodiments and are not intended to be restrictive. As used in the specification and claims, the singular terms "a", "an" and "said" are intended to include both the singular and the plural, unless the context clearly dictates otherwise. In addition, when used in this specification, the terms "include" and / or "comprise" and / or similar terms specify the presence of stated features, elements and / or parts, but do not exclude the presence or addition of one or more other features, elements, parts and / or groups thereof. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0037] Unless otherwise indicated, when an element or component is referred to as being “connected to,” “coupled to,” or “adjacent to” another element or component, it will be understood that the element or component can be directly connected or coupled to the other element or component, or that intervening elements or components may be present. In other words, these and similar terms encompass situations where one or more intermediate elements or components may be employed to connect two elements or components. However, when an element or component is referred to as being “directly connected” to another element or component, this only encompasses situations where the two elements or components are connected to each other without any intermediate or intervening elements or components.

[0038] In view of the foregoing, the present disclosure is intended to present one or more of the advantages specifically noted below through one or more of its various aspects, embodiments, and / or specific features or subcomponents. For purposes of explanation and not limitation, exemplary embodiments disclosing specific details are set forth to provide a thorough understanding of embodiments according to the present teachings. However, other embodiments consistent with the present disclosure that depart from the specific details disclosed herein remain within the scope of the claims. In addition, descriptions of well-known devices and methods may be omitted so as not to obscure the description of the exemplary embodiments. Such methods and devices are within the scope of the present disclosure.

[0039] Figure 2 A system for dynamically intervening in deformation of a three-dimensional model is illustrated, according to a representative embodiment.

[0040] Figure 2 The system 200 includes a first medical imaging system 210 , a computer 220 , a display 225 , a second medical imaging system 240 , and a tracking device 250 . Figure 2 The system may be used for dynamic interventional three-dimensional model deformation, such as for medical intervention to repair, replace, or adjust an organ such as a lung, or otherwise intervene in a subject's body.

[0041] An example of the first medical imaging system 210 is an X-ray system including an X-ray machine. The first medical imaging system 210 can perform a computed tomography scan or a cone-based computed tomography scan prior to a medical intervention involving the three-dimensional model deformation described herein. The three-dimensional model can be generated by the first medical imaging system 210 based on the scan before or during the medical intervention. Furthermore, while the medical imaging performed by the first medical imaging system 210 can be performed before the medical intervention, the medical imaging can also be performed during the medical intervention.

[0042] Tracking device 250 can use optical shape sensing or any other tracking technology, such as electromagnetic sensor-based tracking. Tracking device 250 sends position information to computer 220. Computer 220 processes the medical images from first medical imaging system 210 to generate a three-dimensional model. Computer 220 also processes data from tracking device 250 to perform deformation and adjustments on the three-dimensional model.

[0043] An example of a second medical imaging system 240 is an ultrasound device used to obtain ultrasound images during a medical intervention involving the three-dimensional model deformation described herein. Another example of a second medical imaging system 240 is another X-ray system including an X-ray machine used to obtain fluoroscopic images during a medical intervention involving the three-dimensional model deformation described herein. The second medical imaging system 240 can be used to track an otherwise untracked medical device, causing the medical device to become a tracking device 250. For example, some medical devices do not have optical shape sensing or electromagnetic tracking components disposed therein or thereon. Positioning such a medical device in an X-ray image or using anatomical information from an ultrasound image provides position information for the otherwise untracked medical device, causing it to become a tracking device 250. Of course, the medical device can be a tracking device 250 having tracking technology (e.g., optical shape sensing) embedded therein or thereon, while also being tracked simultaneously with the medical imaging modality of the second medical imaging system 240.

[0044] Furthermore, both the first medical imaging system 210 and the second medical imaging system 240 may be present and used during a medical intervention, such as when the first medical imaging system 210 is an X-ray system for performing fluoroscopic imaging during the medical intervention and the second medical imaging system 240 is an ultrasound system for performing ultrasound imaging during the medical intervention. Alternatively, only one of the first medical imaging system 210 or the second medical imaging system 240 may be present and used during the medical intervention. For example, the first medical imaging system 210 may be used during a certain type of medical intervention, while the second medical imaging system 240 is merely used as an additional component, such as when X-rays or ultrasound are used as a tracking method.

[0045] The first medical imaging system 210 can be used to generate computed tomography images that serve as a basis for generating a three-dimensional model as described herein. Computed tomography images are examples of three-dimensional anatomical images. The second medical imaging system 240 can be used to perform imaging that is used to track a tracking device. A tracking device can refer to an interventional medical device on or in which a sensor, optical shape sensing element, or other tracking element is provided. Figure 2In the present invention, imaging for tracking the tracking device can be performed in real time using the second medical imaging system 240 alone or using both the second medical imaging system 240 and the first medical imaging system 210. In other words, the second medical imaging system 240 can be present and used during the interventional procedure in the absence or presence of the first medical imaging system 210. The second medical imaging system 240 provides imaging data to the computer 220.

[0046] exist Figure 2 In the embodiment of the present invention, computer 220 includes a display 225. The display can be used to display a three-dimensional model based on imaging performed by first medical imaging system 210, along with images obtained during the medical intervention based on imaging performed by second medical imaging system 240. The images obtained during the medical intervention can, for example, be images of an inflated lung being deflated during the medical intervention. When the term "display" is used herein, the term should be interpreted as including a class of features, such as "display device" or "display unit," and these terms encompass output devices or user interfaces suitable for displaying images and / or data. The display can output visual, auditory, and / or tactile data. Examples of displays include, but are not limited to, computer monitors, television screens, touch screens, tactile electronic displays, Braille screens, cathode ray tubes (CRTs), memory tubes, bi-stable displays, electronic paper, vectorscope displays, flat panel displays, vacuum fluorescent displays (VFs), light emitting diode (LED) displays, electroluminescent displays (ELDs), plasma display panels (PDPs), liquid crystal displays (LCDs), organic light emitting diode displays (OLEDs), projectors, and head-mounted displays.

[0047] Figure 2Any of the elements in can include the controller described herein. The controller described herein can include a memory for storing instructions and a combination of a processor for executing instructions to implement the process described herein. The controller can be housed in a workstation or linked to a workstation, such as another component of a computer 220 or one or more computing devices, a display / monitor, and one or more input devices (e.g., keyboard, joystick and mouse) in the form of an independent computing system, a client computer of a server system, a desktop computer or a tablet computer. The descriptive mark used in this article for the term "controller" promotes the distinction between controllers as described herein without specifying or implying any additional restrictions of the term "controller". The term "controller" broadly encompasses all structural configurations of a dedicated motherboard or a dedicated integrated circuit for controlling the application of the various principles described in this disclosure, as understood in the field of this disclosure and as exemplarily described in this disclosure. The structural configuration of the controller can include, but is not limited to, (one or more) processors, (one or more) computer-usable / computer-readable storage media, an operating system, (one or more) application modules, (one or more) peripheral device controllers, (one or more) slots and (one or more) ports.

[0048] In addition, despite Figure 2 Components are shown networked together, but two such components may be integrated into a single system. For example, the computer 220 may be integrated with the first medical imaging system 210. That is, in an embodiment, the functionality attributed to the computer 220 may be implemented (e.g., performed) by a system including the first medical imaging system 210. In another aspect, Figure 2 The networking components shown may also be spatially distributed, such as by being distributed in different rooms or different buildings, in which case the networking components may be connected via data connections. Figure 2 One or more of the components in the system are not connected to the other components via a data connection and are instead provided with input or output manually, such as via a memory stick or other form of storage. In yet another embodiment, the functionality described herein may be based on Figure 2 The function of the components in Figure 2 Executed outside the system shown.

[0049] Figure 2 Any of the first medical imaging system 210, the computer 220, and the second medical imaging system 240 may include the following Figure 3Some or all of the elements and functions of the general computer system described herein may be included. For example, the computer 220 may include a controller for determining whether the current position of the tracking device is outside of a path in the three-dimensional model. The process performed by the controller may include receiving a three-dimensional model of a path having multiple branches in an object of an interventional procedure or receiving image data, and generating a three-dimensional model of a path having multiple branches in an object of an interventional procedure based on the image data.

[0050] The process implemented when the controller of computer 220 executes instructions further includes determining whether the current position of the tracking device is outside the path in the three-dimensional model, and, if the current position of the tracking device is outside the path in the three-dimensional model, deforming the three-dimensional model to the current position of the tracking device. The same controller may also perform the function of generating a three-dimensional model based on segmenting a path having multiple branches in an object of an interventional procedure. However, the controller tracking the position of the tracking device may obtain a segmented three-dimensional model generated and segmented elsewhere, such as by the first medical imaging system 210 or by the controller 220 executing instructions to process medical images created by the first medical imaging system 210. That is, the process implemented by the controller as described herein may include obtaining a three-dimensional model generated prior to the interventional procedure, wherein the three-dimensional model is generated based on segmenting a path having multiple branches in an object of the interventional procedure. However, the three-dimensional model need not be generated "prior" to the interventional procedure. For example, in some embodiments, the model is generated or updated during the interventional procedure, and subsequent processes are still performed thereafter. As described above, even the medical imaging performed by the first medical imaging system 210 can be performed during the same medical intervention in which the deformation of the interventional three-dimensional model is performed.

[0051] Figure 3 A general computer system is illustrated on which a method for dynamically intervening in deformation of a three-dimensional model may be implemented according to another representative embodiment.

[0052] Computer system 300 may include a set of instructions that can be executed to cause computer system 300 to perform any one or more of the methods or computer-based functions disclosed herein. Computer system 300 may operate as a standalone device or may be connected to other computer systems or peripheral devices, for example using network 301.

[0053] In a networked deployment, the computer system 300 can operate in the capacity of a server or as a client user computer in a server-client user network environment or as a peer computer system in a peer (or distributed) network environment. The computer system 300 can also be implemented as or incorporated into various devices, such as the first medical imaging system 210, the computer 220, the second medical imaging system 240, a fixed computer, a mobile computer, a personal computer (PC), a laptop computer, a tablet computer, or any other machine capable of executing a set of instructions (sequential or otherwise) specifying an action to be taken by the machine. The computer system 300 can be incorporated into or into an integrated system that then includes additional devices. In an embodiment, the computer system 300 can be implemented using electronic devices that provide voice, video, or data communication. In addition, although the computer system 300 is illustrated in the singular, the term "system" should also be understood to include any collection of systems or subsystems that individually or jointly execute one or more sets of instructions to perform one or more computer functions.

[0054] like Figure 3 As shown, computer system 300 includes processor 310. The processor for computer system 300 is tangible and non-transient. As used herein, the term "non-transient" should not be interpreted as a permanent characteristic of a state, but rather as a characteristic of a state that will persist over a period of time. The term "non-transient" explicitly negates fleeting characteristics, such as characteristics of a carrier wave or signal or other forms that only exist transiently at any place at any time. A processor is an article of manufacture and / or a machine component. The processor for computer system 300 is configured to execute software instructions to perform the functions described in various embodiments of this document. The processor for computer system 300 can be a general-purpose processor, or can be part of an application-specific integrated circuit (ASIC). The processor for computer system 300 can also be a microprocessor, a microcomputer, a processor chip, a controller, a microcontroller, a digital signal processor (DSP), a state machine, or a programmable logic device. The processor for computer system 300 can also be a logic circuit, including a programmable gate array (PGA) such as a field programmable gate array (FPGA), or another type of circuit including discrete gates and / or transistor logic. The processor for computer system 300 can be a central processing unit (CPU), a graphics processing unit (GPU), or both. In addition, any processor described herein may include multiple processors, parallel processors, or both. Multiple processors can be included in a single device or multiple devices, or coupled to a single device or multiple devices.

[0055] As used herein, "processor" encompasses an electronic component capable of executing a program or machine-executable instructions. References to a computing device that includes a "processor" should be interpreted as potentially including more than one processor or processing core. The processor may, for example, be a multi-core processor. A processor may also refer to a collection of processors within a single computer system or distributed across multiple computer systems. The term computing device should also be interpreted as potentially referring to a collection or network of computing devices that each include one or more processors. Many programs have instructions that are executed by multiple processors that may be within the same computing device or may even be distributed across multiple computing devices.

[0056] In addition, the computer system 300 may include a main memory 320 and a static memory 330, wherein the memories may communicate with each other via a bus 308. The memory described herein is a tangible storage medium that can store data and executable instructions and is non-transient during the time the instructions are stored therein. As used herein, the term "non-transient" should not be interpreted as a permanent characteristic of a state, but rather as a characteristic of a state that will persist over a period of time. The term "non-transient" explicitly disclaims fleeting characteristics, such as those of a carrier wave or signal or other forms that only exist transiently in one place at any time. The memory described herein is an article of manufacture and / or a machine component. The memory described herein is a computer-readable medium from which data and executable instructions can be read by a computer. The memory described herein may be a random access memory (RAM), a read-only memory (ROM), a flash memory, an electrically programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a register, a hard disk, a removable disk, a magnetic tape, a compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a floppy disk, a Blu-ray disc, or any other form of storage medium known in the art. The memory may be volatile or non-volatile, secure and / or encrypted, unsecure and / or unencrypted.

[0057] "Memory" is an example of a computer-readable storage medium. Computer memory is any memory directly accessible by a processor. Examples of computer memory include, but are not limited to, RAM, registers, and register files. References to "computer memory" or "memory" should be interpreted as potentially including multiple memories. A memory may, for example, be multiple memories within the same computer system. A memory may also be multiple memories distributed across multiple computer systems or computing devices.

[0058] As shown, the computer system 300 may also include a video display unit 350, such as a liquid crystal display (LCD), an organic light emitting diode (OLED), a flat panel display, a solid state display, or a cathode ray tube (CRT). In addition, the computer system 300 may include an input device 360, such as a keyboard / virtual keyboard or a touch-sensitive input screen or voice input with voice recognition, and a cursor control device 370, such as a mouse or a touch-sensitive input screen or pad. The computer system 300 may also include a disk drive unit 380, a signal generating device 390, such as a speaker or a remote control, and a network interface device 340.

[0059] In an embodiment, Figure 3 As depicted in FIG, disk drive unit 380 may include a computer-readable medium 382 in which one or more sets of instructions 384, such as software, may be embedded. The sets of instructions 384 may be read from computer-readable medium 382. Furthermore, when executed by a processor, the instructions 384 may be used to perform one or more of the methods and processes described herein. In an embodiment, the instructions 384 may reside completely or at least partially within main memory 320, static memory 330, and / or processor 310 during execution by computer system 300.

[0060] In alternative embodiments, dedicated hardware implementations, such as application specific integrated circuits (ASICs), programmable logic arrays, and other hardware components, can be constructed to implement one or more of the methods described herein. One or more embodiments described herein can use two or more specific interconnected hardware modules or devices to implement functions using related control and data signals that can be communicated between modules and through modules. Therefore, the present disclosure encompasses software, firmware, and hardware implementations. Nothing in this application should be interpreted as being implemented or achievable solely using software without hardware such as tangible non-transient processors and / or memories.

[0061] According to various embodiments of the present disclosure, the methods described herein may be implemented using a hardware computer system that executes a software program. Furthermore, in exemplary, non-limiting embodiments, implementations may include distributed processing, component / object distributed processing, and parallel processing. Virtual computer system processing may be constructed to implement one or more of the methods or functions described herein, and the processors described herein may be used to support virtual processing environments.

[0062] This disclosure contemplates computer-readable media 382 including instructions 384 or receiving and executing instructions 384 in response to a propagated signal; such that devices connected to network 301 can communicate video or data over network 301. Additionally, instructions 384 can be transmitted or received over network 301 via network interface device 340.

[0063] Figure 4 A method for dynamically intervening in deformation of a three-dimensional model according to a representative embodiment is illustrated.

[0064] Figure 4 The method in begins at S405 by generating, capturing, and storing a medical image of the subject of the interventional procedure. In some embodiments of the dynamic interventional three-dimensional model deformation, a preoperative computed tomography (CT) or cone beam computed tomography (CBCT) image is acquired. Cone beam CT is a medical imaging technique that includes X-ray computed tomography in which X-rays diverge and thus form a cone. In most embodiments, the airways are segmented as described herein. In most embodiments, computed tomography or cone beam CT and segmentation can be performed prior to the interventional procedure. The three-dimensional image can be acquired using computed tomography, cone beam CT, magnetic resonance, or other imaging modalities that provide a three-dimensional representation of the anatomical structure.

[0065] At S410, Figure 4 The method in includes generating a three-dimensional model based on segmenting a path having multiple (many) branches in an object of an interventional procedure. The three-dimensional model of the path can be generated based on cone beam computed tomography or any other three-dimensional imaging modality. The three-dimensional model has airways in the example of a lung, but can be blood vessels in an alternative embodiment. In addition, although S410 specifies that the three-dimensional model is created by segmentation, the three-dimensional model can also be created by other mechanisms such as rendering. Segmentation is a representation of the surface of a structure such as the paths and branches of a lung or a component of a heart and, for example, includes a set of points in three-dimensional (3D) coordinates on the surface of the organ, and triangular plane segments defined by connecting adjacent groups of 3 points so that the entire structure is covered by a mesh of non-intersecting triangular planes.

[0066] At S415, Figure 4 The method includes generating a path from a starting point to a target point in a three-dimensional model.

[0067] As specified for the embodiments described later, the tracking device can be registered to the three-dimensional model. Registration involves placing different components or systems onto a common coordinate system. For example, the shape of a tracked interventional medical device can be registered to a preoperative lung image, which then enables the ability to account for respiratory motion, misalignment (e.g., lung deformation) and other movements of anatomical structures based on interaction with the tracked interventional medical device, and the ability to plan lung navigation in the airway. Shape sensing can be used for real-time guidance in the presence of respiratory / cardiac motion using error deviations of the current pathway from the planned pathway to help the user time navigation inside and outside the airway (off-road).

[0068] At S470, Figure 4The method includes determining whether a current position of the tracking device is outside a path in the three-dimensional model. Figure 4 In the embodiment of the present invention, the tracking device tracks in three dimensions as it navigates the airway. Tracking can be performed using optical shape sensing, but in other embodiments, tracking can involve electromagnetic sensors, ultrasound-based sensors, or X-ray-based device identification based on dynamic analysis of fluoroscopic images. In the methods described herein, optical shape sensing is generally described. However, the tracking device described herein can alternatively track using electromagnetic sensors, ultrasound-based sensors, or X-ray-based device identification from fluoroscopic images.

[0069] At S480, Figure 4 The method includes deforming the three-dimensional model to the current position of the tracking device when the current position of the tracking device is outside the path in the three-dimensional model. The deformation at step S480 results in adjusting the visualization viewed by the physician using the preoperative three-dimensional model of the airway and the real-time interventional medical device position. Thus, the deformation at step S480 results from tracking, as long as the current path of the tracking device can be compared to the airway lumen (open space), and if the current path is outside the nearest airway, the computed tomography / cone-beam computed tomography image and the three-dimensional model are deformed.

[0070] At S485, Figure 4 The method includes: when the current position of the tracking device is within the path, deforming the three-dimensional model to center the tracking device within the current path of the three-dimensional model. As an alternative to S485, the method may include: when the current position of the tracking device is within the path, not deforming the three-dimensional model at all.

[0071] Figure 4 The method provides physicians with the ability to position interventional medical devices relative to actual anatomical structures. In implementing anatomical segmentation of a three-dimensional model, the three-dimensional model can be based on image acquisition as at S405. Physicians using this three-dimensional model for real-time guidance can obtain information about how the anatomical structure changes in real time that would otherwise be lost. For example, the airway can stretch or contract several centimeters throughout a respiratory cycle and will shift when a hard device such as a bronchoscope is inserted. Tracking of interventional medical devices allows real-time information about the position to be used to update the three-dimensional model based on its preoperative state.

[0072] The specific order shown will generally apply to the embodiments described herein. Figure 4 However, other steps for the methods described herein may be performed in a different order than shown or may be performed on an ongoing basis simultaneously with each other.

[0073] also, Figure 4The numbering of the steps in the embodiment may be partially or completely applicable to other embodiments described herein. For example, Figure 6 The number S680 in the figure can be understood to mean that the step S680 can be replaced by Figure 4 Similarly, Figure 6 The numbering of S660 in the figure can be understood to mean that the step can be relative to the step for Figure 4 The functions described are performed before S470 but after S415. Therefore, the relative values ​​of the last two numbers of the steps described herein for various embodiments may be understood as relative to Figure 4 General or specific placement of the number of steps in the.

[0074] Figure 5A Illustrated are comparative views of a tracking device and a three-dimensional model of a path with multiple branches before applying a dynamic intervention three-dimensional model deformation, according to a representative embodiment.

[0075] exist Figure 5A In the present invention, the three-dimensional model of the pathway structure(s) is based on a point in time at which computed tomography images are acquired before being used for segmentation to generate the three-dimensional model based on the images. Figure 5A illustrates how tracking interventional medical devices can deviate from a 3D model as the anatomy changes in real time. This is illustrated in Figure 5A The 3D model is shown in images #2, #4, and #6 of Figure 1. Therefore, if a physician is using an underlying 3D model for real-time guidance, they lack information about how the anatomy is changing in real time. For example, the airway can expand or contract by several centimeters throughout a respiratory cycle and will shift when a hard device such as a bronchoscope is inserted. Figure 5A The 3D preoperative model is used to visually show how the interventional medical device will appear relative to the preoperative model. Tracking the interventional medical device allows real-time information about its location and is used herein to update what the 3D preoperative model looks like to the physician. This can be particularly useful when intraoperative fluoroscopy is not used.

[0076] exist Figure 5A In the images #1, #3, #5, and #7, the tracking device is aligned with the airway. However, in images #2, #4, and #6, the interventional medical device appears outside the airway, even though the tracking device is actually within the lumen of the airway. Figure 5A During several phases of the respiratory cycle shown, the interventional medical device appears to be outside the airway when, in fact, the interventional medical device remains within the lumen.

[0077] Figure 5B Illustrated is progress for dynamic interventional three-dimensional model deformation using optical shape sensing, according to a representative embodiment.

[0078] exist Figure 5B In the present invention, the three-dimensional segmented airway is deformed using optical shape sensing feedback about respiratory motion, cardiac motion, and anatomical motion due to interaction with the tracking device (e.g., lung deformation). The segmented airway shown in image #1 is a preoperative segmented airway, where R=0 refers to the acquisition of the underlying medical image at a specific phase of the respiratory cycle (e.g., at full inspiration). For example, the medical image underlying image #1 can be a computed tomography image acquired by the first medical imaging system 210 long before the medical intervention. Image #1 shows a preoperative segmented model of the airway without tracking the interventional medical device, and image #2 shows a preoperative segmented model of the airway with the endobronchial interventional medical device navigating the airway. In image #2, tracking is performed using optical shape sensing as described herein. R=0 in image #2 can be understood to mean that the respiratory phase at this point in the medical intervention is at the same respiratory phase as was seen when the preoperative image was acquired.

[0079] exist Figure 5B , at R=1, the airway moves during the respiratory cycle such that the current position(s) of the airway are partially, primarily, or completely offset from the original position of the preoperative segmentation model shown in image #1. This is illustrated by the replicated path in image #3 being over and offset from the original path of the preoperative segmentation model carried over image #1. That is, as shown in image #3, at time R=1 (e.g., full exhalation of the respiratory cycle), the tracked interventional medical device and the actual path structure have moved away from the preoperative segmentation model shown in image #1. Thus, in image #4, the segmentation model is deformed such that the tracked interventional medical device and the actual path structure are adapted to the actual path structure present at R=1.

[0080] As shown in image #4, deformation can involve shrinking, expanding, shifting, or otherwise moving the 3D model to fit the current position of the tracking device. Figure 5B In [ 1 ], the position of the lungs during the respiratory phase of the preoperative 3D scan acquisition is defined as R = 0, and another phase of the respiratory cycle different from the preoperative acquisition is defined as R = 1. The 3D segmented airway from R = 0 is shown in each of images #1, #2, #3, and #4, while in images #3 and #4, an offset structure is superimposed on the 3D segmented airway before and after deformation. The offset structure represents the actual position of the path at R = 1. The tracked interventional medical device 501 is marked in each of images #2, #3, and #4.

[0081] Tracking interventional medical devices using optical shape sensing tracking is described Figure 5BHowever, electromagnetic sensing may alternatively be used with either a single sensor at the tip of the tracking device or multiple sensors along the length of the tracking device. For the electromagnetic sensor at the tip, recording of the tip position may be performed continuously to track the path taken by the entire tracking device.

[0082] As above relative to Figure 4 As described, the main elements of the methods described herein may include acquiring a three-dimensional scan of the patient's airway using either preoperative computed tomography or intraoperative cone-beam computed tomography. Typically, the preoperative computed tomography or intraoperative cone-beam computed tomography is performed at one stage of the respiratory cycle. The airway and lesions are segmented from the computed tomography data to create a three-dimensional model of the airway. The resulting three-dimensional model of the airway is Figure 5B The starting point in image #1 in FIG. During an endobronchial pulmonary procedure, an interventional medical device can be navigated over a segmented model of the airway to the desired position and, if there is no cardiac or respiratory motion, be made to appear consistently within the airway. Examples of endobronchial pulmonary procedures include lesion or lymph node biopsy, tumor ablation, airway stenting, tumor resection, and other forms of pulmonary procedures. If there is no cardiac or respiratory motion, the interventional medical device can be navigated over a segmented model of the airway and will always appear to remain within the airway. However, significant motion occurs due to cardiac and respiratory motion as well as slight movements of the patient and inserted devices. When motion occurs, the interventional medical device appears to be outside the modeled airway and can mislead the physician. One benefit of optical shape sensing is that the positional information of the interventional medical device is always known along its entire length. This information can be used to update the model of the airway in real time to provide a more realistic image of the interventional medical device relative to the anatomy. The upper airway is very rigid, and therefore, it is most likely that the interventional medical device will be within the lumen of the airway. However, in the peripheral (distal) airways, the walls are quite thin and the tip of the interventional medical device can easily protrude beyond the wall. Therefore, the deformation of the model should primarily use the information of the interventional medical device when in the upper airway.

[0083] Figure 6 Another method of dynamically intervening in deformation of a three-dimensional model according to a representative embodiment is illustrated.

[0084] Figure 6 The process in begins at S630 by registering the tracking device in the path to the three-dimensional model. Registration is a process that involves aligning different coordinate systems or assigning coordinates of an existing coordinate system to newly introduced elements. At S630, the coordinate system of the three-dimensional model can be used to assign coordinates to the tracking device so that the tracking device can track in the coordinate system of the three-dimensional model. The tracking device can track using optical shape sensing, and the three-dimensional model can be such as Figure 5BPreoperative models of all four images in .

[0085] At S675, Figure 6 The process in includes calculating the offset between the immediately previous position of the tracking device and the current position of the tracking device relative to the three-dimensional model. To the extent that the position of the tracking device can be continuously and rapidly calculated during the interventional procedure, the offset can be used to help plot the trajectory and path for the tracking device and also to help as in Figure 4 The calculation at S675 can be performed at a position such as S480 in FIG. Figure 4 The deformation at S470 is performed after that, wherein a check is performed as to whether the tracking device is inside the airway lumen of the mesh of the three-dimensional model.

[0086] At S680, Figure 6 The process includes transforming the three-dimensional model to the current position of the tracking device based on the offset. For example, at S680, the transformation may involve adjusting the deformation of the three-dimensional model and the image so that the three-dimensional model includes one or more immediately previous positions of the tracking device and the current position of the tracking device.

[0087] The transformation at S680 may involve warping the entire preoperative three-dimensional model to the current position of the tracking device. The transformation may be based on recording the last known position of the tracking device and the current position of the tracking device, and calculating the offset at S675, as long as the recorded position is useful, for example, to identify which branches the tracked interventional medical device has traversed. The system 200 may remember the history of the tracking device at all times and all positions to facilitate the calculation at S675 and the transformation at S680.

[0088] As an alternative to S680, if the tracking device is inside the airway lumen of the mesh of the three-dimensional model as determined at S470, then Figure 6 The process in may involve showing the pre-operative three-dimensional model in an undeformed state. Alternatively, small deformation corrections may be applied to keep the interventional medical device in the center of the lumen.

[0089] At S690, Figure 6 The process in step 690 includes iteratively locally deforming each new branch of the three-dimensional model containing the tracking device each time the tracking device moves to a new branch. At step S690, the position of the tracking device in each airway at each time point can be iteratively used to locally deform the airway to fit the tracking device, thereby maintaining the tracking device in the center of the airway lumen. The position can be identified after each deformation and each time the tracked interventional medical device is further advanced, and each time the tracking device is further advanced, the local deformation can be performed again.

[0090] exist Figure 6In the method, we can obtain Figure 5B The deformation shown in image #4. Figure 6 In the embodiment of the present invention, the tracking device can be tracked using optical shape sensing. When the tracking device is out of the path, the path of the 3D model is adjusted to the position of the tracking device. Figure 6 An important part of the method is that the system 200 stores a history of multiple or even all time points and positions of the tracking device as it moves through the airway. These time points and positions may be critical for tracking devices where only the tip is tracked or where there is minimal tracking along the length of the tracking device. The position history can be used to deform the 3D model along the entire device trajectory, not just at the tip of the tracking device.

[0091] exist Figure 6 In the embodiment of Figure 5B The method is performed while navigating the optical shape sensing device in the manner shown in images #2, image #3 and image #4 of the airway. The optical shape sensing device is registered to the preoperative three-dimensional model. When checking whether the optical shape sensing device is inside the airway lumen / model grid, if so, then the preoperative three-dimensional model is shown in a non-deformed state or alternatively, only a small deformation correction is applied to keep the tracking device in the center of the lumen. On the other hand, if the optical shape sensing device is not inside the airway lumen / model grid, then the preoperative three-dimensional model is deformed to the current position of the optical shape sensing device. Since the position of the optical shape sensing device is continuously recorded, the offset between the last position and the current position can be calculated relative to the position in the preoperative three-dimensional model of the airway to help identify which branches have been traversed. In this embodiment, the system can remember the history of all time points and positions of the optical shape sensing device.

[0092] If also for Figure 6 As described in the embodiment of the present invention, the preoperative three-dimensional model is transformed as a whole to the new device position. Later, as the optical shape sensing device is advanced in the airway, the position is recorded at each time point and then used to deform the airway to fit the optical shape sensing device, such as by keeping the optical shape sensing device centered in the airway lumen. Figure 6 In the method, the system stores a history of all device time points and positions as the interventional medical device moves through the airway. These time points and positions may be important for devices where only the tip is tracked or where there is minimal tracking along the length of the interventional medical device. The position history can be used to deform the model along the entire device trajectory, not just at the tip of the interventional medical device.

[0093] Description of the device using optical shape sensing Figure 6However, the electromagnetic sensing device can be used with either a single sensor at the tip or multiple sensors along the length of the interventional medical device. For the electromagnetic sensor at the tip, recording of the tip position can occur continuously to track the path taken by the electromagnetic sensing device.

[0094] Figure 7 Illustrated is a bronchoscopic view of the airway from the current position of the tracking device determined using dynamic interventional three-dimensional model deformation, according to a representative embodiment.

[0095] exist Figure 7 In FIG, the bronchoscopic view is shown as an image separate from the image of the three-dimensional model. The tracking device is shown in the path of the three-dimensional model together with the path taken by the tracking device. Figure 4 and Figure 6 The method described in

[0014] can determine the position of a tracked interventional medical device relative to a three-dimensional model of an anatomical structure. A bronchoscopic view can be created based on the position of the tip of the tracked interventional medical device, such as when the tracked interventional medical device is tracked using optical shape sensing. It can also be shown from any other fixed position on the medical device of interest.

[0096] A virtual bronchoscopic view can be formed from a segmented 3D model based on computed tomography data. The tracking device's position information can be used to determine the tracking device's most recent position relative to the planned pathway. Simply calculating the distance between the planned pathway and the current pathway and executing one or more error minimization routines can determine the position of the interventional medical device along the planned pathway. The bronchoscopic view at that location can then be displayed to the user and automatically updated as the tracking device moves through the airway.

[0097] The three-dimensional position information of the tracking device relative to the three-dimensional model can be used to show a virtual bronchoscopic view of the interior of the airway.

[0098] For example, the three-dimensional model can be dynamically updated, and the tip of the tracking device can be shown using the deformed three-dimensional model. Alternatively, a pathway from the trachea to the target can be planned, and then the position of the tracking device can be compared to the planned pathway. Error minimization can be used to select the most appropriate position for the device tip for the planned pathway and show the bronchoscopic view from that point in the three-dimensional model. As an example of a method implemented by the controller, a virtual bronchoscopic view can be created based on the position of a fixed point of the tracking device; the closest position of the planned pathway to the position of the fixed point of the tracking device is determined; and the bronchoscopic view is automatically updated as the tracking device moves through the three-dimensional model.

[0099] In an embodiment, the orientation of the tracking device can be identified and then used to determine the viewpoint for the virtual bronchoscopic view based on the orientation of the tracking device. For example, if the user wants to show a bronchoscopic view in front of the tip of the tracking device and show where the device tip will move next, the orientation can be used to ensure that the user does not view areas to the side or behind the tip. Similarly, knowing when to show a side view can also provide advantages to the user, such as when the tracking device tip approaches a branch in the three-dimensional model. Proximity to a branch can automatically be used to trigger the display of multiple virtual bronchoscopic views to help the user move down the correct branch.

[0100] Figure 8 Another method of dynamically intervening in deformation of a three-dimensional model according to a representative embodiment is illustrated.

[0101] Figure 8 The method in begins at S810 by generating a three-dimensional model based on segmenting a path having multiple (many) branches in an object of an interventional procedure.

[0102] At S815, Figure 8 The method includes generating a path from a starting point to a target point in a three-dimensional model.

[0103] At S830, Figure 8 The method includes registering a tracking device in a path to a three-dimensional model.

[0104] At S850 , the position of the tracking device is tracked relative to the three-dimensional model as the lungs deflate.

[0105] At S870, Figure 8 The method includes deforming only a local branch of a path containing a tracking device, or two local branches of a path closest to the tracking device, based on the amount of movement of the tracking device over time. For example, the amount of movement can be used as a basis for calculating an offset, and the 3D model can then be corrected based on the offset.

[0106] Figure 8 The approach in this paper is similar in many key respects to Figure 4 The methods in

[15] overlap, but also differ in several respects. For example, Figure 8 The method in is specific to medical interventions involving the lungs and even involves tracking the position of the tracking device as the lungs deflate. Furthermore, at S880, deformation is limited to only one or two local branches, but this is not prohibited at S480. On the contrary, as should be clear, such as Figure 8 Features of various embodiments of the present invention may be exchanged with features of other embodiments, or added to features of other embodiments.

[0107] As described above, Figure 8The method involves using interventional medical device tracking to align a deflated lung model and an inflated lung model. An optical shape sensing device (or another tracking device) is aligned to a preoperative three-dimensional model. Prior to surgery, the optical shape sensing device (or other tracking device) is navigated to the location of the target lesion (intrabronchial). For example, navigation can be performed using the method described above. The history of all device time points and positions is recorded with respect to the preoperative model in order to identify which branches have been traversed and in which lobe of the lung the optical shape sensing device is. The lung is then deflated, and the position of the optical shape sensing device is tracked relative to the preoperative model as the lung is deflated. The preoperative model is then deformed to the current position of the optical shape sensing device. The deformation can be local, such as deforming only the lobe containing the optical shape sensing device and the target lesion. Alternatively, in the case where the lesion is located between two lobes, both lobes can be deformed. The newly deformed three-dimensional model is presented to the user to represent the deflated lung state.

[0108] Figure 9 Another method of dynamically intervening in deformation of a three-dimensional model according to a representative embodiment is illustrated.

[0109] exist Figure 9 In , the process begins at S950 by continuously tracking the tracking device while manipulating the lung.

[0110] At S980, Figure 9 The process in includes deforming only the local branch of the path containing the tracked device or the two local branches of the path closest to the tracked device to correct for lung motion.

[0111] As described above, for Figure 9 The method of the present invention can use the deformation described in this article to track lung motion during surgical resection. As with other embodiments, Figure 9

[0014] Embodiments of the present invention relate to registering an optical shape sensing device (or other tracking device) to a preoperative model. The tracking device is navigated intrabronchially to the location of the target lesion prior to surgery.

[0112] During surgical dissection, the motion of the registered optical shape sensing device is tracked as the lung is manipulated (such as by being pulled, stretched, or flipped by the surgeon). Based on the motion of the optical shape sensing device, the preoperative model is deformed to the current position of the optical shape sensing device. The deformation can be localized, such as by deforming only the lobe containing the optical shape sensing device and the target lesion. Alternatively, in cases where the lesion is located between two lobes, both lobes can be deformed.

[0113] Figure 6 、 Figure 8 and Figure 9 The methods of have some degree of similarity, but describe three different use cases. Figure 6The use case for this is to move an optical shape sensing tracking device from the trachea to the distal airways. Figure 8 The use case for [ ] is to hold the optical shape sensing tracking device in one airway and collapse the lung. Figure 9 The use case is to keep the optical shape sensing tracking device in one airway, but physically move the lung using tools from the external surface of the lung, as is done during surgery.

[0114] against Figure 9 The algorithm of the surgical approach is now described and also applies to Figure 6 and Figure 8 The following description details the use of Figure 9 The algorithm is based on real data of the surgical method. Initially, a three-dimensional image of the anatomical structure of interest can be acquired preoperatively or intraoperatively, and the three-dimensional anatomical image is segmented to form a mesh of the anatomical structure of interest, such as the airway. The three-dimensional mesh includes multiple faces and vertices (i.e., points in a three-dimensional coordinate system). The coordinate system of the optical shape sensing device and the imaging modality are aligned so that the x, y, z position of the optical shape sensing device is in the same coordinate system as the x, y, z position of the airway mesh. The position of the optical shape sensing device is continuously measured at this point in the workflow, and the position coordinates are sent to a computer processor.

[0115] For a sequence of frames starting at frame (n-1), the optical shape sensing device coordinates are stored. At this point, the tissue can be manipulated in many ways, but manipulation of the tissue is not particularly required here. At frame (n), the distance between the optical shape sensing device coordinates at frame (n) and at frame (n-1) is calculated. If the optical shape sensing device includes multiple position points, the distance between each corresponding point along the optical shape sensing device is calculated. This offset between frames is then stored. For each point in the grid, the distance between grid point (i) and optical shape sensing device point (j:j+N) is calculated. The optical shape sensing device point closest to grid point (i) is determined via a distance minimization calculation. The previously calculated offset at the optical shape sensing device point is added to grid point (i), and the new grid point (i) coordinates are stored. This is repeated for all grid points. After all grid points are adjusted by the optical shape sensing device offset, a new grid that has been dynamically modified based on the position of the optical shape sensing device can be displayed. This continuous process can be performed for all frames (n:n+M) of optical shape sensing data - this results in real-time mesh visualization.

[0116] against Figure 9 The above implementation of the embodiment can also be applied to the device movement and deflated lung respectively. Figure 6 and Figure 8However, the algorithm can be modified, such as by adding thresholds to the distance calculations and offsets to exclude deformations of portions of the mesh. For example, if the optical shape sensing device is in the airway of the right lung, deformation of the left lung may be undesirable because information about how much the lung is moving may not be available. Furthermore, it may be optimal to deform only the lobe of the lung where the optical shape sensing device is present, rather than other lobes of the same lung. Setting a threshold on the maximum acceptable distance from the optical shape sensing device to a grid point can limit which grid points are adjusted.

[0117] Furthermore, the grid is a three-dimensional volume of anatomical structures. A centerline can be drawn within the grid. A determination can be made as to whether the optical shape sensing device is within the grid volume or, if not, how far off-center the grid is. The optical shape sensing device position can be compared to the grid centerline. Using error minimization techniques, the optical shape sensing device position relative to the centerline can be calculated for the best-matched position.

[0118] In an embodiment, a threshold may be implemented to determine whether the optical shape sensing device is outside the mesh volume. Thus, the mesh may be fine-tuned to deform.

[0119] In another embodiment, the best matching position will determine which branch, lobe, or lung the optical shape sensing device is currently located in. This information can also be used to also limit which parts of the mesh are deformed and which parts are left undeformed.

[0120] In yet another embodiment, the distance from the centerline can also be used to transform the grid. For example, if the optical shape sensing device is only slightly offset from the centerline, the transformation (or movement) required to re-center the optical shape sensing device relative to the grid can be calculated, allowing the optical shape sensing device to always "appear" in the center of the grid.

[0121] In another embodiment, the continuous storage of the optical shape sensing device's position can be used to determine the current branch in which the optical shape sensing device is located. The branch in which the optical shape sensing device is located can be determined using, for example, the best match described above, but historical information can be useful in reducing the computational processing time required to determine the current branch. If historical data indicates that the optical shape sensing device is already on the left side of the lung, the centerline calculation described above can be used to eliminate the right branch from the calculation and focus solely on the left branch.

[0122] Continuous storage of the device's position can also be used for devices that do not use optical shape sensing or that only track one or a few selected points. These coordinates can be saved for each frame of data to establish device tracking that can then be used in a similar manner to the implementation described above.

[0123] Figure 10Another method of dynamically intervening in deformation of a three-dimensional model according to a representative embodiment is illustrated.

[0124] exist Figure 10 In , the process starts at S1030 by registering the three-dimensional model to the two-dimensional X-ray image space.

[0125] At S1050, Figure 10 The process in continues by tracking the tracking device in two dimensions based on the X-ray imaging and identifying the position of the tracking device in the fluoroscopic image based on the X-ray imaging.

[0126] At S1055, Figure 10 The process includes projecting a three-dimensional model to overlay the three-dimensional model onto a two-dimensional X-ray image space as a tracking device navigates through a path under guidance of X-ray based tracking.

[0127] As described above, Figure 10 The method involves using X-ray based tracking of the device to identify the interventional medical device position in two dimensions and adjusting the three-dimensional orientation and position of the three-dimensional segmentation model. Figure 10 The method comprises registering the three-dimensional segmentation model to the two-dimensional X-ray image space, such as by using a fiducial, a marker, or an isocenter. The three-dimensional segmentation model is projected in two dimensions to superimpose the model on the X-ray image. Next, the tracking device is navigated down the airway under two-dimensional fluoroscopic guidance, and the image frames from the fluoroscopic view are analyzed using image processing techniques to automatically identify the interventional medical device. The position of the tracking device can be extracted from the image and compared to the projected image of the three-dimensional model. The position of the tracking device can be checked to see if the tracking device is within the projected airway. Alternatively, the distance between the position of the interventional medical device and the centerline of the projected airway can be calculated using a threshold value that is set to define whether the interventional medical device is within the airway.

[0128] Figure 10 The method can be modified to apply to the segmentation of blood vessels instead of airways. The distance between the interventional medical device position and the center line of the projected blood vessel can be calculated again. Similarly, the Figure 10 The proposed method allows for the projection model of the airway to be deformed to fit the actual device location if the interventional medical device is found outside the actual airway or vessel. A small fiducial can be used to account for the out-of-plane component of motion in 2D X-ray images, as this can be challenging using only 3D projections. Furthermore, X-ray images can be acquired from alternative angles to occasionally adjust for out-of-plane information.

[0129] In addition, despite the Figure 2 Not shown, but the tracking device 250 can be tracked with the assistance of a hub, such as when the tracking device 250 is being used as Figure 10When registering the tracking device to the fluoroscopic imaging in the trachea. An alternative method for registering the tracking device to the fluoroscopy is described as follows: a hub as described herein can be placed in the trachea. The hub can contain a pathway of a known pattern. When the tracking device 250 is guided through the hub, the trajectory of the tracking device 250 is checked for that particular pathway. Once a particular hub pathway is found in the device trajectory, it can be used to register the tracking device 250 to the hub. The fluoroscopic image can then be used to register the hub relative to the fluoroscopy; the hub can contain radiopaque markers or other identifiable features that help to locate the hub relative to the fluoroscopy image. This allows the hub and fluoroscopy to be registered to each other. When these are combined into sub-registrations, a full registration between the tracking device 250 and the fluoroscopy is achieved.

[0130] Figure 11 Another method of dynamically intervening in deformation of a three-dimensional model according to a representative embodiment is illustrated.

[0131] exist Figure 11 In , the process starts at S1149 by determining the position of the tip of the tracking device with respect to (relative to) the three-dimensional model. At S1150, the process continues by creating a virtual bronchoscopic view based on the position of the tip of the tracking device.

[0132] At S1160 , the tracking device moves along the planned pathway, and the process returns to S1149 and S1150 to update the position of the tip and the virtual bronchoscope view.

[0133] Figure 11 The method may involve showing a bronchoscopic view of the tip of an interventional medical device using optical shape sensing position. Figure 11 In the method, the position of the interventional medical device is determined relative to an anatomical model, and a bronchoscopic view can be created based on the position of the tip of the interventional medical device. The position information of the interventional medical device at the location of the fixed point of the tracking device can be used to determine the closest path of the planned pathway. By simply calculating the distance between the planned pathway and the current pathway and performing an error minimization routine, the position of the tip of the interventional medical device along the planned pathway can be determined. The bronchoscopic view at this location can then be shown to the user and automatically updated as the interventional medical device moves through the airway. For example, the method implemented by the controller may include: creating a virtual bronchoscopic view based on each of a plurality of positions of the fixed point of the tracking device; determining the closest position of the planned pathway to the location of the fixed point of the tracking device; and automatically updating the bronchoscopic view as the tracking device moves through the three-dimensional model. The fixed point of the tracking device can be the tip of the tracking device.

[0134] Figure 7 As shown in Figure 11Example of a bronchoscopic view of the airway at the points marked on the 3D preoperative model acquired in . In addition to the lumen only, Figure 7 Anatomical features are shown. Figure 11 The method can provide additional imaging views of objects directly outside the airway wall to provide better guidance on when to exit the airway. For example, imaging of objects outside the airway wall can assist the interventionalist by providing visual guidance to not exit when there are large blood vessels directly outside the airway wall.

[0135] Figure 12 Another method of dynamically intervening in deformation of a three-dimensional model according to a representative embodiment is illustrated.

[0136] exist Figure 12 In , the process begins at S1222 by marking each branch of the path in the pathway as a progressive sequence through the three-dimensional model.

[0137] At S1225, a tracking system (e.g., optical shape sensing) is registered to the tracking device in order to track the tracking device. This registration may be inherent in the integration of the tracking mechanism into the tracking device, such as a tracking element being fixed to the distal tip of the tracking device. In the case of shape sensing, the hub may be fixed to the tracking device in a predetermined manner, which then allows the shape sensing fiber to be registered to the tracking device. However, Figure 12 Position tracking in is not limited to optical shape sensing, and other forms of position tracking such as sensors may also or alternatively be used.

[0138] At S1230 , the three-dimensional model is registered to the tracking device.

[0139] At S1235, Figure 12 The process in includes updating the three-dimensional model to the actual anatomy of the object.

[0140] At S1240, Figure 12 The process in includes highlighting a planned pathway through a path in a three-dimensional model and alerting the interventionalist when the tracking device moves out of the planned pathway.

[0141] At S1252, Figure 12 The process includes presenting, on a display, labels for branches of a path of a three-dimensional model proximate to the tracking device as the tracking device is navigated.

[0142] At S1280, Figure 12 The process includes deforming each of the plurality of (many) branches of a path in a three-dimensional model based on a trajectory of the tracking device when the tracking device approaches each of the plurality of (many) branches.

[0143] exist Figure 12In the method, the airway is first segmented and a planned pathway is created, as described above. Additionally, each branch in the airway model is labeled differently. The intuitive labeling scheme can be hierarchical and / or based on clinical terminology. The net effect is that, in addition to a visual sequential process, the planned pathway can be conveyed in terms of a sequence of branches. Figure 12 The method can be based on the following assumptions: the bronchoscope has a shape sensing fiber, or another position tracking mechanism embedded therein, so that the bronchoscope is fully tracked; or the shape sensing device is provided through the working channel of the bronchoscope; or the shape sensing device is provided through the working channel and a hub is used to track a non-shape sensing bronchoscope. A working hub can be used in embodiments where an optical shape sensing fiber is used, but is not necessarily required.

[0144] In addition, Figure 12 In the method, a model of the airway is registered to a shape sensing device, a bronchoscope image and / or fluoroscopy, which allows the system to transfer branch labels from the computed tomography to an image later in the procedure. Respiratory motion, as well as patient motion, is tracked via optical shape sensing or another position tracking mechanism as long as the interventional medical device is in the airway. Because the upper airway is relatively rigid, the upper airway deforms less due to physiological motion. The model of the airway can be updated to the live anatomy by initializing the position and orientation of the model for the upper airway as seen in the fluoroscopic image, and then continuously updating following the tracked respiratory / patient motion. To further assist in registration in the case of an optical shape sensing device or another position tracking device, a small device operating similarly to a dynamic hub can be placed in the trachea to provide a reference in the body regarding the position of the bronchoscope.

[0145] Using models and markings of the airway, Figure 12 The operator involved in the method navigates the bronchoscope to the next branch on the planned pathway. During navigation, a branch label appears in the bronchoscopic image as the bronchoscope approaches a branch, or in the virtual bronchoscopic image if the shape sensing device navigates through the working channel and deeper into the airway beyond the bronchoscope itself. This can indicate to the user that a navigation decision must be made soon and which branch the user should direct the bronchoscope or optical shape sensing device (or other position tracking device) toward.

[0146] When a user navigates a bronchoscope or optical shape sensing device (or other position tracking device) through Figure 12When a branch in the airway is found, the system detects that the branch has been traversed and which branch has been taken. This determination is possible because the computed tomography and segmented airways are registered to the shape sensing bronchoscope or device. This particular segmentation of the model of the airway is deformed based on the interventional medical device trajectory. Note that the position of the branch label is updated to the deformed model. In addition, the deformation is anchored by the position of the upper airway where the deformation is less severe and by the position of the branch taken. The distal segment of the airway that the device has not yet reached can be rigidly coupled to the newly deformed airway segment as an approximation to maintain a perceptible visualization of the remaining airways to the user. If the correct branch is taken based on the desired planned pathway, the pathway is highlighted to indicate the successful branch taken. Otherwise, an alert is provided. As navigation progresses, airways not related to the current pathway can be gradually de-emphasized visually while still remaining visible to provide overall context. In addition, each traversed branch can be marked in the fluoroscopic image so that the user can see the pathway / branch traversed in the live image.

[0147] The above sequence of branch traversal and pathway highlighting proceeds until the goal is reached. Figure 12 The net effect of the approach in is that deformation of the model only needs to address the planned pathway and the path taken, as opposed to the entire model, which removes visual clutter and is easier to implement; the user retains greater confidence in navigating down the correct pathway even in the presence of registration uncertainties and imperfect deformations of the model; and the accuracy of the registration can be improved based on the bronchoscope image and knowing whether the bronchoscope is in the center of the lumen or closer to the wall.

[0148] Furthermore, even without tracking, Figure 12 The bronchoscope in [ 0 ] can also be registered to the preoperative segmentation model based solely on the location / orientation of the branches in the bronchoscopic view.

[0149] Hybrid approaches to intraoperatively registering the 3D model to the anatomy are also possible, such as to provide navigation prior to registration. As the tracked bronchoscope is guided down the trachea, the bronchoscope images can be analyzed in real time to estimate the bronchoscope's path and posture when there is not yet sufficient information in the tracking system to register the model to the anatomy. The branches seen in the bronchoscope images can then be used to further refine the registration. This provides a reasonable early estimate of the bronchoscope trajectory, particularly in the rigid upper airways. Then, as branches are taken, the tracking and information of the branches and paths taken are used to refine the registration.

[0150] In one or more embodiments alternative to those already described, the pre-operative three-dimensional model may remain static and the tracked interventional medical device may instead deform based on tracheal reference, pathways traversed, branches taken, and the like.

[0151] Figure 13Illustrated is an endoluminal view of an airway from the current position of a tracking device determined using dynamic interventional three-dimensional model deformation, according to a representative embodiment.

[0152] exist Figure 13 In the endobronchial ultrasound (REBUS) procedure, a radial endobronchial ultrasound (REBUS) catheter is navigated down the right main branch and into the small peripheral airways. The ultrasound transducer produces a radial image of the airway where it is located. When in contact with the wall of the airway, the image appears white, and when there are branching airways, the image appears black because all signal is lost to the air at that location.

[0153] exist Figure 13 In an embodiment of the invention, the REBUS catheter utilizes ultrasound imaging information as a pseudo tracking mechanism instead of utilizing optical shape sensing for tracking. Figure 13 In an embodiment, the REBUS catheter is tracked using optical shape sensing along with ultrasound imaging information.

[0154] Figure 14 Another method of dynamically intervening in deformation of a three-dimensional model according to a representative embodiment is illustrated.

[0155] exist Figure 14 In , the process starts at S1430 by registering the ultrasound probe and the three-dimensional model when the branches of the path are initially obtained based on the ultrasound image.

[0156] At S1445, Figure 14 The process includes continuously acquiring ultrasound images of the object as the tracking device navigates along the path.

[0157] At S1454, Figure 14 The process involves virtually marking the current position of the ultrasound probe on the 3D model as the ultrasound probe is navigated, and continuously acquiring ultrasound images and aligning each branch of the path to the 3D model. For example, the diameter of the airway can be compared to the 3D model to provide a rough context for which branch the tracking device is in. In addition, Figure 14 In the embodiment of FIG, ultrasound itself is the tracking method. However, an interventional tool such as an ultrasound catheter may also be tracked using another type of tracking technology such as optical shape sensing or electromagnetic.

[0158] In the above described Figure 14 In the method of , radial endobronchial ultrasound (REBUS) is used to assess the airway and guide needle biopsy and to guide navigation and determine the position of the transducer relative to the three-dimensional segmentation model. The method for navigation using (R)EBUS guidance and tracking is described in part by Figure 4 Description. The (R)EBUS probe is navigated in the left or right main branch, and ultrasound images are continuously acquired from the (R)EBUS probe. Figure 14The ultrasound imaging in the image can be assumed to be of good image quality and that there are no air pockets between the transducer and the airway wall. When the first branch is seen in the ultrasound image, the initial registration between the probe and the 3D segmentation model is established. The first branch is Figure 13 A virtual marker can be placed on the 3D model to indicate the current position of the transducer. As the probe is navigated further into the airway, ultrasound images are continuously acquired, and each branch is registered to the preoperative computed tomography or segmentation 3D model. Furthermore, each time a branch is visualized, the virtual marker position is updated on the 3D model; if a branch is not visible within the ultrasound image, the virtual marker position can be estimated with some indication of uncertainty about its position.

[0159] The automatic marking described in the previous embodiment can be combined with Figure 4 For example, the diameter and wall thickness of the airway where the EBUS transducer is currently located can be calculated and returned to the reference 3D model. The 3D model generated from the 3D imaging can also include information about the airway diameter and airway wall thickness at all locations. Registering this information between the EBUS probe and the 3D model determines the approximate position of the transducer. Adding additional information about which branches have been passed can be used to determine the exact position of the transducer. In addition, although Figure 14 The description of most embodiments herein focuses on the airway, but examples such as Figure 14 Methods of

[00145] including updated methods using automatic labeling can also be used in vascular navigation.

[0160] The method implemented by the controller may include automatically determining a position of a tip of an ultrasound probe used to acquire the ultrasound image by calculating a diameter and a wall thickness of a path from the ultrasound image and comparing the diameter and the wall thickness to a three-dimensional model. The controller may also optimize the position of the tip of the ultrasound probe based on a previous position of the ultrasound probe.

[0161] exist Figure 14 In the 3D segmentation model, the position and orientation of the transducer can be further refined by measuring the diameter of the branch airways and the orientation of the airways relative to the transducer in the image. Keeping track of each branch the transducer passes through can provide a better assessment of which branch and pathway the entire device has descended. A virtual image of the entire (R)EBUS probe can then be shown on the 3D segmentation model.

[0162] In addition, Figure 14 In

[0015] , a virtual 3D ultrasound image or roadmap of the airway can be constructed based on the tracking (R)EBUS. This can be displayed on the preoperative segmentation model. Furthermore, the preoperative model can be updated to match the virtual (R)EBUS roadmap, for example based on the deformable correction technique described previously.

[0163] Intermittent X-ray images can also be used to update the position of virtual markers or to check for Figure 14 Similarly, tracking such as optical shape sensing can be merged with ultrasound-based methods to reduce uncertainty, particularly in branch locations where image quality is degraded.

[0164] Thus, dynamic interventional 3D model deformation enables dynamic navigation correction during interventional procedures. Dynamic interventional 3D model deformation is applicable to multiple different imaging modes such as X-ray and ultrasound, as well as multiple different organs of human subjects, including the lungs and the vascular system. Dynamic interventional 3D model deformation can be added as a feature to existing products, such as device tracking products. The primary application discussed for lung biopsy is not an absolute requirement, as dynamic interventional 3D model deformation can also be used for other pulmonology applications, such as surgical resection or ablation of tissue. In addition, dynamic interventional 3D model deformation can be applicable to other areas, such as blood vessels or the gastrointestinal tract. In addition, dynamic interventional 3D model deformation is applicable to both Rayleigh (enhanced and regular) and fiber Bragg implementations of shape sensing optical fibers, as well as both manual and robotic manipulation of such devices. Finally, in addition to optical shape sensing implementations, dynamic interventional 3D model deformation is applicable to both X-ray-based tracking of devices and ultrasound tracking of devices.

[0165] Although the dynamic intervention 3D model deformation has been described with reference to several exemplary embodiments, it should be understood that the words used are words of description and illustration rather than limitation. Changes may be made within the scope of the claims, as presently described and amended, without departing from the scope and spirit of the dynamic intervention 3D model deformation enabling technology in its various aspects. Although the dynamic intervention 3D model deformation has been described with reference to specific modules, materials, and embodiments, the dynamic intervention 3D model deformation enabling technology is not intended to be limited to the details disclosed; rather, the dynamic intervention 3D model deformation enabling technology extends to all functionally equivalent structures, methods, and uses, such as those within the scope of the claims.

[0166] The illustrations of the embodiments described herein are intended to provide an overall understanding of the structures of the various embodiments. These illustrations are not intended to be used as a complete description of all elements and features of the disclosure described herein. After reviewing this disclosure, many other embodiments will be apparent to those skilled in the art. Other embodiments may be utilized and derived from this disclosure so that structural and logical replacements and changes can be made without departing from the scope of this disclosure. In addition, the illustrations are merely representative and may not be drawn to scale. Certain proportions within the illustrations may be exaggerated, while other proportions may be minimized. Therefore, this disclosure and the accompanying drawings should be considered illustrative and not restrictive.

[0167] It is for convenience only that one or more embodiments of the present disclosure are referred to herein individually and / or collectively by the term "invention" and are not intended to voluntarily limit the scope of this application to any particular invention or inventive concept. In addition, although specific embodiments have been illustrated and described herein, it should be appreciated that any subsequent arrangement designed to achieve the same or similar purpose may replace the specific embodiments shown. This disclosure is intended to cover any and all subsequent modifications or variations of the various embodiments. Combinations of the above-described embodiments, as well as other embodiments not specifically described herein, will be apparent to those skilled in the art upon review of this specification.

[0168] The Abstract of the present disclosure is provided to comply with 37 CFR §1.72(b) and is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. In addition, in the foregoing detailed description, various features may be grouped together or described in a single embodiment for the purpose of simplifying the present disclosure. This disclosure should not be interpreted as reflecting an intention that the claimed embodiments require more features than expressly recited in each claim. Rather, as reflected in the following claims, inventive subject matter may involve fewer than all the features of any disclosed embodiment. Accordingly, the following claims are incorporated into the detailed description, with each claim standing on its own as defining separately claimed subject matter.

[0169] The foregoing description of the disclosed embodiments is provided to enable those skilled in the art to practice the concepts described in this disclosure. As such, the above-disclosed subject matter is to be considered illustrative and not restrictive, and the appended claims are intended to cover all such modifications, enhancements, and other embodiments that fall within the true spirit and scope of the disclosure. Therefore, to the maximum extent allowed by law, the scope of the present disclosure is to be determined by the broadest permissible interpretation of the claims and their equivalents, and shall not be restricted or limited by the foregoing detailed description.

Claims

1. A controller for assisting navigation in an interventional procedure, comprising: a memory (320 / 330 / 382) that stores instructions, and a processor (310) that executes the instructions, When executed by the processor (310), the instructions cause the controller to implement a process comprising the following operations: Obtaining ( S410 ) a three-dimensional model generated before the intervention procedure based on segmenting a path having a plurality of branches in an object of the intervention procedure; registering (S630) a tracking device (250) in the path to the three-dimensional model; Calculating (S675) an offset between immediately previous position information of the tracking device (250) relative to the three-dimensional model and current position information of the tracking device (250) relative to the three-dimensional model; transforming (S680) the three-dimensional model to the current position information of the tracking device (250) based on the offset by adjusting the three-dimensional model so that the three-dimensional model includes one or more immediately previous positions of the tracking device and the current position of the tracking device, and Each new branch of the three-dimensional model containing the tracking device (250) is iteratively and locally deformed each time the tracking device (250) moves to a new branch (S690).

2. The controller according to claim 1, in, When executed by the processor (310), the instructions cause the controller to implement a process further comprising: determining (S470) whether current position information of the tracking device (250) is outside the path in the three-dimensional model during the intervention procedure; and When the current position information of the tracking device (250) is outside the path in the three-dimensional model, the three-dimensional model is deformed (S480) to the current position information of the tracking device (250).

3. The controller according to claim 2, wherein: The three-dimensional model has been segmented to form a mesh of the object in a coordinate system in which the optical shape sensing of the tracking device is registered, the process further comprising, for each point (i) of the mesh: a) calculating a distance between a point (i) of the grid and a point (j: j+N) of the optical shape sensing device sensed by the optical shape sensing along the tracking device; b) determining, through distance minimization calculation, an optical shape sensing device point among the optical shape sensing device points (j:j+N) that is closest to the point (i) of the grid; c) adding said calculated offset of the optical shape sensing device point to point (i) of the grid; d) storing the coordinates of point (i) of the new grid; e) Repeat (a) to (d) for all mesh points to finally obtain the deformed three-dimensional model.

4. The controller according to claim 1, wherein: The process implemented when the processor (310) executes the instructions further includes: registering (S830) the tracking device (250) in the path to the three-dimensional model; and When the current position information of the tracking device (250) is outside the path in the three-dimensional model, deforming the three-dimensional model includes deforming only the local branch of the path containing the tracking device (250) or the two local branches of the path closest to the tracking device (250) (S870), wherein the pathway passes through a lung, a vascular system, or a gastrointestinal system, and the tracking device (250) is navigated to a location in the lung, the vascular system, or the gastrointestinal system prior to the interventional procedure.

5. The controller according to claim 1, wherein The process implemented when the processor (310) executes the instructions further includes: When the current position information of the tracking device (250) is outside the path in the three-dimensional model, deforming the three-dimensional model includes deforming only a local branch in the path containing the tracking device (250) or two local branches in the path closest to the tracking device (250) (S870), wherein the path passes through the lungs, the vascular system, or the gastrointestinal system; deforming (S980) the three-dimensional model to the current position information of the tracking device (250) to correct lung motion, vascular system motion or gastrointestinal system motion or lung deformation, vascular system deformation or gastrointestinal system deformation, and The tracking device (250) is continuously tracked (S950) while the lung, the vascular system, or the gastrointestinal system is manipulated.

6. The controller according to claim 1, wherein: The process implemented when the processor (310) executes the instructions further includes: Tracking (S1050) the tracking device (250) in two dimensions based on X-ray imaging; and The three-dimensional model is registered (S1030) to the two-dimensional X-ray image space.

7. The controller according to claim 6, wherein: The process implemented when the processor (310) executes the instructions further includes: projecting (S1055) the three-dimensional model to overlay the three-dimensional model onto the two-dimensional X-ray image space as the tracking device (250) is navigated through the path under the guidance of the tracking based on X-ray imaging; and The position of the tracking device (250) in the fluoroscopic image is identified (S1050) based on the X-ray imaging.

8. The controller according to claim 1, wherein: The process implemented when the processor (310) executes the instructions further includes: creating (S1150) a virtual bronchoscopic view based on the position of the fixed point of the tracking device (250); determining (S1149) the closest position of the planned path to the position of the fixed point of the tracking device (250); and The bronchoscopic view is automatically updated (S1150) as the tracking device (250) is moved through the three-dimensional model.

9. The controller according to claim 1, wherein: The process implemented when the processor (310) executes the instructions further includes: labeling ( S1222 ) each branch of a pathway in the path as a progressive sequence through the three-dimensional model; registering (S1230) the three-dimensional model to the tracking device (250); updating (S1235) the three-dimensional model to the subject's actual anatomy; and When the tracking device (250) is navigated, labels for branches in the path of the three-dimensional model that are proximal to the tracking device (250) are presented (S1252) on the display (225).

10. The controller according to claim 1, wherein: The process implemented when the processor (310) executes the instructions further includes: When the tracking device (250) approaches each of the multiple branches of the path in the three-dimensional model, each of the multiple branches is deformed based on the trajectory of the tracking device (250) (S1280).

11. The controller according to claim 1 , further comprising: A planned path through the path in the three-dimensional model is highlighted (S1240) and a user is alerted when the tracking device (250) moves out of the planned path.

12. The controller according to claim 1, wherein: The process implemented when the processor (310) executes the instructions further includes: continuously acquiring (S1445) ultrasound images of the path as the tracking device (250) is navigated on the path; registering ( S1430 ) an ultrasound probe with the three-dimensional model when initially obtaining the branches of the path based on the ultrasound image; As the ultrasound probe is navigated, the current position of the ultrasound probe on the three-dimensional model is virtually marked ( S1454 ), and ultrasound images are continuously acquired and each branch of the path is aligned to the three-dimensional model.

13. The controller according to claim 1, wherein: The process implemented when the processor (310) executes the instructions further includes: continuously acquiring (S1445) ultrasound images of the path as the tracking device (250) is navigated on the path; and The three-dimensional model is virtually marked (S1454) each time a branch of the path is visualized.

14. The controller according to claim 13, wherein: The process implemented when the processor (310) executes the instructions further includes: automatically determining (S1149) a position of a tip of an ultrasound probe used to acquire the ultrasound image by calculating a diameter and a wall thickness of a path from the ultrasound image and comparing the diameter and the wall thickness with the three-dimensional model; and The position of the tip of the ultrasound probe is optimized based on a previous position of the ultrasound probe.

15. A system (200) for assisting navigation in an interventional procedure, comprising: an imaging device that generates a three-dimensional anatomical image of a subject of the interventional procedure before the interventional procedure for generating a three-dimensional model based on segmenting a path having a plurality of branches in the subject of the interventional procedure before the interventional procedure; a computer (220) having a memory for storing instructions and a processor (310) for executing the instructions; Wherein, when executed by the processor (310), the instructions cause the system (200) to perform a process comprising the following operations: registering (S630) a tracking device (250) in the path to the three-dimensional model; Calculating (S675) an offset between immediately previous position information of the tracking device (250) relative to the three-dimensional model and current position information of the tracking device (250) relative to the three-dimensional model; transforming (S680) the three-dimensional model to the current position information of the tracking device (250) based on the offset by adjusting the three-dimensional model so that the three-dimensional model includes one or more immediately previous positions of the tracking device and the current position of the tracking device, and Each new branch of the three-dimensional model containing the tracking device (250) is iteratively and locally deformed each time the tracking device (250) moves to a new branch (S690).

Images