Dynamic Deformation Tracking for Navigation Bronchoscopy

Abstract

A system and method for tracking the movement of a catheter within a lung's airways. The deformable lung model represents the lung's airways as airway segments joined at bifurcation points. The lung model deformation uses modifications to the angle and / or position of the airway segments relative to each other. The initial model may be generated, for example, based on segmentation of CT images. A baseline deformable alignment of the initial model to the patient's lung shape at the start of treatment may be established from catheter position measurements along multiple different paths. Optionally, the baseline alignment is dynamic according to respiratory phase. Compared to the baseline alignment, real-time changes in lung shape may be modeled by using additional measurements acquired using the catheter as it navigates to the target, preferably using position sensors distributed along the catheter body.

Description

[Technical Field]

[0001] Related Applications This application claims the benefit of priority under 35 U.S.C. §119(e) to U.S. Provisional Patent Application No. 63 / 123,590, filed December 10, 2020, entitled "DEFORMABLE REGISTRATION AND DYNAMIC DEFORMATION TRACKING AND VIEWS FOR NAVIGATIONAL BRONCHOSCOPY," and to U.S. Provisional Patent Application No. 63 / 238,165, filed August 29, 2021, entitled "DEFORMATION TRACKING," the contents of each of which are incorporated herein by reference in their entirety. [Background technology]

[0002] The present invention, in some embodiments, relates to the field of bronchoscopy, and more particularly, but not exclusively, to electromagnetic navigation bronchoscopy.

[0003] Systems have been proposed and / or marketed for navigating a probe moving through the lungs, including non-robotic navigation guided by single-sensor electromagnetic sensing using fluoroscopy, robotic navigation guided by fiber optic morphometry, and robotic navigation guided by single-sensor electromagnetic sensing. Other systems known in the art include fluoroscopy-based systems, CBT (cone-beam CT) guided systems, and video registration-based bronchoscopy. Summary of the Invention

[0004] According to an aspect of some embodiments of the present disclosure, there is provided a method for tracking positioning of an interventional instrument relative to an airway of a lung, the method including: accessing a skeletonized model of the airway of the lung, the model including a plurality of airway segments connected at bifurcation points; accessing anatomically defined constraints that limit changes to a shape of the skeletonized model; accessing measurements indicative of a plurality of positions within the lung along the interventional instrument; modifying within the model relative angles and positions of the airway segments at one or more of the bifurcation points, the modifications being calculated to reduce a modeled error between the shape of the lung indicated by the plurality of positions and the placement of the airway segments corresponding to the plurality of positions; and using anatomically based constraints to limit how the shape of the airway may change to reduce the modeled error.

[0005] According to some embodiments of the present disclosure, modifying includes modifying the relative angles of airway segments joined at a common bifurcation point, where the common bifurcation point is not traversed by an interventional instrument extending between the multiple locations.

[0006] According to some embodiments of the present disclosure, the anatomically based constraints include the resistance of the airways at the common bifurcation point to an increase in change of their relative angle.

[0007] According to some embodiments of the present disclosure, the anatomically based constraint includes a fixation force applied distally to at least a portion of the airway segment.

[0008] According to some embodiments of the present disclosure, the anatomically based constraint includes a limit on bending along the length of at least a portion of the airway segment.

[0009] According to some embodiments of the present disclosure, the anatomically based constraints include limitations on the separate movement of airway segments that are connected to one another through the lung parenchyma.

[0010] According to some embodiments of the present disclosure, the anatomically based constraints include restrictions on airway segments that form airway shapes that do not match a range of global shapes represented by a dataset including multiple global shapes.

[0011] According to some embodiments of the present disclosure, testing the airway segment shapes for consistency with a global shape range includes applying a machine learning product configured to score the airway segment shapes according to their likelihood of falling within the global shape range.

[0012] According to some embodiments of the present disclosure, the global shape extent is derived from a plurality of measured lung measurements.

[0013] According to some embodiments of the present disclosure, the global shape bounds are derived from a modified version of the skeletonized model that is pre-computed and accessed as part of the anatomically defined constraints.

[0014] According to some embodiments of the present disclosure, the anatomically based constraints include restrictions on airway segments representing lung portions along multiple locations that form branching shapes that do not match the range of branching shapes represented by a dataset including multiple branching shapes.

[0015] According to some embodiments of the present disclosure, testing the airway segment branching shape for consistency with a range of branching shapes includes applying a machine learning product configured to score the airway segment branching shape according to its likelihood of falling within the range of branching shapes.

[0016] According to some embodiments of the present disclosure, a range of branching shapes is derived from a plurality of measured lung measurements.

[0017] According to some embodiments of the present disclosure, the global shape bounds are derived from a modified version of the skeletonized model that is pre-computed and accessed as part of the anatomically defined constraints.

[0018] According to some embodiments of the present disclosure, a method includes constructing a skeletonized model based on a segmentation of a CT image of the lung.

[0019] According to some embodiments of the present disclosure, constructing includes skeletonizing a segmentation of the CT image, identifying bifurcation points in the skeletonization, and assigning parameter values ​​to the bifurcation points that represent relative angles of the bifurcation points.

[0020] According to some embodiments of the present disclosure, the accessed measurements are measurements made along multiple different airway tracks using an interventional instrument at different times and at different positions of the interventional instrument, and relative angles of the airway segments are assigned to bifurcation points to match the imaged geometry of the lungs, and modifying modifies the relative angles of the airway segments according to changes in lung position between the time the lung geometry was imaged and the time of measurement along the multiple different airway tracks.

[0021] According to some embodiments of the present disclosure, the accessed measurements are associated with respiratory phases by their measurement times, and the modifying modifies the relative angles to create a phase-varying skeletonized model of the lung airways representing the shape of the lungs during multiple different respiratory phases based on the respiratory phase association of the accessed measurements.

[0022] According to some embodiments of the present disclosure, the modifying is performed for new measurements associated with a respiratory phase, modifying the phase-varying skeletonized model of the airway according to both the location of the new measurement and its associated respiratory phase.

[0023] According to some embodiments of the present disclosure, the measurements are used in modifying the relative angles of the airway segments for different modeled respiratory phases according to a weighting that depends on the difference between the respiratory phase associated with each measurement and the different modeled respiratory phases.

[0024] According to some embodiments of the present disclosure, the shape of the lungs during multiple different respiratory phases is represented by a lung state at a first respiratory phase calculated by correction from measurements associated with the phases, a lung state at a second respiratory phase calculated by correction from measurements associated with the phases, and a lung state at at least a third respiratory phase calculated by interpolation of the lung shapes between the first and second phases.

[0025] According to some embodiments of the present disclosure, the shape of the lungs during multiple different phases is further represented by the lungs in a fourth respiratory phase.

[0026] According to some embodiments of the present disclosure, the third and fourth respiratory phases are both equally spaced between the first and second respiratory phases and are distinct from one another.

[0027] According to some embodiments of the present disclosure, the phase of respiration is determined based on the movement of markers attached to the external surface of the body, including the lungs and their airways.

[0028] According to some embodiments of the present disclosure, a method includes displaying an image representing the arrangement of airway segments of a model.

[0029] According to some embodiments of the present disclosure, a method includes displaying an object in an image associated with a lung model, the position of the object being updated according to changes in the placement of the airway segments.

[0030] According to some embodiments of the present disclosure, a method includes associating a position of an object with a position of at least one of the airway segments and moving the object according to the associated movement of the at least one of the airway segments.

[0031] According to some embodiments of the present disclosure, the images also represent slices of the 3D image of the lung in a known correspondence relationship to a skeletonized model of the lung's airways, the slices being selected to extend along the portion of the displayed lung that corresponds to the airway along which the interventional instrument will extend.

[0032] According to some embodiments of the present disclosure, a known correspondence between a skeletonized model of the lung airways and a 3D image of the lung is established by deriving the skeletonized model of the airways from a segmentation of the 3D image of the lung.

[0033] According to some embodiments of the present disclosure, the displayed image includes a 3D representation of the lung, and slices of the 3D image are curved out of a planar configuration to follow the 3D configuration of the interventional instrument.

[0034] According to some embodiments of the present disclosure, the displayed image represents the interventional instrument flattened into a planar representation, with the slices of the 3D image flattened along with it.

[0035] According to some embodiments of the present disclosure, the relative angles of the airway segments are assigned to branch points to match the geometry of the lungs imaged during a first period and are corrected by changes to the lung geometry determined based on further measurements taken during a second period, the measurements accessed being measurements taken along the interventional instrument, during a third period, and while the interventional instrument remains in the same position, and the correcting modifies the relative velocities of the airway segments according to changes in the position of the lungs between the second and third periods.

[0036] According to some embodiments of the present disclosure, the method includes repeatedly accessing successive sets of measurements and modifying the relative angles of the airway segments based on each successive set of measurements.

[0037] According to some embodiments of the present disclosure, the correcting includes identifying a new value of the relative angle that reduces an error between the shape of the lungs indicated by the multiple positions and the position of the airway segments corresponding to the multiple positions, filtering the new value based on previous values ​​of the relative angle to reduce the magnitude of change from the previous value, and using the new value to generate the calculated correction.

[0038] According to some embodiments of the present disclosure, filtering increases the error between the lung shape indicated by the multiple locations and the position of the airway segments corresponding to the multiple locations compared to the new values ​​identified in the pre-filtering.

[0039] According to some embodiments of the present disclosure, the error is reduced, but not entirely, by correcting between each successive set of measurements.

[0040] According to some embodiments of the present disclosure, the correcting includes correcting the relative angles differently in multiple different copies of the skeletonized model, repeating accessing the measurements and correcting for each of the different copies using the new measurements, selecting one of the different copies based on a more plausible representation of the lung shape, and continuing to repeat accessing the measurements and correcting using only the selected copy.

[0041] According to some embodiments of the present disclosure, correcting includes perturbing the relative angles of the airway segments in the correction to calculate a perturbed correction, determining that the post-perturbation correction reduces the error compared to the pre-perturbation correction, and performing the correction using the perturbed correction.

[0042] According to one aspect of some embodiments of the present disclosure, a computer memory storage medium is provided that stores a deformable model of a lung, the model including data elements corresponding to a skeletonized representation of a branching structure of an airway of the lung and, for each of a plurality of branching points of the branching structure, a representation of the branching orientation of the branching structure, and computer instructions configured to instruct a processor to modify the model to satisfy provided geometric constraints by modifying the representation of the relative orientation of the branches of the branching structure.

[0043] According to some embodiments of the present disclosure, the plurality of branching points includes branching points up to a third level of branching of the branching structure.

[0044] According to some embodiments of the present disclosure, computer instructions instruct a processor to modify a representation of branch orientations of a branching structure without modifying distances along segments of the branching structure.

[0045] According to some embodiments of the present disclosure, computer instructions instruct a processor to propagate changes that modify the relative orientation of branches at a parent branch point to changes that modify the spatial offset and orientation of branches at child branch points of the parent branch point.

[0046] According to one aspect of some embodiments of the present disclosure, there is provided a system for tracking movement of an interventional instrument within an airway of a lung, the system including a processor and a memory holding instructions instructing the processor to access a 3D representation of a current shape and positioning of the interventional instrument, access a model of the airway representing the airway segment including bifurcation points of the airway segment, and match the airway segment to the current shape and positioning of the interventional instrument, wherein the processor is instructed to determine a rotational correction of the bifurcation of the bifurcation points based on an improvement in the correspondence of the airway segment to the current shape and positioning of the interventional instrument, and apply the correction to the model.

[0047] According to some embodiments of the present disclosure, the rotational correction is applied to the modeled airway segment corresponding to the section of the airway along which the interventional instrument extends.

[0048] According to some embodiments of the present disclosure, a processor is instructed to access measurements of body motion, and the processor is instructed to modify the orientation of airway segments to modify the model of the airway to match changes in lung shape corresponding to the measurements of body motion.

[0049] According to some embodiments of the present disclosure, the corrections are applied to the model using position data associated with bifurcation points as control points.

[0050] According to some embodiments of the present disclosure, different types of rotational corrections are weighted differently in their availability to improve the airway model's match to changes in lung shape corresponding to measurements of body motion.

[0051] According to some embodiments of the present disclosure, different types of rotational modifications are weighted differently in their availability to improve commensurability with the current shape and positioning of the interventional device in the airway segment.

[0052] According to some embodiments of the present disclosure, longitudinal bending rotation corrections are weighted as being more available to improve commensurability than longitudinal torsional rotation corrections.

[0053] According to an aspect of some embodiments of the present disclosure, there is provided a method for aligning position data of an interventional instrument to an imaged lung shape, the method including receiving a baseline pulmonary airway model constructed based on the imaged lung shape of the lung; thereafter, using one or more intrapulmonary probes, measuring positions along multiple branches of the airways of the lung, including at least two different branches of a same branching point of the airway; modifying the baseline pulmonary airway model to align with the measured positions and generate a deformed baseline pulmonary airway model, wherein the deformed baseline pulmonary airway model represents the pulmonary airways in multiple respiratory phases; determining a transformation applicable to the deformed baseline pulmonary airway model based on measurements indicative of a current shape of the lung, to adjust the deformed baseline pulmonary airway model to represent the current shape of the lung; and displaying an image indicative of the deformed baseline airway model transformed according to the transformation.

[0054] According to some embodiments of the present disclosure, measurements of positions along multiple branches of the lung airways are each associated with a respiratory phase, and representations of the lung airways within different phases of the multiple respiratory phases are generated using the associations of the measurements with the different respiratory phases.

[0055] According to some embodiments of the present disclosure, determining the transformation includes determining a respiratory phase-dependent shape of the deformed baseline lung airway model using a respiratory phase corresponding to the current represented shape of the lung, and calculating a transformation to convert the determined respiratory phase-dependent shape to the current represented shape of the lung.

[0056] According to some embodiments of the present disclosure, in a modified baseline lung airway model, the shape of the lung during multiple respiratory phases is represented by a lung state at a first respiratory phase generated by modifying the baseline lung airway model and a lung state at a second respiratory phase generated by modifying the baseline lung airway model.

[0057] According to some embodiments of the present disclosure, determining includes generating a baseline shape of the lung at the third respiratory phase calculated by interpolating the lung shape between the first phase and the second phase from the deformed baseline airway model.

[0058] According to some embodiments of the present disclosure, the shapes of the lungs during multiple respiratory phases further include a state of the lungs at a fourth respiratory phase, which is generated by modifying the baseline lung airway model.

[0059] According to some embodiments of the present disclosure, the third and fourth respiratory phases are similarly spaced in phase between the first and second respiratory phases, respectively, and differ from each other.

[0060] According to one aspect of some embodiments of the present disclosure, there is provided a method for displaying features along a winding path through an anatomical structure, the method comprising: accessing a 3D image of the anatomical structure; defining a path through the anatomical structure; calculating a spatial correspondence of positions along the path with locations in the 3D image and a surface extending through the 3D image including the locations in the 3D image; and displaying a 3D display image showing the path extending through 3D space together with data from the locations in the 3D image of the anatomical structure defined by the surface, wherein the path is iteratively defined using measurements of at least one dynamically changing parameter of the shape of the anatomical structure, and the calculating and displaying are repeated each time the path is iteratively defined.

[0061] According to some embodiments of the present disclosure, the surface is calculated as a strip extending along the path on either side of the route.

[0062] According to some embodiments of the present disclosure, the strips extend approximately equal distances on either side of the path.

[0063] According to some embodiments of the present disclosure, a method includes accessing a 3D model of a portion of an anatomical structure that is in known spatial correspondence with the path, and including a representation of the 3D model of the anatomical structure within a 3D display image.

[0064] According to some embodiments of the present disclosure, the 3D model of the portion of the anatomy also changes dynamically.

[0065] According to some embodiments of the present disclosure, the anatomical structure includes an airway of the lung.

[0066] According to some embodiments of the present disclosure, the pathway extends through an airway of the lung.

[0067] According to some embodiments of the present disclosure, the measurements of the at least one dynamically changing parameter include position measurements of an interventional instrument positioned along the path.

[0068] According to some embodiments of the present disclosure, the position measurements of an interventional instrument positioned along a path include position measurements from multiple positions along the path.

[0069] The measurement of the at least one dynamically changing parameter is related to the phase of the breath by the time of the measurement.

[0070] Unless otherwise defined, all technical and / or scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. Although methods and materials similar or equivalent to those described herein can be used in the practice or testing of embodiments of the present disclosure, exemplary methods and / or materials are described below. In case of conflict, the patent specification, including definitions, will control. Furthermore, the materials, methods, and examples are illustrative only and are not intended to be necessarily limiting.

[0071] As will be appreciated by those skilled in the art, aspects of the present disclosure may be embodied as a system, method, or computer program product. Accordingly, aspects of the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects (e.g., a method may be implemented using "computer circuitry"), which may be generally referred to herein as a "circuit," "module," or "system." Furthermore, some embodiments of the present disclosure may take the form of a computer program product embodied in one or more computer-readable medium(s) having computer-readable program code embodied therein. Implementation of the methods and / or systems of some embodiments of the present disclosure may involve performing and / or accomplishing selected tasks manually, automatically, or a combination thereof. Furthermore, depending on the actual instrumentation and implementation of some embodiments of the methods and / or systems of the present disclosure, some selected tasks may be implemented by hardware, software, or firmware, and / or a combination thereof, for example, using an operating system.

[0072] For example, hardware for performing selected tasks according to some embodiments of the present disclosure may be implemented as a chip or circuit. As software, selected tasks according to some embodiments of the present disclosure may be implemented as multiple software instructions executed by a computer using any suitable operating system. In some embodiments of the present disclosure, one or more tasks performed in a method and / or by a system are performed by a data processor (also referred to herein as a "data processor," referring to a data processor that operates manually using groups of digital bits), such as a computing platform for executing multiple instructions. Optionally, the data processor includes volatile memory for storing instructions and / or data, and / or non-volatile storage, e.g., a magnetic hard disk and / or removable media, for storing instructions and / or data. Optionally, a network connection is also provided. A display and / or user input device, such as a keyboard or mouse, are also optionally provided. Any of these implementations are more generally referred to herein as instances of computer circuitry.

[0073] Some embodiments of the present disclosure may utilize any combination of one or more computer-readable medium(s). The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media would include an electrical connection having one or more communication lines, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable PROM (EPROM or flash memory), an optical fiber, a compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer-readable storage medium may be any tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. A computer-readable storage medium may also contain or store information for use by such programs, e.g., data structured to be recorded by the computer-readable storage medium so that the computer program can access it, for example, as one or more tables, lists, arrays, data trees, and / or other data structures. Herein, a computer-readable storage medium that records data in a retrievable form as groups of digital bits is also referred to as a digital memory. In the case of a computer-readable storage medium that is not inherently read-only and / or is in a read-only state, it should be understood that the computer-readable storage medium, in some embodiments, is also optionally used as a computer-writable storage medium.

[0074] As used herein, a data processor is said to be "configured" to perform data processing operations to the extent that it is coupled to a computer-readable medium to receive instructions and / or data therefrom, process them, and / or store the results of the processing in the same or another computer-readable medium. The operations performed (optionally on data) are specified by instructions, with the effect that the processor acts in accordance with the instructions. The act of processing may additionally or alternatively be referred to by one or more other terms, such as comparing, estimating, determining, calculating, identifying, associating, storing, analyzing, selecting, and / or transforming. For example, in some embodiments, a digital processor receives instructions and data from a digital memory, processes the data in accordance with the instructions, and / or stores the results of the processing in the digital memory. In some embodiments, "providing" the results of the processing includes one or more of transmitting, storing, and / or presenting the results of the processing. Presenting optionally includes showing on a display, sounding, printing on a printout, or otherwise providing the results in a form accessible to human sensory functions.

[0075] A computer-readable signal medium may include a propagated data signal in which computer-readable program code is embodied, for example, in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including but not limited to, electromagnetic, optical, or any combination thereof. A computer-readable signal medium is not a computer-readable storage medium but may be any computer-readable medium that can communicate, propagate, or carry a program for use by or in connection with an instruction execution system, apparatus, or device.

[0076] The program code embodied on the computer-readable storage medium and / or data used thereby may be transmitted using any suitable medium, including but not limited to wireless, wired, fiber optic cable, RF, etc., or any suitable combination of the above.

[0077] Computer program code for carrying out operations for some embodiments of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" programming language or similar programming languages. The program code may run entirely on the user's computer, partially on the user's computer as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet Service Provider).

[0078] Some embodiments of the present disclosure are described below with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus to create a machine such that the instructions, executed by the processor of the computer or other programmable data processing apparatus, create means for performing the function(s) / acts specified in the block or blocks of the flowchart illustrations and / or block diagrams.

[0079] These computer program instructions may also be stored on a computer-readable medium that can instruct a computer, other programmable data processing apparatus, or other device to function in a particular manner, whereby the instructions stored on the computer-readable medium produce an article of manufacture that includes instructions that implement the functions / acts specified in the flowcharts and / or block diagrams of a block or blocks.

[0080] Computer program instructions may be loaded onto a computer, other programmable data processing apparatus, or other device to create a computer-implemented process such that the instructions executing on the computer or other programmable apparatus cause a series of operational steps to be executed by the computer, other programmable apparatus, or other device to provide a process for performing the functions / acts specified in the flowcharts and / or block diagrams of a block or blocks.

[0081] Some embodiments of the present disclosure are herein described, by way of example only, with reference to the accompanying drawings. Specific reference will now be made in detail to the drawings, it being emphasized that the details shown are for the purpose of illustrating and discussing embodiments of the present disclosure by way of example. In this regard, the description using the drawings will make apparent to those skilled in the art how embodiments of the present disclosure may be practiced. [Brief explanation of the drawings]

[0082] [Figure 1] 1 is a schematic flowchart illustrating a method for generating a deformed model of a lung, according to some embodiments of the present disclosure. [Figure 2] 1A-1C are schematic representations of lung airway skeletonization, according to some embodiments of the present disclosure; [Figure 3] 1A and 1B illustrate schematic representations of a bifurcation coordinate system (also referred to as coordinate system Ti) and a bifurcation point plane, according to some embodiments of the present disclosure. [Figure 4]1A-1C are schematic illustrations of a lung airway skeleton before (left) and after (right) deformation between bifurcation points, according to some embodiments of the present disclosure. [Figure 5] 1 is a schematic flowchart illustrating a method for performing deformable registration between the position of a catheter placed in a lung airway and a bifurcation model of the airway, according to some embodiments of the present disclosure. [Figure 6] 10A-10C are schematic representations of lung models differently registered with survey data, according to some embodiments of the present disclosure; [Figure 7A] 10A-10C schematically illustrate elements of determining a deformation transformation for a deformation model to match available data and using the deformation transformation in further dynamic modeling, according to some embodiments of the present disclosure. [Figure 7B] 7B is a flowchart that outlines an embodiment of the block of FIG. 7A, in accordance with some embodiments of the present disclosure. [Figure 8A] 1 is a schematic flow chart outlining a method for generating a deformable lung model view, according to some embodiments of the present disclosure. [Figure 8B] 10A-10C are schematic representations of pre- and post-deformation D-meshes renderings of a branch point and its descendants of a single model, according to some embodiments of the present invention; [Figure 8C] 10A-10C are schematic representations of pre- and post-deformation D-meshes renderings of a branch point and its descendants of a single model, according to some embodiments of the present invention; [Figure 9A] 9A and 9B illustrate schematic diagrams of an x-slice interpolation grid of a single branch point and all its descendants before (FIG. 9A) and after (FIG. 9B) deformation, according to some embodiments of the present disclosure. [Figure 9B] 9A and 9B illustrate schematic diagrams of an x-slice interpolation grid of a single branch point and all its descendants before (FIG. 9A) and after (FIG. 9B) deformation, according to some embodiments of the present disclosure. [Figure 9C] 1A-1C schematically illustrate CT slices extending along a geometric cross section defined to be curved to include regions of the original CT image corresponding to positions along a catheter path, according to some embodiments of the present invention. [Figure 10A]10A and 10B show schematic diagrams of CT strips along a corridor in a composite D scene, including rendered mesh views before (FIG. 10A) and after (FIG. 10B) deformation according to some embodiments of the present disclosure. [Figure 10B] 10A and 10B show schematic diagrams of CT strips along a corridor in a composite D scene, including rendered mesh views before (FIG. 10A) and after (FIG. 10B) deformation according to some embodiments of the present disclosure. [Figure 11] 10 illustrates a flattened strip of CT image data extracted along the path D followed by the catheter, according to some embodiments of the present disclosure. [Figure 12] 1 illustrates a schematic diagram of a system for tracking catheter movement within a pulmonary airway, according to some embodiments of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0083] The present invention, in some embodiments, relates to the field of bronchoscopy, and more particularly, but not exclusively, to electromagnetic navigation bronchoscopy.

[0084] overview A broad aspect of some embodiments of the present disclosure relates to robotic navigation of a probe within the branching and dynamically moving airways of a living lung.

[0085] Tracking interventional devices in the lungs The complex anatomy of the lungs poses challenges in the field of navigation guidance for bronchoscopy procedures. In addition, the lungs are flexible and move dynamically, making positioning systems that do not account for lung motion prone to inherent errors.

[0086] The use of fluoroscopy, CBCT, or EBUS (endobronchial ultrasound) to track the location of the probe and / or target potentially increases the accuracy of navigation, but at the expense of, for example, additional equipment, increased procedural complexity, and / or potential radiation dose to the patient and / or physician.

[0087] Another approach uses an ENB (electromagnetic navigation bronchoscopy) system to determine the probe's localization. Such systems use a dedicated antenna (or transmitter) to generate an electromagnetic field and an EM (electromagnetic) sensor to sense the field. The EM sensor can provide 6-DOF (degrees of freedom) position and orientation information of the tip relative to the localization (transmitter) coordinates. EM localization technology has potential advantages for medical applications. For example, the body is virtually transparent to the generated EM field, and no ionizing radiation is used.

[0088] Conventional ENB systems use tip-positioned EM sensors. Limited to such sensors, it is difficult to perform position tracking that adequately distinguishes, for example, between controlled movement (relative to the lung's airway) and movement caused by the lung itself. One method proposed to overcome this limitation is to use optical fibers to achieve shape sensing of the catheter (or other interventional instrument). Shape sensing can enable the interventional instrument to sense the relative positioning of its own components. For example, a tip-positioned EM sensor can be used to provide an anchor position relative to the lung being navigated.

[0089] The term "interventional instrument" is used herein as a generic alternative to more generally a flexible instrument that includes a thin, longitudinally elongated body that is used by advancing distally along a lumen to reach a distal target. Advancement is generally performed using pressure from the proximal side of the instrument toward the distal side, so there is some degree of stiffness to the instrument even if it is flexible. Examples of interventional instruments include catheters, bronchoscopes, and endoscopes. When an instrument is described herein as being "similar to" a catheter, bronchoscope, or endoscope, these characteristics are being referred to.

[0090] In some embodiments of the present disclosure, the EM sensing capability is extended to allow tracking to extend from the tip of the catheter body proximally along the catheter body. For simplicity, this is also referred to herein as “full interventional instrument” tracking. It should be understood that “full interventional instrument” does not necessarily refer to the entire length of the instrument, but rather a length long enough (e.g., at least 5 cm, at least 10 cm, at least 20 cm) to help overcome positioning errors and / or ambiguities that tip-only sensing is prone to. This may be, for example, the entire length inserted into the airway being navigated, or the entire length to a reference point, such as a reference point established along the trachea.

[0091] Instrument-to-anatomy coordinate registration techniques The lungs contain a complex, tree-like, hierarchical system of airways. This complexity presents potential challenges for video guidance. Particularly in the periphery of the lung, airways become smaller and may appear identical, and the number of branching points increases exponentially. Additionally, image quality may be degraded by, for example, mucus present in the lung, making it difficult to navigate based on video alone. As is known in the art, methods for achieving surgically useful registration between the probe position and the anatomical structure it is traversing may be performed in multiple stages using multiple sources of information.

[0092] Coarse Registration: Some NB (navigation bronchoscopy) systems perform registration between a system of 3D localization coordinates (LOC) and the patient's preoperative CT or MRI, and this registration can help manage these difficulties. As used herein, "localization system" refers to the coordinate system in which the probe's position is first described (or "located"). That position is referred to herein as being described by the probe's "localization coordinates."

[0093] CT or MRI images contain 3D information about the exact location and shape of the lungs, specifically the patient's airways. The airways are resolved beyond a certain diameter, which itself depends on the resolution of the CT or MRI scan. The CT or MRI imaging device essentially maps the airways, which are the "roads" along which the interventional instrument may travel. The map is provided to the NB system, which uses the map to provide navigation instructions to the physician, e.g., which direction to turn and how far to travel. The navigation instructions may be presented overlaid on top of the patient's static preoperative CT scan ("map"), along with a path showing the route to the target.

[0094] To display the localized interventional instrument at its actual anatomical location within the lung, a corresponding precise registration (3D transformation) is applied to map the measured 3D interventional instrument localization coordinates to a CT-derived model of the patient's anatomy. One approach to constraining and / or calibrating the parameters of this transformation involves attaching reference sensors to the patient at fixed anatomical locations and using these to achieve a rough registration between the anatomy and the localization system. The reference sensors may move with the patient so the system knows how the body is positioned within the localization coordinates. In some systems, how the reference sensors are actually positioned relative to the anatomy may not be known, and the reference sensors are used to calculate the LOC body anatomy registration relative to some separately determined baseline registration.

[0095] Refining the rough registration: To generate a registration useful for distinguishing adjacent airways, the rough registration can be enhanced with additional information. For example, the registration can be refined using marking of several anatomical fiducials using the tip of an interventional instrument inside the airway.

[0096] In one approach to this, during an initial video-assisted study, the physician visually identifies major predefined airway bifurcations and marks them with the tip of a localized instrument. The marked locations are then collected into LOC coordinates and matched with their corresponding anatomical (e.g., CT or MRI) coordinates. From this, a LOC-to-anatomical alignment is calculated, which can be highly accurate given the proximity of the fiducials involved.

[0097] In another approach, a physician may perform an unsupervised survey with an interventional instrument inside the major airways of the lung. From this survey, the physician collects the position of the interventional instrument in LOC coordinates. An unsupervised registration algorithm matches these with known "roads" from the airway map, for example, using energy minimization optimization methods.

[0098] Once enough samples have been collected, a unique solution is found and an initial LOC-to-anatomy registration is achieved. This can provide high accuracy, especially in the region of the lung where the study was performed (usually the central region). The LOC coordinates may be fixed relative to the patient bed (e.g., where the antenna is located) or may move with the patient by using standard reference sensor compensation methods.

[0099] Dynamic Registration: While these techniques provide an initial registration between the LOC coordinates and the airway map, it can be appreciated that these techniques do not address the inaccuracies of real-time registration due to the flexible and dynamic nature of the lungs.

[0100] The lungs change shape as a result of, for example, breathing, changes in body position, and / or forces exerted on the lungs by interventional instruments moving within the lungs, such as bronchoscopes or other endoscopes, tools used therewith, and / or similar surgical instruments. These (potentially among other causes) result in deviations between the initial registration and the real-time state of the airway anatomy relative to the localization system.

[0101] In some systems, the registration is updated based on history. For example, instead of using only a single 6-DOF measurement of the interventional instrument's tip from the current time frame, a path for the interventional instrument is constructed over a short time window (e.g., a few seconds). The path is then searched within a map to match any possible "roads." As a result, instead of simply locating the interventional instrument's tip within the airway, the map matches the longer path that the interventional instrument traverses over time. For example, if the interventional instrument makes a "right turn" at a particular junction (e.g., as viewed in the interventional instrument's path constructed from sample history), it is known that the final interventional instrument position cannot be on the left airway, but must be on the right.

[0102] One aspect of some embodiments of the present disclosure relates to a parametric lung deformation model that is adjusted through modification of parameters that affect the relevant airway segments of the lung model, while constraining the parameter modifications so that the airway segments move within anatomically based constraints.

[0103] The inventors have discovered that by assigning parameters (e.g., up to about 50, 100, or 150, much fewer than the number of voxels in a CT view and optionally much fewer than the total number of branch points in the airway skeleton), the model is general enough to describe the deformations the lung may undergo due to, for example, breathing, changes in the patient's position, and / or forces applied to the lung by a bronchoscope or other endoscope, catheter, tools used therewith, and / or similar interventional device. The position and orientation of the interventional device along its body (i.e., the position and orientation at multiple points along the interventional device body while the interventional device is inserted into the lung) serves as a data source for updating those parameters as the lung moves, moving the interventional device body with it, or vice versa. In some embodiments, the interventional device body is tracked from its tip to at least the trachea.

[0104] In some embodiments, the model itself includes a skeletonized representation of the lung's airways. The skeletonized representation may be generated, for example, by performing a skeletonization operation on airway segmentation performed on CT or MRI images of the lung. Deformability is introduced into the lung model by parameterizing the branch points of the skeletonized representation, for example, by parameterizing the relative translation and orientation of the segments joined at each branch point. Adjustments to the parameters of branch points in larger (more proximal) airways are optionally allowed to propagate to more distal branch points, simulating the relative movement of the lung lobes relative to one another.

[0105] In some embodiments, constraints on the degrees of freedom for adjusting branch point parameters are added to simulate mechanical properties that limit the range of rearrangements that can occur at individual branch points. Constraints provided in some embodiments of the present disclosure include, for example, one or more of the following:

[0106] Constraints on how much segments at a bifurcation point can change their orientation (angle) relative to one another. This may include allowing relatively free movement for small angular changes, and increasing resistance to change as the relative angular change becomes larger. In some embodiments, resistance to angular change is also specified as a function of airway diameter, diameter wall thickness, and / or bifurcation order to reflect different relative stiffness at different bifurcation points. The effect of such constraints on the model may include, for example, forcing some of the segment repositioning to bend along the segment length rather than at the bifurcation point itself. When multiple bifurcation points are involved, the model may "favor" (due to the constraint) more bending at some bifurcation points and less bending at others.

[0107] Fixation performed distal to the airway segment, i.e., away from the airway branching structures that secure the airway segment to the trachea. Some regions of the lung may be held in place partially by attachment to, interdigitation with, and / or entrapment by structures that are not part of the airway branching hierarchy, e.g., the pleura and septum. The effect of this constraint on the model may be to, for example, cause the airway segment to obliquely align its orientation relative to the proximal fixed branching point so that the airway segment remains oriented more or less in its original distal position.

[0108] Airway segments that are modeled as mechanically interconnected laterally through the lung parenchyma (i.e., not through the branching structure itself). This coupling may be conveyed through some airways that are separated from each other within the branching structure, for example, by multiple proximal branching points. The effect of this constraint on the model may be, for example, to reduce the tendency for airway angles at two or more separate branching points to change their angle in a direction that would push the segments too far apart or push them too far together.

[0109] Airway segments modeled with flexibility along their length. The effect of this constraint is to reduce distortion due to changes in segment angle branching points. This may also help reduce error, as bends along the airway segment of an interventional device can be accommodated by the corresponding parameters of the airway segment in the model.

[0110] Note that the representation of anatomical constraints allows the model to propagate deformations beyond directly measured locations. Changes in measured shape may be most plausible (in terms of the model's configuration) in the context of more global changes in lung shape that propagate away from the current position of the interventional instrument to segments and / or branching points, effects that interact with constraints.

[0111] Another way to apply anatomically based constraints to a model is to use the results of a machine learning algorithm that learns from provided inputs that represent anatomically realistic configurations of the lung's airways. The provided inputs may be based on properly skeletonized images of actual lungs. The placement of the model's airway skeletons is determined according to whether they adequately match the placements on which the machine learning algorithm was trained.

[0112] In some embodiments, machine learning is hybridized with a parameter-based (e.g., error reduction-based) determination of anatomical adequacy of the airway skeleton. For a training set, multiple skeletonized models of the lung are subjected to perturbations (e.g., shapes imposed and / or sensed by models of interventional instruments). A high-fidelity error reduction model (where many constraints are defined, e.g., simply enumerated most or all types) is applied to the perturbed models. While arriving at a result may be computationally intensive, the error reduction can occur offline, so computation time is less critical. Machine learning is then applied to the training set just generated, for example, by training methods known in the art. The results of the machine learning are then optionally used as a method to perform real-time determination of lung airway shape, potentially encapsulating the results of more computationally intensive methods in a less computationally intensive manner. Machine learning can be performed at the level of the entire lung skeleton (e.g., modeled up to three, four, five or more branch orders) and / or on selected portions of the lung skeleton, e.g., a single branch occupied by an interventional instrument, preferably along with the next branch point that the modeled interventional instrument will encounter and traverse.

[0113] This type of representation offers potential advantages for realistic modeling of dynamically changing lung geometry and / or for reducing the number of calculations involved in updating the model lung geometry to match measurements of the modeled lung itself. In some embodiments, the updates occur in real time, allowing the model to be used in dynamic tracking of lung shape, for example, during a navigational bronchoscopy procedure.

[0114] In identifying dynamic parameters of the lung model with bifurcation location and angle, the model also creates potential synergies with lung geometry measurement methods that themselves readily provide bifurcation angle information. In particular, an interventional instrument traverses a bifurcation point as it advances into the lung. The change in orientation of the interventional instrument at a distance corresponding to the modeled location of the bifurcation point provides information about the location and angle of the bifurcation point.

[0115] If only the tip of the interventional instrument is used for measurement, the measured tip position and / or orientation information may become outdated as a representation of lung shape as the interventional instrument continues its distal advancement. If subsequent lung deformation causes angular repositioning of the proximal bifurcation points relative to the current interventional instrument tip position, it may be ambiguous which bifurcation point or points need to be adjusted to return the lung model to an alignment consistent with the newly moved interventional instrument tip position.

[0116] In some embodiments of the invention, this ambiguity is addressed by simultaneously collecting position data along a longitudinally extending region of the interventional instrument. Optionally, the longitudinally extending region extends proximally from the interventional instrument tip back to at least the first pulmonary bifurcation point traversed by the interventional instrument. In this situation, the problem of dynamically identifying which bifurcation points will deform, and to what extent, is constrained by data collected in real time from portions of the interventional instrument that occupy those bifurcation points.

[0117] An aspect of some embodiments of the present disclosure relates to a two-stage process for aligning a deformable model of a static lung shape to a dynamically changing lung shape, in accordance with some embodiments of the present disclosure.

[0118] In some embodiments, in an initial deformable registration step, an unsupervised search is performed (preferably using a fully tracked interventional instrument) to collect samples within the LOC (localization coordinates of the spatially tracked interventional instrument). The deformable airway map is then fitted to the samples using an energy minimization optimization method. This is one example of a method to provide an initial deformable registration between the LOC and the anatomical structures.

[0119] Optionally, in the presence of reference sensor(s), an instantaneous respiratory phase is assigned to each sample collected during registration. This allows the deformable registration algorithm to fit two or more models to the collected samples, such as a model for "inhale" and a model for "exhale." The resulting 4D position samples (3D position and respiratory phase) are collected into a respiratory-interpolated airway map. Optionally, another dimensionality is used, such as 7D, i.e., 3 degrees of freedom for position, 3 degrees of freedom for orientation, and respiration.

[0120] After initial registration, a dynamic deformation tracking algorithm adjusts the deformation model to the deformable lung in real time to provide realistic deformation tracking. The preoperative CT or MRI airway map is correspondingly deformed and placed in localized coordinates to represent the actual state of the airway during the procedure.

[0121] The model is fitted to the lung using an energy-constrained optimization method. The energy function incorporates both shape constraints (which restrict the solved deformation to exactly reasonable configurations, where reasonable is determined by a collection of constraints based on theoretical and / or observational limits on how the lung can move) and target objectives (which require one or more located interventional instruments to be inside the deformed airway).

[0122] The deformation algorithm performs the adaptation using inputs from any combination of localization system sources used as its target object, for example, one or more fully tracked or single sensor interventional instruments, with or without historical data. However, there are certain potential advantages to providing the deformation tracking algorithm with the real-time fully tracked position and orientation of one or more interventional instruments. The historical position of the interventional instrument tip at some location in the anatomy may become outdated if that location in the anatomy itself moves. However, the full shape of the interventional instrument "updates" with the anatomy and provides information that is particularly useful as a constraint on lung dynamics.

[0123] Using a deformable model for both initial registration and dynamic deformation tracking, the airway can then be displayed in its realistic state during the NB procedure. The modeled lung "breathes" in real time in sync with the patient and otherwise dynamically deforms when additional forces are applied. This allows interventional instruments to be displayed in their actual, deformable anatomical location inside the airway, potentially improving the accuracy of the overall system.

[0124] It should be noted that modeling deformations occurring at lung bifurcations other than the one currently traversed by the interventional instrument (especially those relatively far from the currently traversed lung bifurcation) cannot be fully constrained (due to the lack of direct live measurement data). However, as long as a lung model is used to guide navigation toward a target at the end of the interventional instrument's current track, the impact of this ambiguity may be reduced to irrelevance. For example, · The movement of some lung regions away from the navigation track may be irrelevant since it has no effect on the target position. Other such movements may affect the target position via deformation forces applied to branch points on (or near) the current track of the interventional instrument. In this case, new measurements will detect the changes and allow the model to be appropriately deformed to reposition at least the part of the lung containing the target, if necessary. A third possibility is that such off-track lung deformations affect the geometry of portions of the lung airway that have not yet been traversed en route to the target. However, these changes generally become apparent as the interventional instrument continues its advance, allowing for continuous detection of geometry changes at more distal airway bifurcations and enabling successful navigation to the target. It should be appreciated that this applies more generally to distal deformations.

[0125] In principle, some cases belonging to the third possibility could still introduce ambiguity into navigation by deforming the distal bifurcation so much that one branch "displaces" the other when the interventional instrument reaches the distal bifurcation—i.e., one branch occupies a position consistent with the model's current representation of the other branch. However, because mechanical constraints on local tissue deformation tend to diffuse such severe deformations over a relatively large longitudinal extent of the airway track, the incremental deformation experienced between two adjacent bifurcation points can be assumed to be small enough to avoid confusion between the two bifurcation points.

[0126] In any case, it should be noted that even if the branch of one branch point is deformed enough to be confused with (and displaced from) the positioning of the other, the reverse remains unlikely. That case would be roughly equivalent to a 180° twist along the single segment joining the two branch points, which is unlikely to be biomechanically true. Therefore, if the risk of ambiguity is recognized, both branches can be traversed alternately, allowing the risk to be avoided.

[0127] One aspect of some embodiments of the present disclosure relates to dynamic display of 3D image data surfaces transformed to correspond to the current position of the anatomical features they represent based on real-time updated measurements indicating the shape of the represented anatomical features.

[0128] In some embodiments, the 3D image is acquired at a time prior to the interventional procedure, including images of the anatomical regions along the path that will be traversed to reach the intervention target, and optionally the target itself. Particularly if it is a path defined according to the lumen of a branched anatomical structure such as the lung, the path may be tortuous in shape—winding through 3D space so as not to be confined to any single plane in space.

[0129] The path may also be dynamic in shape, insofar as it changes between the time of imaging and the time of the interventional procedure itself and / or changes during the procedure. For example, in the case of the lungs, the lungs may move in position over time for any of the reasons mentioned above. Particularly during the procedure, the respiratory cycle leads to constant changes in the shape of the organ, and interventional instruments may apply varying forces that induce movement and / or deformation of the organ.

[0130] The early acquired 3D image data may include features relevant to the situation for the ongoing procedure, such as the size, shape, and / or location of the target, and the relative location of potentially vulnerable structures such as blood vessels. It is a potential advantage for the physician to be able to view any of these or other structures in their current, dynamically updated shape relative to the path the interventional instrument is traveling and / or is planned to travel.

[0131] In some embodiments of the present disclosure, a method for displaying features along a tortuous path includes calculating a surface along a given path that defines the locations of data elements in a 3D image. The data along the surface is transformed in position so that it is aligned with the current (dynamically changing) shape of the path. The current shape of the path is determined by measurements indicative of its shape, for example, the shape of an interventional instrument occupying the path or a portion thereof, measurements indicative of the current lung inflation state, or another measurement.

[0132] The surface may be displayed (in its transformed three-dimensional shape) to the extent of the available image data, or the surface may be limited to a strip of data locations adjacent to the pathway. In addition to the surface, the pathway itself may be shown. Displayed with the surface may be another representation, such as a 3D mesh, showing the shape of the lumen through which the pathway extends (e.g., the shape of the airway, or the shape of another luminal anatomical structure, such as a blood vessel, digestive tract, urinary tract, or another anatomical structure).

[0133] In some embodiments, the pathway is converted into a planar and / or linear path, and image data for both sides of the path on the defined surface is shown along the path in a flattened image display. Image data is optionally taken from locations along a series of lines defined perpendicular to the path at several locations. However, there is no specific restriction that the surface must be flat along the direction perpendicular to the path; the surface may also be curved in this direction. In some embodiments, the surface is defined as an estimated result that optimally and jointly satisfies constraints on the path shape and constraints such as including portions of important structures, such as interventional procedure targets and / or potential hazards along the path, such as blood vessels.

[0134] Before describing at least one embodiment of the present disclosure in detail, it is to be understood that the present disclosure is not necessarily limited in its application to the details of construction and arrangement of components and / or methods set forth in the following description and / or illustrated in the drawings. Features described in the present disclosure, including features of the present invention, enable other embodiments or can be practiced or carried out in various ways.

[0135] Deformed Model Reference is now made to FIG. 1, which is a schematic flow chart illustrating a method for generating a deformed model of a lung, according to some embodiments of the present disclosure.

[0136] In overview, lung imaging data is converted into a base model (preferably a skeletonized representation of the lung airways), to which parameters are applied that describe how the base model can be deformed to match the particular condition of the actual lung.

[0137] In some embodiments, a deformation model (e.g., as described below) is defined that uses a relatively small number of parameters to define the outcomes of the different possible deformation mechanisms described above. Potential advantages of defining a deformation model with a small number of parameters include avoiding overfitting the model to the observed data and accelerating the optimization process.

[0138] Lung deformation may be treated as a distributed property, unless one airway cannot be deformed significantly differently from its neighbors. Global parameters may be defined to describe deformation features such as how the lungs globally expand or contract during breathing, how one lung deforms relative to the other due to forces applied by the bronchoscope, and / or how both lungs are subjected to global stretching or shearing (e.g., due to different patient postures). These deformation features are also important to the overall accuracy of the navigation system. Because these features are global in nature, they can be described with only a few parameters and constraints.

[0139] The following model is used as the basis for implementing deformation modeling in some embodiments of the present disclosure. The inventors derived it from experimental results using rigid and semi-rigid lung phantoms, animal models, and recorded human data. It should be understood that different models can be used by deformation tracking algorithms that are not limited to the specific parameterization below. While deformation tracking algorithms are generally independent of the specific parameterization of the model in use, appropriate selection of the model can reduce the risk of overfitting and significantly improve the performance of the tracking algorithm.

[0140] In some embodiments, the model is based on an airway skeletonization that provides the centerlines of the airways in the lungs. The airway skeletonization is generated, for example, after data preparation, as follows:

[0141] At block 102, in some embodiments, pre-operative or early operative CT, MRI, or other 3D images of the patient are loaded into the system during an initialization phase prior to the NB procedure.

[0142] In block 104, in some embodiments, the CT or MRI images are processed. 3D voxels belonging to the airways are segmented and marked (e.g., labeled in computer memory) using a segmentation method such as adaptive region growing, deep learning 2D / 3D CNN (convolutional neural network), a combination of both, or any other suitable method. Optionally, other anatomical features, such as blood vessels, that are not navigated in the NB procedure but may provide information about the shape of the lung and / or its degrees of freedom to change shape in different ways are also segmented. For example, blood vessels may constrain the lung lobes from bending or twisting in a particular direction or beyond a particular amount. It should be understood that reference is made more generally to CT or MRI images, which are an example of any detailed source of anatomical information about a particular lung. While CT or MRI images may be considered the current gold standard for providing such information, it is not excluded that current and / or future developments in the field of lung imaging may make it possible to use other sources of anatomical information, such as information obtained from X-ray imaging, magnetic resonance imaging, or other data acquisition methods, for example.

[0143] In some embodiments, the airway segmentation resulting from the operations of block 104 represents a binary 3D volume where each voxel is 1 if it belongs to a cavity or 0 otherwise, which provides an initial representation of the airway map information used by the NB guidance system, as described above.

[0144] At block 108, in some embodiments, skeletonization is performed as a step to determine the tree structure of the airways from the 3D binary volume of segmentation generated at block 104.

[0145] Skeletonization converts the raw airway segmentation volume into a tree-like structure that encodes the airway map. As a result of skeletonization, the 3D segmented tube volume is converted into a graph of centerlines with radii.

[0146] In block 110, in some embodiments, the skeletonization generated in block 108 is further converted into a graph data structure containing nodes connected by edges. Each node represents a single tube of the segmentation (a centerline segment, whether straight or curved, of a particular length, with a constant or variable radius along the centerline). Each edge (directed or undirected) connecting a pair of nodes represents the connectivity of those tubes between the proximal side of one tube and the distal side of another. Thus, the graph in block 110 represents the lung as a tree (acyclic) graph of tubes.

[0147] The root of the tree graph is defined to be the trachea. The first branching point is the carina, with the remainder of the airway hierarchy following as descendants in the tree. The skeleton can be evaluated for completeness by viewing the skeleton as a compressed data representation of the airway segmentation from which the segmented volume can be recovered, and performing an inverse transformation from skeletonization to segmentation. A good match with the original segmentation (which is found to be true) means that the skeleton encodes most of the volume of segmented information. Skeletons offer potential advantages for use as representations of airway maps, as they are much smaller in size and much easier to traverse and analyze.

[0148] Reference is now made briefly to Figure 2, which shows a schematic representation of a lung airway skeletonization 200, according to some embodiments of the present disclosure. The skeletonization 200 is rendered with thickness (black regions 201) and centerlines (light lines 202 along the centers of the black segments).

[0149] Returning to FIG. 1 , in some embodiments, at block 112, deformation parameters are defined for the tree graph. These parameters provide flexibility to the model. Tree graph deformation parameters are defined for each skeletonized branch node, but not necessarily for every such node in the tree graph. In some embodiments, nodes are defined for a branch order of up to about three or four levels. Nodes may be defined for higher levels as appropriate, such as along a path traversed by an interventional instrument.

[0150] In some embodiments, each branch point of the skeleton has a 4x4 3D rigid transformation matrix from local branch point coordinates to CT coordinates.

number

[0151] According to some embodiments of the present disclosure, the branch coordinate system 304 (coordinate system

number

[0152] The coordinate system 304 may be, for example, a branch point

number

number

number

number

number

number

number

[0153] Additionally, the first (root) vertex in the tree also has a coordinate system whose Z axis points in the direction of the beginning of the trachea.

number

[0154] A coordinate system with a specially chosen orientation as described above

number

[0155] Each 3D conversion

number

number

number

number

number

number

number

number

number

number

number

number

number

number

[0156] Reference is now made to Figure 4, which shows a schematic illustration of a lung airway skeleton before (left side 410) and after (right side 411) transformation between bifurcation points 402 and 401, according to some embodiments of the present disclosure.

number

number

[0157] Also,

number

number

[0158] In some cases, it may be desirable to apply a transformation only to a single branch point without affecting its descendants.

number

number

number

[0159]

number

number

number

[0160] To model lung deformation,

number

number

number

number

number

number

number

number

number

number

number

number

number

[0161] In parent branch point coordinates

number

number

[0162] In some embodiments, interpolation of points inside a branch is performed as follows.

number

number

number

number

number

number

number

number

[0163] Modified branch point transformation

number

number

number

number

number

number

number

[0164] Following these definitions, in some embodiments, the following interpolation formula is applied:

number

[0165] In the above equation,

number

number

[0166]

number

number

number

number

[0167] In summary, the deformation model described with respect to Figures 1 to 4 assigns a coordinate system to the branching points of the skeletonized airway model, and each branching point is

number

number

number

number

number

number

[0168] Although bifurcation points have been described, bifurcation curves are correspondingly interpolated using the bifurcation points at their start and end points. Using only a few parameters, the model describes most of the natural deformations that occur in the lung during NB procedures. The model can also be applied to the use of real-time dynamic deformation tracking and respiratory compensation, as shown below.

[0169] Deformable Alignment Reference is now made to FIG. 5, which is a schematic flow chart illustrating a method for performing deformable registration between the position of an interventional instrument placed in a lung airway and a bifurcation model of the airway, according to some embodiments of the present disclosure.

[0170] In block 502, in some embodiments, a physician performs an unsupervised survey using one or more single sensors or fully tracked interventional instruments, as described above. Samples are captured in LOC coordinates; their location within the anatomy is unknown and determined by a registration process. Position data 505 generated by the survey is made available to the operations of block 504, and in particular block 504A.

[0171] Baseline airway map 503 is constructed using at least a partial representation of some previously measured geometry of the lung, generated, for example, as described in connection with Figure 1. The lung interrogated in block 502 and represented by baseline airway map 503 is the same lung, although there may have been changes in geometry between the time the data for baseline airway map 503 was acquired and the interrogation in block 502.

[0172] In block 504, in some embodiments, a tight registration is determined between the unsupervised survey data and an airway map (generated from prior imaging, e.g., using CT). Block 504 is divided into blocks 504A and 504B, which may be performed simultaneously or sequentially, and optionally iteratively. In block 504A, modifications to the airway map are made to improve its resemblance to the current anatomical situation. Block 504B represents the tight registration (e.g., translation and rotation).

[0173] The registration algorithm of block 504B is known in the art. The algorithm typically starts with an unsupervised search performed using a single sensor and performs a rigorous registration determination using an energy minimization optimization method. However, without additional correction (provided in block 504A), the airway map generated from the preoperative CT may not accurately reflect the anatomy at the time of the procedure. That is, there will most likely be deformations between the preoperative CT airway map and the actual lung condition during the procedure. Potential reasons for this include, for example, the preoperative CT (or other preoperative imaging procedure) being typically taken days or weeks before the procedure, the patient being examined in an altered position relative to the CT scan (or other baseline imaging procedure), and / or forces exerted on the lung by a bronchoscope or other endoscope, catheter, tools used therewith, and / or similar surgical instruments that modify its shape.

[0174] Therefore, in some embodiments of the present disclosure, an additional operation represented by block 504A adjusts the deformation model so that it is more suitable for finding a tight registration between the LOC and the anatomical structures. The anatomical structures are initially represented by a deformable model to which a "null" deformation model has been applied to the CT (representing the state of the lungs during the CT as the baseline airway map 503). During registration, the deformation model is modified by deformations selected to compensate for changes in the state of the lungs between the time of the CT or other 3D imaging and the time of the NB procedure.

[0175] Block 506 represents the initial registration of the deformation model (airway map), which may continue to change during the procedure as the lungs undergo further dynamic changes.

[0176] In some embodiments of the present disclosure, the selection of changes to the airway map parameters (transformations) used in block 504A is based on position and orientation sensory data obtained from one or more fully tracked interventional instruments positioned and / or moving within the lung. Position data from tracking is provided from block 505 to block 504A. For example, compared to an interventional instrument that is tracked only at its tip, a fully tracked interventional instrument provides more data, particularly data (e.g., a timestamp) that more directly indicates the longitudinal extent of the interventional instrument's position at the current time.

[0177] Because the lungs move during the procedure, interpretation of old position data can be ambiguous or misleading, making the data less useful than data that clearly indicates the current location of all or most of the interventional device. As an indication of the types of problems that can occur with older data, as the lungs move, the assumption that the interventional device location shown in the older data is still within the original airway becomes invalid (or at least questionable). To correct for this, some information about how the lungs have moved is needed. However, with tip-only tracking, no sensors are in place to provide that information. On the other hand, with a fully tracked interventional device, the moved position of the portion of the interventional device proximal to the tip can be relied upon to faithfully represent the moved position of the airway itself. Furthermore, the connection between the old and new data is clear, as the interventional device itself serves as a kind of constant “ruler.” Therefore, comparing the old and new data can directly indicate what kind of lung movement may have occurred.

[0178] Reference is now made briefly to Figure 6, which schematically illustrates differently aligned lung model states 602, 602A where survey data is aligned as survey data 603, 603A, 603B, in accordance with some embodiments of the present disclosure. In this example, the survey data 603 samples were collected using a fully tracked interventional instrument. In the transition from state 602 to state 602A, regions 608 and 609 (for example) have changed their position relative to the rest of the lung model.

[0179] Panel 610 (left) shows the original survey location 603 roughly superimposed on lung model 602, resulting in a significant misalignment with survey data 603A. Panel 620 (center) shows the result of a strict registration of survey data 603A to lung model 602, which is similar to the result of running block 504B without prior processing of the model by block 504A, resulting in errors, especially in the more distal portions of the lung. Panel 630 (right) shows lung model 602A transformed by deformations induced by survey data 603B to correct the strict registration errors seen in panel 620.

[0180] Returning to the discussion of Figure 5, in some embodiments of the present disclosure, the baseline airway map 503 is constructed as an instance of the deformation model described with respect to Figure 1. To capture both the deformation and the rigorous alignment between the LOC and the anatomy, the baseline airway map 503 is constructed as a bifurcation coordinate system within the LOC coordinate system.

number

number

number

number

[0181] In some embodiments, the optimization process of block 504 is configured to (a) place all collected interventional instrument LOC samples inside the deformed and transformed airway (essentially to perform the operations of block 504B), and (b) fit a model describing a reasonable deformation of the lung that allows this (the operations of block 504A). These two conditions—target objectives and shape constraints—are optionally encoded into a single energy function. Alternatively, the fit is performed using multiple energy functions, alternating iteratively.

[0182] To prevent overfitting and make convergence more robust, branch points of generations above a certain threshold (e.g., above the third generation) are optionally omitted from the model. Optionally, one or more of the higher branch point generations are included in the fit calculation only after the initial fit with the lower generation branch points has converged to a solution. Intervention instrument position measurements (samples collected by survey) are optionally preprocessed or assigned weights to balance their contribution to the fit.

[0183] Six parameters are used to encode the exact transformation between the locally deformed airway map in CT coordinates and the LOC coordinates.

number

number

[0184] Reference is now made to Figure 7A, which generally illustrates elements of determining a deformation transformation for a deformation model to match available data and using the deformation transformation in further dynamic modeling, according to some embodiments of the present disclosure. The description of Figure 7A outlines some more specific embodiments of the operations of Figure 5, and adds blocks showing how the results of that embodiment can be further used to model dynamic changes in the lung during treatment.

[0185] In some embodiments, the local deformation (e.g., performed by the operations of block 504A) may be

number

number

number

[0186] Block 706 represents the geometry assigned to the bifurcation point, in the form of at least the rotational states assumed in the model by the bifurcation of the bifurcation point. Block 708 represents geometry constraints. Optionally, these constraints are defined for each bifurcation point to limit and / or guide how the geometry of block 706 can be modified.

number

number

number

[0187] moreover,

number

[0188] The above

number

number

number

number

number

[0189]

number

number

number

number

number

number

number

number

[0190] The optimization process is independent of the choice of a particular energy function. The overall shape energy function is then:

number

[0191] To achieve both goals of block 504, i.e., to detect rigid body transformations (block 504B) and local deformations (block 504A) so that position data 505 (e.g., from interventional instrument samples collected during the interventional instrument's interrogation movement) is within the deformed and transformed airway, in some embodiments, the distance between the collected interventional instrument samples on the one hand and the deformed and transformed airway tube (in LOC coordinates) on the other hand is taken into account.

[0192] When a sample from position data 505 fits nicely inside the airway tube defined by the airway map (baseline airway map 503, or another airway map after modification), then it should not contribute any error to the optimization. However, if the sample lies outside the boundary defined by the branching airway model, then the model and / or the exact registration needs to be corrected to minimize this error.

[0193]

number

[0194] Variations in the use of collected interventional instrument position samples are optionally implemented according to the type of interventional instrument position data available. For example, when implemented in a 5DOF tracking system, each

number

number

number

number

number

number

number

number

number

[0195] In the above equation,

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

[0196] The total energy function is:

number

[0197] In the above equation,

number

number

number

number

number

number

number

[0198] The output of the deformable registration process is a transformation and deformation model that can be used as the initial state for the NB procedure.

number

[0199] Dynamic Deformation Tracking The deformable registration in Figure 7A is based on the anatomical structure.

number

number

number

[0200] In some embodiments, the updated registered airway map 720 is

number

number

number

[0201] In some embodiments, the breathing transformation (e.g., as described in the section on breathing transformations)

number

number

[0202] At the start of the NB procedure, after generating the initial aligned airway map 506,

number

number

number

number

number

[0203] The dynamic deformation tracking algorithm implemented by the operations of block 720 is similar to the deformable registration implemented by the operations of block 504 .

[0204] Multiple times

number

number

[0205] For single sensor interventional instruments:

number

number

[0206] Similar to the initial deformable registration that produces the initially registered airway map 506, the deformation tracking algorithm of block 720 is independent of the number of interventional instruments used and the configuration of each interventional instrument. The dynamic deformation tracking algorithm of block 720 determines whether, at each time t, most samples from the position data 505A will be inside the airway and the applied deformation will describe a plausible lung deformation as described above.

number

[0207] However, compared to the deformable registration algorithm implemented by block 504, the deformation tracker of block 720 does not have as many interventional instrument samples at its disposal.

[0208] In some embodiments, deformation tracking only uses the current position of the full interventional instrument at the current time t. In some embodiments, deformation tracking also uses past samples, e.g., with decreasing weighting as the data gets older.

number

[0209] It is an objective of some embodiments of the present invention to provide fast (real-time) and low-latency continuously updated deformation tracking. This objective is supported in part by incrementally updating the deformation model to match new data from position data 505A as it is acquired. This incremental updating is consistent with the low-latency objective as long as the actual deformation rate of the lungs is relatively slow compared to the sampling rate of position data 505A.

[0210] In some embodiments, the deformation tracker of block 720 reuses the definition of the energy function of the deformable registration of block 504 to define the energy of the dynamically deforming part of the model.

number

[0211] In the above equation,

number

number

number

number

number

number

[0212] The dynamic deformation tracker, in some embodiments, assumes that the shape obtained from the deformable registration is true - i.e., does not include it in its shape energy function. The dynamic deformation tracker performs local deformation correction to approximate a sample of the interventional instrument, such that the shape does not deviate significantly compared to the original shape defined by the deformable registration performed in block 504.

number

[0213] As long as the information available to the dynamic deformation tracking algorithm per iteration is limited, there is a risk of overfitting. In any case, convergence and run time must occur quickly to maintain real-time updates.

[0214] In some embodiments, meeting these criteria is aided by restricting the attention of the tracker algorithm implemented in block 720 to bifurcation points that are close to the interventional device.

[0215] For example, suppose the tracked interventional instrument is located inside the right lung:

number

[0216] In some embodiments, the optimization tools used when recalculating the dynamic transformation state vector 712 may affect the energy function.

number

[0217] Again, similar to some embodiments of the deformable registration algorithm implemented in block 504,

number

number

[0218]

number

number

number

number

[0219] Reference is now made to Figure 7B, which is a flowchart outlining an embodiment of block 720 of Figure 7A, in accordance with some embodiments of the present disclosure. The operations of Figure 7B are performed iteratively as interventional instrument position data is continuously acquired to maintain an updated model of the lung during treatment.

[0220] At block 730, in some embodiments, the deformation tracker

number

number

[0221] At block 732, in some embodiments, the tracking device:

number

[0222] At block 734, in some embodiments, the model state vector is calculated from the previous iteration

number

[0223] At block 736, in some embodiments, one or more convergence steps are performed using a nonlinear iterative optimization method, such as, for example, Levenberg-Marquardt (LM),

number

number

[0224] At block 738, in some embodiments, the resulting

number

number

number

number

number

[0225] Additionally or alternatively, dynamic deformations may be performed at every update step

number

number

number

number

number

[0226]

number

number

number

[0227] The remaining remarks in this section relate to the details of the error minimization calculation of block 736.

[0228] By performing only local aggregation steps,

number

[0229] This may occur, for example, when the interventional instrument is positioned near a deep, highly deformed bifurcation point and / or a bifurcation point whose deformation was not modeled during the deformable registration step.

[0230] From such a bifurcation point, it may not initially be entirely clear whether the interventional instrument is turning right or left at the bifurcation point. In the example described below, it is assumed that the interventional instrument is turning right at the ambiguously modeled bifurcation point. A potential advantage of a perfectly tracked interventional instrument combined with the deformation model described herein is that after entering a particular airway and moving a short distance, the shape of the interventional instrument itself may be significantly deformed, but still teach the system whether the interventional instrument has turned right or left. After the right turn, for example,

number

number

number

number

[0231] After the interventional device has traveled a short distance within the airway,

number

number

number

number

[0232] In some embodiments of the present disclosure, the global search is incorporated into a dynamic deformation tracking algorithm that may allow escaping local minima.

[0233] An example of a global search implementation is

number

number

number

number

number

[0234] Another option is to use a multiple hypothesis approach. At a particular branch point, the skeleton can be split into two sub-skeleton: one containing the left turn and all descendants (but not the right turn), and the other containing the right turn and all descendants (but not the left turn). The tracker will use one "left" skeleton and the other "right" skeleton.

number

[0235] Formalizing this approach reduces the iteration time

number

number

number

number

number

number

number

number

number

number

[0236]

number

number

number

number

number

number

number

number

number

number

number

[0237] Similar to deformable registration, the dynamic deformation tracking algorithm can be formalized in terms of an error minimization optimization problem. The overall deformation error function

number

[0238] Along with deformable alignment,

number

number

number

number

[0239] Breathing Modification In some embodiments of the present disclosure, the rigid registration determined in block 504 of Figures 5 and 7A is a periodically changing registration. More specifically, in some embodiments, the registration changes according to the movement of the respiratory cycle.

[0240] A challenge in NB procedures is overcoming the deformations and imprecision caused by breathing, which can be particularly important for peripheral targets within the lungs, which are more distant from the relatively fixed and / or rigid structures of the trachea and large bronchi.

[0241] An approach to address this problem is to use a reference sensor attached to the patient's chest to create a generalized LOC coordinate that moves with the body and chest. Respiratory-induced lung and interventional device movements are compensated for to some extent using changes in reference sensor location, as the two movements tend to be correlated.

[0242] However, lung deformation due to breathing is complex: for example, some airways may deform in one direction while others deform in the opposite direction, which can reduce the accuracy of simple reference sensor-based breathing compensation algorithms.

[0243] To partially compensate for such complicating factors, the reference sensor readings may be interpreted with respect to the respiratory phase.

number

number

number

number

number

number

number

number

[0244] Once the respiratory phase is calculated, a function that correlates between respiratory motion and respiratory phase can be applied. In summary, the function accepts respiratory phase as a parameter that predicts an output that estimates the respiratory motion at a specific location of the interventional device. The function can use a respiratory model adjusted for patient motion, for example, learned by imaging and / or measurements performed offline (before the interventional device navigation procedure) and / or during the NB procedure.

[0245] By applying the function in real time, the respiratory motion of the interventional device can be compensated for, for example, by making it static relative to the static airway map, even though both the interventional device and the mapped airway are actually deforming with breathing, although motion compensation degrades once the lungs deform enough that they no longer match the cyclical predictions of the respiratory model.

[0246] In some embodiments of the present disclosure, anatomical structures and interventional instruments are dynamically displayed in their measured and / or estimated actual shape / position states as the interventional instrument moves through the lungs and including deformations due to respiration or other causes, thus preserving rather than eliminating the respiratory movement of the interventional instrument through the use of compensation.

[0247] In some embodiments, to model breathing, respiratory phases are calculated based on one or more reference sensors attached to the patient's chest, for example.

number

[0248] A high-pass filter may be used to filter out other body movements of the patient. Additionally or alternatively, multiple reference sensors may be utilized, and the location of each sensor may be calculated within the sensor's PCA coordinates. In some embodiments, instead of using the absolute or relative height of the reference sensors, the area of ​​a triangle between, for example, three reference sensors may be examined. For example, when the area of ​​this triangle is maximum, the patient may be assumed to be in a fully inhaled state, and when the area of ​​the triangle is minimum, the patient may be assumed to be in a fully exhaled state. The respiratory phase φ may be calculated in a manner similar to one of the techniques described above, or by any other suitable method.

[0249] In some embodiments of the present disclosure,

number

number

number

number

number

number

number

[0250] For example, the dynamic deformation tracker of block 720 of FIG.

number

number

number

number

number

[0251] Even if no interventional device is present in the airway, the respiratory phase is optionally calculated using one or more reference sensors attached to the patient, allowing the airway map to continue "breathing" in sync with the patient's actual breathing.

[0252] respiratory phase

number

number

number

number

number

number

[0253] In some embodiments, the definition of the state vector and energy function for the deformable registration is modified to be defined as follows:

number

[0254] The state vector now incorporates both models, exhalation and inhalation. The shape energy function is optionally modified so that both models impose similar shape constraints.

[0255] For further customization, each model may be assigned different shape constraints, or at least weighted differently. For example, weighting may be based on the knowledge that the preoperative CT was taken in a fully inhaled state, and therefore there is expected to be more significant deformation in the "exhalation model" compared to the "inhalation" model. Weighted corrections may be made, for example, by adjusting the phase-varying term

number

[0256] Each interventional device sample is represented as a single deformable model

number

number

number

number

number

[0257] This allows both models

number

number

number

number

number

[0258] Two models of interventional instruments were collected.

number

number

number

[0259] Note that the two major phase breathing model is optionally extended to any number of major breathing phases. For example:

number

number

number

number

number

[0260]

number

number

number

number

[0261] Deformation and Respiration View In some embodiments of the present disclosure, the system is configured to optionally present a view of the deformable lung model together with a view showing the position of one or more interventional instruments positioned within the lung modeled by the deformable lung model. Reference is now made to Figure 8, which is a schematic flowchart outlining a method for generating a deformable lung model view, according to some embodiments of the present disclosure.

[0262] A 3D mesh describing the smooth boundary surfaces of the airway segmentation can be used to represent an airway map for use during navigational bronchoscopy. The 3D mesh consists of a set of vertices and triangular faces that connect the vertices to form a surface. Optionally, the airway 3D mesh is generated directly from the airway skeleton (centerline and radii) to form a "synthetic" 3D mesh representing the airway map in a skinning process. Additionally or alternatively, gradient information obtained from the CT volume or directly from the segmented airway volume is used to capture the fine texture of the airway. This may result in a more "realistic" mesh.

[0263] In some embodiments of the present invention, the current deformation state of the deformable lung model is preferably applied to any view of lung-related 2D / 3D features (view objects) displayed by the system (i.e., not just the skeleton of the model), which has the potential advantage of realistically showing the anatomical structures in their real-time state.

[0264] Examples of such view objects include, for example, an airway 3D mesh, a target representation such as a sphere or a realistic target mesh, a path through the lung, a waypoint within the lung, and / or a CT slice imaging the lung.

[0265] Referring now to block 810 of FIG. 8A, in some embodiments of the present disclosure, determining the positioning of the deformed state of a view object begins with associating the view object with a skeletal airway of a deformable lung model.

[0266] In some embodiments, this association is performed by associating one or more vertices of the displayed object (e.g., defined in a pre-distortion space such as the space of the original CT image used to define the model skeleton) with their respective airway distances and normalized distances along the airway.

number

number

number

[0267] At block 812, in some embodiments, the 3D mesh (or other representation of the view object) is

number

number

number

[0268] In the above formula, in CT (or other 3D image) coordinates

number

number

number

number

number

number

[0269] Optionally, at block 814, the lighting model used to render the deformed 3D mesh is improved by applying a similar transformation to the vertex normals (e.g., reducing the appearance of facets).

number

[0270] In the above equation,

number

number

number

number

[0271] Optionally, the transformations of blocks 812, 814 are applied by a central processing unit (CPU), which transforms all vertices (and normals) using the appropriate transformation transform. Optionally, the transformations are applied by a graphics processing unit (GPU), for example, using a 3D shader. Shaders are programs that run on the GPU for specific sections of the graphics pipeline. Vertex shader programs are responsible for processing each vertex in the graphics pipeline and can be used to modify the geometry of an object before it is displayed.

[0272] Dividing a 3D mesh into a hierarchy of branches (sub-meshes) helps to apply deformation models using the vertex shader. Before drawing each branch (sub-mesh), the vertex shader calculates the matrix

number

number

number

number

[0273] To apply the deformation model to small objects such as 3D target spheres, guide arrows, text, etc., it is observed that the deformation model is smooth and does not change significantly in its small neighborhood. Therefore, the small object can be assumed to undergo a uniform deformation transformation. To apply the deformation model, the small object is assigned a certain number of points from the airway map indicating a particular branch (e.g., the nearest branch) and its distance along the branch (or its centroid distance).

number

[0274] Reference is now made to Figures 8B and 8C, which schematically depict renderings of a 3D mesh 800 of a single model branch point 802 and its descendants before and after deformation, according to some embodiments of the present invention. Also shown is a representation of a target 801.

[0275] The transformation applied to convert the underlying model of FIG. 8B to that of FIG. 8C is the same single-bifurcation transformation of bifurcation point 802 as shown for FIG. 4. However, the transformation is now applied to the vertices (and normals) of the 3D mesh, as described above, to display a smoothly deformed 3D mesh. Parameter adjustments are made to the bifurcation point itself. The increased curvature seen in FIG. 8C is the result of the airway being interpolated between the changed configuration at bifurcation point 802 and the bifurcation point closest to it. The interpolation used can be, but need not be, a uniform curve. For example, in some embodiments, stiffness is constrained to be higher near the more proximal bifurcation point where the airway is larger and / or the airway wall is thicker. Position measurements from an interventional instrument passing through the airway region connecting the two bifurcation points, if available, can also or instead be used as a constraint. The curvature is optionally constrained by the curvature present in the baseline state of the lung.

[0276] Target 801 corresponds to

number

number

[0277] In other situations, the target's deformation position can be predicted based on a tracked, deformable airway. This is optionally implemented by using a trained model (e.g., a final element model) that predicts how deformations propagate through tissue (as represented by a CT or MRI scan). In this case, the target is not associated with an airway or any other specific anatomical structure. Instead, its position is deformed by propagating deformations of the tracked airway toward the target and surrounding tissue.

[0278] The deformation model is a deformation transformation

number

[0279] The problem of extending the deformation model to regions outside the airways is a problem of extrapolation. The deformation model is known at specific locations in 3D space and does not need to be extended outward to other points. Extrapolation can be achieved, for example, by using thin-plate splines (TPS) or other radial basis functions (RBFs) or general basis methods more generally. Additionally or alternatively, machine learning methods can be applied that predict the most likely deformation propagation within lung tissue based on training on a training set of examples, and / or finite element simulation models can be used to predict the propagation of deformation through tissue represented in CT or MRI scans.

[0280] In some embodiments, for example, a set of basis functions (e.g., TPS) is trained on the known control points of the deformation model so that the deformation model also sends bifurcation points or full-length centerline points (according to the deformation model) to their known destinations. The extrapolation model then uses basis functions that can be sampled anywhere. Because the basis functions approximate the known control points with reasonable accuracy, it is assumed that the basis functions will send external points (not belonging to the airways) to reasonable destinations. These models can be combined with a smoothness criterion that is suitable for the lungs.

[0281] Another example of an extrapolation method uses the K-nearest neighbor (KNN) method.

number

number

number

number

number

number

number

number

number

number

number

number

[0282] Or multiple transformations

number

number

number

[0283] Instead of using a simple extrapolation scheme based solely on smoothly expanding control points, more complex deformation extrapolation can be implemented. For example, ribs can be assumed to be static unless they deform significantly during the NB procedure. This information can generate points on the segmented rib cage surface that can be added to the extrapolation function in terms of additional control points, forcing those points to remain stationary during deformation. This provides boundary conditions for the extrapolation function, forcing it to decay as it approaches the rib cage. Alternatively, the lung can be modeled using not only its airways but also blood vessels, fissures, pleura, and other anatomical features. These features can then be fed into a skeleton model (similar to the deformation model described above) and used to predict the deformation of the entire lung based on a few known control points. In some embodiments, the physical properties of the lung are simulated, for example, by applying finite element simulation, thus providing a better prediction of lung deformation between control points. In some embodiments, a trained machine learning model is used to predict the propagation (or extrapolation) of deformation from known control points to other lung tissue, for example, replacing traditional finite element simulation with understanding derived from the behavior of controlled lung tissue.

[0284] 9A-9B, which schematically illustrate a 16x16 slice interpolation grid before (FIG. 9A) and after (FIG. 9B) deformation of a single branch point and all its descendants, in accordance with some embodiments of the present disclosure. Branch point 900 of skeleton 901 has been deformed in FIG. 9B, which also leads to the movement of lung region 902.

[0285] When deforming a complete CT slice, each voxel in the CT slice optionally uses one of the extrapolation methods described above. However, this can be computationally expensive because a CT slice typically contains a large number of voxels (e.g., 512 × 512), and the calculation of the extrapolation formula becomes quite complex. Deformation of a CT slice can be simplified by constructing a low-resolution uniform grid consisting of, for example, 16 × 16 blocks that covers the entire CT slice. Deformation is calculated at each endpoint, and deformations between grid points are then calculated by simple interpolation using nearby endpoints, for example, using 2D bilinear interpolation, bicubic interpolation, spline interpolation, or any other suitable method. Using interpolation methods rather than direct pixel-by-pixel deformation calculations is highly favorable in terms of computational resources, and due to the reasonable smoothness of the deformation model, it is nearly as accurate as direct deformation techniques. When more detailed results are required, a denser grid can be used (e.g., 32 × 32 blocks).

[0286] Reference is now made to FIG. 9C , which schematically illustrates a CT slice 910 extending along a geometric cross section defined to curve to include regions of the original 3D image of the anatomical structure corresponding to positions along a tortuous path 914 through the anatomical structure, in accordance with some embodiments of the present invention. In some embodiments, the path 914 is a path through an airway. In some embodiments, the path 914 is defined by the shape and position of an interventional instrument. In some embodiments, the path 914 is defined to use anatomical features, such as a treatment target, a blood vessel, or other structure of interest, as waypoints. The path 914 may be defined based on conditions during the procedure (e.g., the current position of the interventional instrument) and / or may include predefined sections, such as a planned path to reach the target of the interventional instrument.

[0287] CT section 910 corresponds to data from the source CT image located at a location intersected by a surface extending through the volume of the 3D image, the surface itself including locations along path 914 .

[0288] In some embodiments, the path 914 is rendered to correspond to a 3D shape that corresponds to the measured shape of the interventional instrument to which it corresponds, and the CT sections 910 are warped from their original coordinates to maintain regional correspondence. In some embodiments, the interventional instrument path 914 and the CT sections 910 are shown together in a composite 3D scene that also includes a rendered mesh view of the deformed model 912.

[0289] For a single-sensor interventional instrument, a deformed CT slice 910 is displayed reflecting the deformation of the entire lung using the low-resolution deformation grid described above and is centered on the tip of the interventional instrument. The CT slice 910 may be global (reaching the CT boundary) or local (e.g., surrounding a specific path through the lung's airways, as illustrated in Figures 10A-10B). For a fully tracked interventional instrument, the displayed CT slice may be geometrically curved to intersect all or most of the interventional instrument's curve (not just the tip) and make it non-planar. Such curved CT slices display deformed anatomical features along the entire length of the interventional instrument, which provides valuable information for the NB procedure (e.g., location of blood vessels, fissures, septa, and / or pleura). The curved CT slice may also intersect certain important anatomical features. For example, the curved CT slice may intersect the next navigation bifurcation in the airway map, so that a cross-section of the bifurcation is displayed in the slice before the interventional instrument. A potential advantage of using curved CT slices is that they compress the deformed 3D CT volume into a 2D volume that encodes most of the anatomical features that are valuable for navigation in the area of ​​interventional instruments.

[0290] Because not all important anatomical features always lie on a smooth 2D curved surface, the specific shape of the curved CT slice can be solved with an optimization process that searches for a 2D curved surface that approximates a list of important features (e.g., interventional instrument path 914 and / or target 916) in position and orientation, with weights describing the importance of each feature. The optimization process may include weights to penalize excessive distortion. For example, the optimization may penalize deep and / or shallow bend angles and / or penalize extreme bends based on the amount of additional surface area they contribute. The end result is a curved CT slice that compresses the deformed 3D volume into a compact 2D CT view.

[0291] 9C as an example, it should be understood that the display method is optionally performed using other 3D image types, such as MRI images, ultrasound images, and / or positron emission tomography (PET), etc. Optionally, slices from two or more 3D images (optionally of different imaging modality types) are combined.

[0292] Reference is now made to Figures 10A-B, which schematically illustrate CT strips 1002, 1003 along a corridor in a composite 3D scene including rendered mesh views before (Figure 10A) and after (Figure 10B) deformation according to some embodiments of the present disclosure.

[0293] The low-resolution 2D deformation grid described with respect to Figures 9A-9B allows for rapid computation of deformations for any pixel between grid points through interpolation, enabling real-time display of deformed CT slices. The same technique can be generalized to create a 3D deformation grid that covers the entire volume of a CT (or other 3D image). Deformations are computed for each control point of the grid and then interpolated to the voxels between the grid points. The 3D deformation grid allows for fast computation of deformations anywhere within the imaged volume. This may enable real-time display of deformed 3D views of the lung, such as deformed virtual bronchoscopy or deformed virtual fluoroscopy, both of which use ray-casting rendering techniques. In a standard virtual bronchoscopy view, a virtual camera is positioned inside the airway. Each pixel on the screen represents a single virtual ray of light that passes through the imaged voxel and accumulates the final shadow for that pixel. The end result is a realistic image that appears similar to an actual bronchoscopy image of the inside of the airway. While standard virtual bronchoscopy algorithms sample a static CT volume in their ray tracing steps, a 3D deformation grid can be used in dynamic virtual bronchoscopy algorithms to create ray-traced images of the deformed CT volume. Instead of sampling voxels in a static 3D image, at each ray tracing step, the location of each ray is transformed using the 3D deformation grid to a deformed location within the initial, undeformed 3D image, thereby effectively sampling the deformed CT in real time.

[0294] During a NB procedure, the physician guides a tool (e.g., an interventional instrument) to follow a path from the trachea 1000 to the target 1005. The path can be displayed as a 3D centerline within the airway map. The physician can be presented with raw CT data around the interventional instrument using local CT slices centered on the interventional instrument or its tip, as described above, or the system can render a full CT strip in 3D along the path (optionally a strip extending approximately equidistant on either side of the path), as shown, for example, in FIGS. 10A-10B. As the lung deforms (e.g., as indicated by one measurement and / or another parameter of an interventional instrument), the CT data displays the deformed location along with other deformed anatomical features. To deform a CT slice, its position in space can be transformed to the deformed location (while preserving its original CT data), or its position in space can be kept fixed while its CT data is transformed. Because the 3D CT strip is naturally tied to the airway, it transforms in space along with the airway, as shown in FIGS. 10A-10B. Thus, the 3D CT strip retains its original CT data, but its position in space transforms smoothly along with the airway, similar to the deformation of the 3D airway map.

[0295] Reference is now made to FIG. 11, which illustrates a flattened strip 1100 of CT image data extracted along a 3D path followed by an interventional instrument 1101, according to some embodiments of the present disclosure.

[0296] The 3D CT strip shown in FIGS. 10A-10B is optionally used to generate a 2D pathway view in which the CT strip 1100 is displayed as a static 2D background image, with the interventional instrument 1101 and / or target 1102 projected on top of the static image in pathway coordinates. Like the 3D CT strip, this view potentially encodes the most important anatomical information for navigating along the pathway to a specific target, but it is projected into a 2D view that shows the entire pathway as a flat surface from start to finish, independent of the 3D camera location and without the 3D occlusion that can occur in the 3D CT strip view seen in FIGS. 10A-10B. A potential advantage of the 2D pathway view is its localization, making it invariant to deformations. When a deformation is applied, the CT strip remains unchanged because it follows the deformed airway (as described above), and the interventional instrument typically retains its position within the pathway (e.g., in pathway coordinates).

[0297] Applying the deformation model to all displayed 2D / 3D objects, including the 3D airway mesh and CT slices, allows for the generation of composite deformed scenes in real time. For example, while a patient is breathing, a respiratory 3D airway map can be displayed overlaid with respiratory CT slices, perfectly synchronized with the patient's breathing. As another example, during maneuvering, an interventional instrument may apply forces to the airway it navigates through. The system can then not only display how the 3D airway deforms in real time in response to the interventional instrument's maneuvering (based on the deformation model), but also display corresponding deformed versions of local CT slices in close proximity to the navigated airway. The physician is then presented with real-time deformed anatomical data, including both the navigated airway and deformed, unprocessed anatomical features from the CT volume, which is valuable for guidance, biopsy, and treatment.

[0298] Reference is now made to FIG. 12, which schematically illustrates a system for tracking the movement of an interventional instrument within a pulmonary airway, according to some embodiments of the present disclosure.

[0299] 12 includes a memory 1201, a processor 1202, and optionally a display 1203. Optionally, an interventional instrument 1204 equipped with one or more sensors 1205 is provided with the system, optionally along with a position sensing controller 1206. The position sensor 1205 may be located, for example, at the tip of the interventional instrument 1204 and / or at a location along the body of the interventional instrument 1204.

[0300] Optionally, all or part of the subsystem including the interventional instrument 1204 and the position sensing controller 1206 are provided separately.

[0301] Selected data structures used and / or generated in the operation of the system of FIG. 12 are shown as blocks in memory 1201.

[0302] These include: Instructions 1212 that instruct processor 1202 to perform any one or more operations of the methods described herein, e.g., with respect to Figures 1, 5, 7A-7B, and 8A. Processor 1202 may generate and / or access any of the other data structures in accordance with instructions 1212. A pre-deformation model 1210, for example corresponding to the model generated in block 110 of FIG. 1 and / or baseline airway map 503 of FIG. An initial alignment 1214, for example, corresponding to the alignment applied by block 504 of FIG. 5 and / or transformation state vector 710 of FIG. 7A. For example, interventional instrument position measurements 1218 corresponding to position data 505 in FIG. For example, dynamic alignment 1216 corresponding to dynamic transformation state vector 712 of FIG. 7A.

[0303] Instructions 1212 may instruct processor 1202 to generate one or more views 1220 representing modified (e.g., initially and / or dynamically aligned) versions of pre-deformed model 1210 and provide them to display 1203. Block 1220 illustrates the views being provided as data streamed to display 1203. Data structures representing these views, and / or data structures representing intermediate data structures, may also be generated by processor 1202 and stored in memory 1201 (not explicitly shown). Views 1220 may be generated, for example, as described with respect to FIGS. 8A-11.

[0304] overview When used herein in reference to a quantity or value, the term "about" means "within ±10% of the following."

[0305] The terms "comprises," "comprising," "includes," "including," "having," and their cognates mean "including but not limited to."

[0306] The term "consisting of" means "including and limited to."

[0307] The term "consisting essentially of" means that a composition, method, or structure may include additional ingredients, steps, and / or components, provided that the additional ingredients, steps, and / or components do not materially alter the basic and novel characteristics of the claimed composition, method, or structure.

[0308] As used herein, the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. For example, the term "a compound" or "at least one compound" can include multiple compounds, including mixtures thereof.

[0309] The words "example" and "exemplary" are used herein to mean "serving as an example, instance, or illustration." Any embodiment described as "exemplary" or "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments and / or excludes the incorporation of features from other embodiments.

[0310] The word "optionally" is used herein to mean "provided in some embodiments and not provided in other embodiments." Any particular embodiment of the present disclosure may include more than one "optional" feature, except to the extent that such features are inconsistent.

[0311] As used herein, the term "method" refers to manners, means, techniques and procedures for accomplishing a given task, including, but not limited to, methods known by or readily developed from known manners, means, techniques and procedures by practitioners of the arts of chemistry, pharmacology, biology, biochemistry and medicine.

[0312] As used herein, the term "treating" includes negating, substantially inhibiting, slowing, or reversing the progression of a condition, substantially ameliorating the clinical or cosmetic symptoms of a condition, or substantially preventing the appearance of clinical or cosmetic symptoms of a condition.

[0313] Throughout this application, embodiments may be presented with reference to a range format. It should be understood that the description in range format is merely for convenience and brevity and should not be construed as an inflexible limitation on the scope of the description of the present disclosure. Accordingly, the description of a range should be considered to have specifically disclosed not only each individual numerical value within that range, but also all possible subranges. For example, a description of a range such as "1 to 6" should be considered to have specifically disclosed subranges such as "1 to 3," "1 to 4," "1 to 5," "2 to 4," "2 to 6," "3 to 6," etc., as well as each individual numerical value within that range, e.g., 1, 2, 3, 4, 5, and 6. This applies regardless of the breadth of the range.

[0314] When a range of numerical values ​​is given herein (e.g., a set of numerical values ​​connected by "10-15," "10 to 15," or another such range designator), it is meant to include any number (fractional or integer) within the limits of the stated range, inclusive of the limits of the range, unless the context clearly dictates otherwise. The phrase "range / ranging / ranges between" a first designator and a second designator, and the phrase "range / ranging / ranges from" a first designator "to," "up to," "until," or "through" a second designator, are used interchangeably herein and are meant to include the first designator and the second designator, and all fractional and integer numbers therebetween.

[0315] While the description of this disclosure has been provided in conjunction with specific embodiments, it is evident that many alternatives, modifications, and variations will be apparent to those skilled in the art. Accordingly, it is intended to embrace all such alternatives, modifications, and variations that fall within the spirit and broad scope of the appended claims.

[0316] It is understood that certain features that are, for clarity, described in this disclosure in the context of separate embodiments, can also be provided in combination in a single embodiment. Conversely, various features that are, for brevity, described in the context of a single embodiment, can also be provided separately or in any suitable subcombination, or as suitable in any other described embodiment of this disclosure. Particular features described in the context of various embodiments should not be considered essential features of those embodiments, unless the embodiments cannot function without those elements.

[0317] It is the intention of the applicants (applicants) that all publications, patents, and patent applications mentioned herein be incorporated by reference in their entireties, as if each individual publication, patent, or patent application was specifically and individually indicated to be incorporated by reference herein. Furthermore, citation or identification of any reference in this application should not be construed as an admission that such reference is available as prior art to the present invention. Section headings, if used, should not be construed as necessarily limiting. Additionally, any priority document(s) of this application are hereby incorporated by reference in their entireties.

Claims

1. 1. A method for modeling deformation states of a branched anatomical structure through which an interventional instrument is navigated, comprising: a. accessing a baseline model constructed using imaged shapes of bifurcations and bifurcation points of the branched anatomical structure; b) accessing a plurality of measurements of the shape and position of the interventional instrument made using the interventional instrument while inserted into the branched anatomical structure, the plurality of measurements including a plurality of shape samples corresponding to a plurality of branches traversed by the interventional instrument along its path; c) calculating a deformation defining a modification of the baseline model based on the plurality of measurements, the modification including a modification of modeled branch segments corresponding to the plurality of branches traversed by the interventional instrument; and (d) generating a deformed model by modifying the baseline model according to the deformation, thereby aligning the interventional instrument to the deformed state of the branched anatomical structure.

2. The method of claim 1 , wherein the calculating uses an error function that defines an error of the correction relative to both the target objective and the shape constraints.

3. 3. The method of claim 2, wherein the target objective is characterized by a deviation of the deformed model from a shape of the branched anatomical structure defined by the plurality of measurements in terms of an error that increases as a function of deviation from the shape of the branched anatomical structure.

4. The method of claim 2 , wherein the shape constraint characterizes an anatomical constraint on the modification in terms of an increasing error as a function of deviation from a baseline model weighted by a deformation model.

5. The method of claim 1 , wherein each measurement of the shape and position of the interventional instrument indicates a curved shape and position of the interventional instrument.

6. calculating and generating a dynamically deformed model that modifies the deformed model based on further measurements of the shape and position of the interventional instrument; The method of claim 2 , wherein the calculating uses a second error function that defines shape constraints in terms of increasing error as a function of deviation from the deformed model.

7. each of said plurality of measurements of shape and position associated with a respective respiratory phase; 10. The method of claim 1, comprising determining deformation parameters of the modeled bifurcation in each of a plurality of modeled respiratory phases using shape and position measurements associated with the modeled respiratory phases.

8. Calculating the deformed model comprises: a. acquiring, during a selected respiratory phase, a plurality of measurements associated with the respiratory phase; b. selecting one or more measurements from the plurality of measurements associated with a respiratory phase that were measured during the selected respiratory phase; c. calculating a respiratory phase dependent correction of the baseline model using the selected measurements; The method of claim 1 , wherein the generating comprises modifying the baseline model to represent a respiratory phase dependent shape of the deformed model.

9. generating the deformed model includes calculating a modeled shape of the branched anatomical structure for at least one respiratory phase by combining modeled shapes for at least two other respiratory phases; The method of claim 1 , wherein calculating the modeled shape comprises interpolating between the at least two other respiratory phases.

10. 2. The method of claim 1, wherein the baseline model and the deformed model each comprise a skeletonized model, and wherein branches and branching points of the branched anatomical structures correspond to modeled branches of each model, represented as segments joined at modeled branching points.

11. each of the accessed plurality of measurements measures a shape of the interventional instrument as a plurality of position measurements at positions along the interventional instrument, each of the position measurements being encoded in a probe localization coordinate system including at least three position axes measured by a sensor at each of the positions; the deformed model aligns the probe's localization coordinate system to the imaged geometry of the bifurcation and bifurcation point through modification of the baseline model; The method of claim 1 , wherein the probe's localization coordinate system includes at least one orientation coordinate measured by a sensor at each of the locations.

12. generating the transformed model includes encoding modifications of the modeled branches into an error function; The error function is a. encoding a localization error of the deformed model relative to the accessed plurality of measurements; 2. The method of claim 1, including anatomical constraints that limit how the baseline model can be modified to reduce the localization error of the deformed model.

13. receiving reference measurements from one or more reference sensors attached to fixed anatomical locations on a body including the branched anatomical structure, wherein the calculating includes using the reference measurements to roughly align a plurality of measurements of shape and position of the interventional instrument relative to the branched anatomical structure; b. receiving reference measurements from one or more reference sensors and using the reference measurements to assign a respiratory phase to each measurement of the shape and position of an interventional instrument.

14. 2. The method of claim 1 , wherein generating the deformed model comprises transforming the baseline model with a rigid transformation determined using measured shapes and positions, and modifying the transformed model according to the deformation model.

15. a. accessing at least one additional measurement of the shape and position of the interventional device made using the interventional device while inserted within the branched anatomical structure; b. modifying the transformed model to reduce an error function, the error function being: i. a target condition that reduces the localization error in the modified deformed model for at least one accessed additional measurement, thereby reducing the error function; ii. A shape constraint that increases the error function with increasing difference of the modified deformed model from the deformed model, the shape constraint being weighted according to the deformed model; The method of claim 1 , wherein said modifying comprises:

16. calculating a reduction of the error function, the error function being: a. converging to a minimized error function value from each of a plurality of initial states to a corresponding convergence goal; b) selecting among convergence objectives for defining a modification of the deformed model according to the minimized error function value.

17. a) the plurality of initial states include initial deformation states distinguished by different deformations of the deformed model; b) the plurality of initial states are distinguished by different hypotheses regarding the path the interventional instrument will take within the branched anatomical structure; c. said converging and selecting comprises performing a global search for a minimum of said error function; d) generating the deformed model includes propagating changes that modify the relative orientation of branches at a parent branch point of the model to changes that modify the spatial offsets and orientations of branches at child branch points of the parent branch point, thus propagating deformations beyond directly measured positions; e. the deformation model models the propagation of local deformations through tissue; The method of claim 16 , wherein at least one of:

18. the deformed model includes a skeletonized model representing branches and branching points of the branched anatomical structure as segments connected at modeled branching points, the deformed model models errors as a function of translation and bending of the segments relative to the deformed model; a. the modeled error varies with changes in the relative angles of the segments at their common bifurcation point; b. the modeled error varies as a function of distally applied fixation force of at least some segments; c) the modeled error varies with bending along the length of at least a portion of the segment; d. the modeled error varies with separation of segments that are not directly connected via a branching structure; e. the modeled error limits segments forming a global shape that is inconsistent with a range of global shapes represented by a dataset including a plurality of global shapes; f. the modeled error limits segments that form branch shapes that are inconsistent with the range of branch shapes represented by a dataset that includes multiple branch shapes; The method of claim 15 , wherein at least one of:

Images