Dynamic deformation tracking during navigation bronchoscopy

Abstract

Systems and methods for tracking catheter movement within the airways of the lung. A deformable model of the lung represents the airways as segments connected at bifurcation points. The lung model is deformed by modifying the angles and / or positions of the airway segments relative to each other. For example, an initial model can be generated based on segmentation of CT images. For example, a baseline deformable registration of the patient's lung shape at the start of surgery can be established based on catheter positions measured along multiple different paths. Optionally, the baseline registration is dynamic with the respiratory phase. Real-time changes in lung shape relative to the baseline registration can be modeled using further measurements obtained with the catheter as it navigates to its target, preferably using position sensors distributed along the catheter body.

Description

[0001] Related applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 123,590, filed December 10, 2020, entitled "Deformable Registration and Dynamic Deformation Tracking and View for Navigated Bronchoscopy," filed under 35 USC §119(e), and U.S. Provisional Patent Application No. 63 / 238,165, filed August 29, 2021, entitled "Deformation Tracking," the contents of which are incorporated herein by reference in their entirety.

[0003] Technical Field and Background Art of this Disclosure

[0004] This disclosure relates, in some embodiments thereof, to the field of bronchoscopy, and more specifically, but not exclusively, to electromagnetic navigation bronchoscopy.

[0005] Systems for navigating and / or monitoring probes moving within the lungs have been proposed and / or marketed, including non-robotic navigation guided by electromagnetic induction of a single sensor with a fluorescent lens; robotic navigation guided by fiber optic shape sensing; and robotic navigation guided by electromagnetic induction of a single sensor. Other systems known in the art include fluoroscopic systems, CBT (cone-beam CT) guided systems, and video registration-based bronchoscopy. Summary of the Invention

[0006] According to one aspect of some embodiments of this disclosure, a method for tracking the positioning of an interventional device relative to a lung airway is provided. The method includes: accessing a skeletal model of the lung airway, the model including multiple airway segments connected at bifurcation points; accessing anatomically defined constraints limiting shape variations in the skeletal model; accessing measurements indicating multiple locations within the lung along the interventional device; and modifying the relative angles and positions of one or more airway segments at the bifurcation points within the model. The modification is calculated by reducing the modeling error between the shape of the lung indicated by the multiple locations and the arrangement of the airway segments corresponding to the multiple locations, and by using anatomically based constraints to limit how the shape of the airway changes, thereby reducing the modeling error.

[0007] According to some embodiments of this disclosure, the modification includes modifying the relative angle of the airway segments connected at the common bifurcation, wherein the interventional device extending between the plurality of locations does not pass through the common bifurcation.

[0008] According to some embodiments of this disclosure, the anatomical-based constraint includes an increase in the relative angle of resistance to the airway at the common bifurcation.

[0009] According to some embodiments of this disclosure, the anatomical-based constraint includes an anchoring force applied from the distal end of at least some of the airway segments.

[0010] According to some embodiments of this disclosure, the anatomical-based constraints include restrictions on curvature along at least some of the airway segments.

[0011] According to some embodiments of this disclosure, the anatomical constraints include restrictions on the individual movement of airway segments interconnected by the parenchyma of the lungs.

[0012] According to some embodiments of this disclosure, the anatomical-based constraints include restrictions on airway segments that form global shapes, which are inconsistent with the range of global shapes represented by a dataset comprising multiple global shapes.

[0013] According to some embodiments of this disclosure, testing the consistency between airway segment shape and global shape range includes applying a machine learning product configured to score the airway segment shape based on the probability that the airway segment shape belongs to the global shape range.

[0014] According to some embodiments of this disclosure, the range of the global shape is derived from measurements of multiple measured lungs.

[0015] According to some embodiments of this disclosure, the range of the global shape is derived from a modified version of the skeletal model, which is pre-calculated and accessed as part of the constraints of the anatomical definition.

[0016] According to some embodiments of this disclosure, the anatomical-based constraints include restrictions on airway segments representing lung portions along multiple locations forming branching shapes, the branching shapes being inconsistent with the range of branching shapes represented by a dataset including multiple branching shapes.

[0017] According to some embodiments of this disclosure, testing the consistency between the airway segment branch shape and the range of branch shapes includes applying a machine learning product configured to score the airway segment branch shape based on the probability that the airway segment shape belongs to the range of branch shapes.

[0018] According to some embodiments of this disclosure, the range of branch shapes is derived from measurements of multiple measured lungs.

[0019] According to some embodiments of this disclosure, the range of the global shape is derived from a modified version of the skeletal model, which is pre-calculated and accessed as part of the constraints of the anatomical definition.

[0020] According to some embodiments of this disclosure, the method includes constructing the skeletonized model based on CT image segmentation of the lung.

[0021] According to some embodiments of this disclosure, the construction includes: skeletonizing the segments of the CT image, identifying bifurcations in the skeletonization, and assigning parameter values ​​representing the relative angles of the bifurcations to the bifurcations.

[0022] According to some embodiments of this disclosure, the measurement of the access is a measurement performed at different times at different locations of the interventional device along multiple different airway trajectories using the interventional device; the relative angle of the airway segment is assigned to the bifurcation to conform to the imaging geometry of the lung; and the modification is made to adjust the relative angle of the airway segment according to the change in lung position between the time when the lung geometry is imaged and the time when measurements are taken along the multiple different airway trajectories.

[0023] According to some embodiments of this disclosure, the measurement of the access is correlated with the phase of respiration through its measurement time; and the modification is based on the respiration phase correlation of the measurement of the access to modify the relative angle to create a skeletal model of the phase change of the lung airway, which represents the shape of the lung during multiple different respiratory phases.

[0024] According to some embodiments of this disclosure, the modification is performed for a new measurement associated with the respiratory phase; and the airway phase change skeletal model is modified based on the location of the new measurement and its associated respiratory phase.

[0025] According to some embodiments of this disclosure, measurements are used to modify the relative angles of airway segments at different modeled respiratory phases based on a weighted average that depends on the respiratory phase associated with each measurement and the differences between different modeled respiratory phases.

[0026] According to some embodiments of this disclosure, the shape of the lung during the plurality of different respiratory phases is represented as follows: the lung is in a first respiratory phase, calculated by modification of the measurement associated with the phase; the lung is in a second respiratory phase, calculated by modification of the measurement associated with the phase; and the lung is in at least a third respiratory phase, calculated by interpolation of the lung shape between the first phase and the second phase.

[0027] According to some embodiments of this disclosure, the shape of the lung during the plurality of different phases is further represented by the lung in a fourth respiratory phase.

[0028] According to some embodiments of this disclosure, the third breathing phase and the fourth breathing phase are equally spaced between the first breathing phase and the second breathing phase, and are different from each other.

[0029] According to some embodiments of this disclosure, the phase of respiration is determined based on the movement of markers attached to the outer surface of the body, including the lungs and their airways.

[0030] According to some embodiments of this disclosure, the method includes displaying an image representing the arrangement of airway segments of a model.

[0031] According to some embodiments of this disclosure, the method includes displaying objects associated with the lung model in the image, wherein the positions of the objects are updated according to changes in the arrangement of the airway segments.

[0032] According to some embodiments of this disclosure, the method includes associating the position of the object with the position of at least one of the airway segments, and moving the object according to the movement of the associated at least one of the airway segments.

[0033] According to some embodiments of this disclosure, the image also represents a slice of a 3-D image of the lung that corresponds to a known skeletal model of the airway of the lung, the slice being selected to extend along a portion of the lung shown, the portion of the lung shown corresponding to the airway along which the interventional device extends.

[0034] According to some embodiments of this disclosure, the known correspondence between the skeletal model of the airway of the lung and the 3-D image of the lung is established by deriving the skeletal model of the airway from segments of the 3-D image of the lung.

[0035] According to some embodiments of this disclosure, the displayed image includes a 3-D representation of the lung, and the slices of the 3-D image are curved out of a planar configuration to conform to the 3-D configuration of the interventional device.

[0036] According to some embodiments of this disclosure, the displayed image represents the interventional device flattened into a planar representation, and the slices of the 3-D image are also flattened accordingly.

[0037] According to some embodiments of this disclosure, the relative angle of the airway segment is assigned to the bifurcation to conform to the geometry of the lung imaged during a first time period, and the relative angle is modified by a change in the geometry of the lung determined based on further measurements taken during a second time period; and the access measurement is performed during a third time period along the interventional device and while the interventional device is held in the same position; the modification adjusts the relative angle of the airway segment according to the change in lung position between the second and third time periods.

[0038] According to some embodiments of this disclosure, the method includes repeatedly: accessing successive sets of measurements; and modifying the relative angle of the airway segment based on each successive set of measurements.

[0039] According to some embodiments of this disclosure, the modification includes: identifying a new value for the relative angle, the new value reducing the error between the shape of the lung indicated by the plurality of locations and the arrangement of the airway segments corresponding to the plurality of locations; filtering the new value based on a previous value for the relative angle to reduce the magnitude of change compared to the previous value; and using the new value to generate a calculated modification.

[0040] According to some embodiments of this disclosure, the filtration increases the error between the shape of the lung indicated by the plurality of locations and the arrangement of the airway segments corresponding to the plurality of locations, compared to the new value identified before filtration.

[0041] According to some embodiments of this disclosure, the error cannot be completely reduced by modifying between each successive set of measurements.

[0042] According to some embodiments of this disclosure, the modification includes modifying the relative angles differently in multiple different copies of the skeletalized model; and includes: repeating the access measurement and modifying each of the different copies using the new measurement; selecting one of the different copies based on a more reasonable representation of the lung shape; and continuing to repeat the access measurement and modifying only the selected copy.

[0043] According to some embodiments of this disclosure, the modification includes perturbing the relative angle of the airway segment in the modification to calculate the perturbed modification; identifying that the perturbed modification reduces more error than the unperturbed modification; and using the perturbed modification to perform the modification.

[0044] According to one aspect of some embodiments of this disclosure, a computer memory storage medium for storing a deformable model of a lung is provided. The model includes: data elements corresponding to: a skeletal representation of a branching structure of the airways of the lung, and a representation of the orientation of the branches of the branching structure for each of a plurality of bifurcations; and computer instructions configured to instruct a processor to modify the model by modifying the representation of the relative orientations of the branches of the branching structure to satisfy provided geometric constraints.

[0045] According to some embodiments of this disclosure, the multiple forks include forks that extend upwards to the third-level branch of the branch structure.

[0046] According to some embodiments of this disclosure, the computer instructions instruct the processor to modify the representation of the direction of the branch of the branch structure without modifying the distance of the segment along the branch structure.

[0047] According to some embodiments of this disclosure, the computer instructions instruct the processor to propagate changes in the relative orientation of branches in a parent fork to changes in the spatial offsets and orientations of branches in child forks of the parent fork.

[0048] According to one aspect of some embodiments of this disclosure, a system for tracking the movement of an interventional device within an airway of the lung is provided. The system includes a processor and a memory storing instructions instructing the processor to: access a 3-D representation of the current shape and location of the interventional device; access a model of an airway segment representing a bifurcation of the airway segment; and match the airway segment to the current shape and location of the interventional device. The processor is instructed to: determine appropriate rotational modifications to the branches of the bifurcation based on improvements to the correspondence between the airway segment and the current shape and location of the interventional device, and apply the modifications to the model.

[0049] According to some embodiments of this disclosure, the rotational modification is applied to a modeled airway segment corresponding to a portion of the airway along which the interventional device extends.

[0050] According to some embodiments of this disclosure, the processor is instructed to access measurements of body movement; wherein the processor is instructed to modify the orientation of the airway segment to modify the airway model to match changes in lung shape corresponding to the measurements of body movement.

[0051] According to some embodiments of this disclosure, the modifications are applied to the model using location data associated with the bifurcation as control points.

[0052] According to some embodiments of this disclosure, the availability of different types of reasonable modifications is weighted separately to improve the matching of the airway model with changes in lung shape corresponding to measurements of body movement.

[0053] According to some embodiments of this disclosure, the availability of different types of rotational modifications is individually weighted to improve the correspondence between the airway segment and the current shape and positioning of the interventional device.

[0054] According to some embodiments of this disclosure, longitudinal bending rotation modification is weighted as being more suitable for improving responsiveness than longitudinal twisting rotation modification.

[0055] According to one aspect of some embodiments of this disclosure, a method is provided for registering interventional device location data to an imaged lung shape. The method includes: receiving a baseline lung airway model constructed based on the imaged lung shape; at a later time, measuring and using one or more intrapulmonary probes along the locations of multiple branches of the airways of the lung, including the locations of at least two different branches of the same bifurcation of the airways; modifying the baseline lung airway model to record the measured locations and producing a deformed baseline lung airway model, wherein the deformed baseline lung airway model represents the lung airways at multiple respiratory phases; determining a transformation applicable to the deformed baseline lung airway model based on measurements indicating the current shape of the lung to adjust it to indicate the current shape of the lung; and displaying an image showing the deformed baseline lung airway model transformed according to the transformation.

[0056] According to some embodiments of this disclosure, each of the measurements of the location of the plurality of branches of the airway of the lung is associated with a respiratory phase, and the association of the measurements with different respiratory phases is used to generate the representation of the lung airway in different phases of the plurality of respiratory phases.

[0057] According to some embodiments of this disclosure, determining the deformation includes: using a respiratory phase corresponding to the current shape of the indicated lung to determine a respiratory phase-dependent shape of a baseline lung airway model of the deformation; and calculating the transformation to transform the determined respiratory phase-dependent shape to the current shape of the indicated lung.

[0058] According to some embodiments of this disclosure, in the modified baseline lung airway model, the shape of the lung during the plurality of respiratory phases is represented as: the state of the lung in a first respiratory phase, which is generated by the modification of the baseline lung airway model; and the state of the lung in a second respiratory phase, which is generated by the modification of the baseline lung airway model.

[0059] According to some embodiments of this disclosure, the measurement includes generating a baseline shape of the lung in a third respiratory phase from the deformed baseline airway model, calculated by interpolating the lung shape between the first phase and the second phase.

[0060] According to some embodiments of this disclosure, the shape of the lung during the plurality of respiratory phases also includes the state of the lung in a fourth respiratory phase, generated by modifying the baseline lung airway model.

[0061] According to some embodiments of this disclosure, each of the third and fourth respiratory phases is: the phase interval between the first and second respiratory phases is similar, and they are different from each other.

[0062] According to one aspect of some embodiments of this disclosure, a method is provided for displaying features along a spiral path through an anatomical structure. The method includes: accessing a 3-D image of the anatomical structure; defining a path through the anatomical structure; calculating: a spatial correspondence between a position along the path and a position within the 3-D image, and a surface extending through the 3-D image, the surface including the position within the 3-D image; and displaying a 3-D display image showing the path extending through 3-D space, and data derived from the position of the anatomical structure in the 3-D image defined by the surface. Wherein: the path is repeatedly defined using measurements of at least one dynamic variation parameter of the shape of the anatomical structure, and for each repetition of the path definition, the calculation and display are repeated.

[0063] According to some embodiments of this disclosure, the surface is calculated as a strip extending along the path on either side of the path.

[0064] According to some embodiments of this disclosure, the strip extends approximately equal distances on either side of the path.

[0065] According to some embodiments of this disclosure, the method includes accessing a 3-D model of a portion of the anatomical structure that has a known spatial correspondence with the path, and includes displaying the 3-D model of the anatomical structure in the 3-D display image.

[0066] According to some embodiments of this disclosure, the 3D model of the portion of the anatomical structure may also change dynamically.

[0067] According to some embodiments of this disclosure, the anatomical structure includes the airways of the lungs.

[0068] According to some embodiments of this disclosure, the path extends through the airways of the lungs.

[0069] According to some embodiments of this disclosure, the measurement of the at least one dynamically changing parameter includes the position measurement of the interventional device located along the path.

[0070] According to some embodiments of this disclosure, the position measurement of an interventional device located along a path includes position measurements from multiple locations along the path.

[0071] According to some embodiments of this disclosure, the measured value of the at least one dynamically changing parameter is correlated with the respiratory phase through its measurement time.

[0072] Unless otherwise specified, all technical and / or scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains. While similar or equivalent methods and materials to those described herein may be used in practice or testing of embodiments of this disclosure, exemplary methods and / or materials are described below. In case of any conflict, the patent specification (including limitations) shall prevail. Furthermore, these materials, methods, and embodiments are illustrative only and are not intended to be restrictive.

[0073] As those skilled in the art will understand, aspects of this disclosure can be embodied as systems, methods, or computer program products. Therefore, aspects of this disclosure may take the form of entirely hardware implementations, entirely software implementations (including firmware, resident software, microcode, etc.), or implementations combining software and hardware aspects, which herein may be collectively referred to as “circuit,” “module,” or “system” (e.g., a method implemented using “computer circuit”). Furthermore, some embodiments of this disclosure may take the form of a computer program product contained in one or more computer-readable media, on which computer-readable program code is contained. Implementation of methods and / or systems of some embodiments of this disclosure may involve manually, automatically, or a combination thereof performing and / or completing selected tasks. Furthermore, the actual instruments and equipment according to some embodiments of methods and / or systems of this disclosure may implement several selected tasks by hardware, software or firmware, and / or a combination thereof (e.g., using an operating system).

[0074] For example, according to some embodiments of this disclosure, the hardware for performing the selected task can be implemented as a chip or circuit. As software, according to some embodiments of this disclosure, the selected task can be implemented as a plurality of software instructions executed by a computer using any suitable operating system. In some embodiments of this disclosure, one or more tasks performed in a method and / or system are executed by a data processor (also referred to herein as a "digital processor," with reference to a data processor operating using digital bit sets), such as a computing platform for executing multiple instructions. Optionally, the data processor includes volatile memory for storing instructions and / or data and / or non-volatile memory for storing instructions and / or data, such as a magnetic hard disk and / or removable media. Optionally, a network connection is also provided. A display and / or user input device (e.g., a keyboard or mouse) are also optionally provided. Any of these implementations is more generally referred to herein as an example of a computer circuit.

[0075] Any combination of one or more computer-readable media can be used in some embodiments of this disclosure. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, but not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any suitable combination thereof. More specific examples (not an exhaustive list) of computer-readable storage media will include: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable optical disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In the context of this document, a computer-readable storage medium can be any tangible medium capable of containing or storing a program used by or in connection with an instruction execution system, apparatus, or device. A computer-readable storage medium can also contain or store information used by such a program, for example, data constructed in a manner recorded in the computer-readable storage medium such that a computer program can access it as, for example, one or more tables, lists, arrays, data trees, and / or other data structures. In this document, a computer-readable storage medium that records data in a form that can be retrieved as groups of digital bits is also referred to as a digital memory. It should be understood that, in some embodiments, a computer-readable storage medium may optionally also be used as a computer-writable storage medium when the computer-readable storage medium is not inherently read-only and / or is in a read-only state.

[0076] In this document, a data processor is referred to as being "constructed" to perform data processing actions, provided that it is coupled to a computer-readable medium to receive instructions and / or data therefrom, process them, and / or store the results of processing in the same or another computer-readable medium. The processing performed (optionally with respect to data) is specified by instructions, the effect of which is that the processor operates according to the instructions. Processing actions may be additionally or alternatively referred to by one or more other terms; for example: comparison, estimation, determination, calculation, identification, association, storage, analysis, selection, and / or transformation. For example, in some embodiments, a digital processor receives instructions and data from a digital memory, processes the data according to the instructions, and / or stores the results of processing in the digital memory. In some embodiments, "providing" the processing results includes transmitting, storing, and / or presenting one or more of the processing results. Optionally, presentation includes displaying on a display, providing audio instructions, printing on a printout, or giving the results in a form accessible to human senses.

[0077] Computer-readable signal media may include propagated data signals containing computer-readable program code, for example, in baseband or as part of a carrier wave. Such propagated signals may take any of a variety of forms, including, but not limited to, electromagnetic, optical, or any suitable combination thereof. A computer-readable signal medium may be any computer-readable medium that is not a computer-readable storage medium and may transmit, propagate, or transfer a program used by or in connection with an instruction execution system, apparatus, or device.

[0078] The program code contained on a computer-readable medium and / or the data used therefrom may be transmitted using any suitable medium, including but not limited to wireless, wired, fiber optic cable, RF, or any suitable combination thereof.

[0079] Computer program code used to perform operations of some embodiments of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, Smalltalk, C++, etc., and traditional procedural programming languages ​​such as the "C" programming language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer, partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)) or to an external computer (e.g., via the Internet provided by an Internet service provider).

[0080] Some embodiments of this disclosure may be described below with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / actions specified in the flowchart illustrations and / or block diagram blocks.

[0081] These computer program instructions may also be stored in a computer-readable medium that can instruct a computer, other programmable data processing device, or other device to operate in a particular manner, causing the instructions stored in the computer-readable medium to produce, among other things, implemented in flowcharts and / or blocks. Figure 1 An artifact of instructions specifying functions / actions in one or more blocks.

[0082] Computer program instructions may also be loaded onto a computer, other programmable data processing apparatus or other equipment to cause a series of operational steps to be performed on the computer, other programmable apparatus or other equipment, thereby producing a computer-implemented process, such that the instructions, which execute on the computer or other programmable equipment, provide a process for implementing the function / action specified in one or more boxes of a flowchart and / or block diagram.

[0083] Brief description of several views in the attached diagram

[0084] This document describes some embodiments of the present disclosure by way of example with reference to the accompanying drawings. Referring now to the drawings in detail, it is emphasized that the details shown are exemplary and are for illustrative purposes in discussing embodiments of the present disclosure. In this regard, the description taken in conjunction with the drawings will make it clear to those skilled in the art how to implement embodiments of the present disclosure.

[0085] In the attached diagram:

[0086] Figure 1 This is a schematic flowchart illustrating a method for generating a lung deformation model according to some embodiments of the present disclosure;

[0087] Figure 2 The lung airway framework is schematically illustrated according to some embodiments of the present disclosure;

[0088] Figure 3 A bifurcated coordinate system (also referred to herein as coordinate system T) is schematically illustrated according to some embodiments of this disclosure. i and bifurcation plane;

[0089] Figure 4 The diagram schematically illustrates the lung airway skeleton before (left) and after (right) deformation between the bifurcations, according to some embodiments of the present disclosure.

[0090] Figure 5 This is a schematic flowchart illustrating a method for performing deformable registration between the location of a duct located in the pulmonary airway and a branching model of the airway, according to some embodiments of the present disclosure.

[0091] Figure 6 The diagram illustrates the state of a lung model registered differently from measurement data according to some embodiments of this disclosure;

[0092] Figure 7A The illustrations schematically depict the determination of deformation transformations of deformation models according to some practical methods of this disclosure to match available data and the elements used in further dynamic modeling;

[0093] Figure 7B The embodiments described herein are illustrated schematically. Figure 7A A flowchart of the implementation method of the block;

[0094] Figure 8A This is a schematic flowchart outlining a method for generating a deformable lung model view according to some embodiments of the present disclosure;

[0095] Figures 8B-8C The illustration schematically depicts the rendering of a 3D mesh before and after the deformation of a single model branch and its offspring, according to some embodiments of this disclosure.

[0096] Figures 9A-9B The illustration schematically shows some embodiments according to this disclosure, where a single branch and all its offspring are before deformation ( Figure 9A ) and afterwards ( Figure 9B x-slice interpolation grid;

[0097] Figure 9C The illustration schematically depicts a CT section extending along a surface defined as a curved geometric cross-section, according to some embodiments of the present disclosure, such that it includes a region of the original 3-D image corresponding to a location along a spiral path through an anatomical structure.

[0098] Figures 10A-10B The illustration schematically shows some embodiments according to this disclosure, including before modification ( Figure 10A ) and afterwards ( Figure 10B The CT strip along the path in the composite-D scene of the rendered mesh view of the deformed model;

[0099] Figure 11Flat stripes of CT image data extracted along a 3-D path following a catheter, according to some embodiments of this disclosure, are shown; and

[0100] Figure 12 A system for tracking the movement of a catheter within the airway of the lung is illustrated schematically according to some embodiments of the present disclosure. Detailed Implementation

[0101] In some embodiments of the present invention, the invention relates to the field of bronchoscopy, and more specifically, but not exclusively, to electromagnetic navigation bronchoscopy.

[0102] Overview

[0103] Some general aspects of embodiments of this disclosure relate to robotic navigation of a probe within a branching and dynamically moving airway in a living lung.

[0104] Pulmonary interventional device tracking

[0105] The complex structure of the lungs presents challenges for navigation guidance in bronchoscopy. Further complicating matters, the lungs are flexible and in dynamic motion, thus positioning systems that do not account for lung movement will introduce inherent errors.

[0106] Using fluoroscopy, CBCT, or EBUS (endobronchial ultrasound) to track probes and / or target localization potentially increases navigation accuracy, while weighing against factors such as additional equipment, increased surgical complexity, and / or potential radiation doses to patients and / or physicians.

[0107] Another method uses an ENB (Electromagnetic Navigation Bronchoscopy) system to determine probe positioning. This system uses a dedicated antenna (or transmitter) to generate an electromagnetic field, which is then sensed using EM (electromagnetic) sensors. The EM sensors can provide 6-DOF (degrees of freedom) position and orientation information of the tip relative to the positioning (transmitter) coordinates. This EM positioning technique has potential advantages for medical applications: for example, the body is essentially transparent to the generated electromagnetic field, and ionizing radiation is not used.

[0108] Traditional ENB systems use tip-positioned EM sensors. Limited by this type of sensor, it is difficult to perform position tracking, such as adequately distinguishing between controlled motion (relative to the lung airways) and motion caused by movement of the lung itself. One approach proposed to overcome this limitation is to use fiber optics to achieve catheter (or other interventional device) shape sensing. Shape sensing allows the interventional device to sense the relative positioning between its own parts. For example, a tip-positioned EM sensor can be used to provide an anchoring position relative to the lung being navigated.

[0109] In this document, the term "interventional device" is used as a general term, more generally referring to flexible instruments comprising a thin, longitudinally extending body, used by advancing them distally through body cavities to reach a distal target. Typically, advancement is performed using distal pressure from the proximal side of the instrument; therefore, even though the instrument is flexible, it possesses a degree of stiffness. Examples of interventional devices include catheters, bronchoscopes, and endoscopes. In this document, these characteristics are cited when the device is described as "similar" to a catheter, bronchoscope, or endoscope.

[0110] In some embodiments of this disclosure, the EM sensing capability is extended to allow tracking to extend proximally from its tip along the catheter body. For brevity, this is also referred to herein as “fully interventional device” tracking. It should be understood that “fully interventional device” does not necessarily refer to the entire length of the device, but rather to a sufficiently long length (e.g., at least 5 cm, at least 10 cm, at least 20 cm) to help overcome the positioning errors and / or ambiguities that may be easily introduced by tip sensing alone. For example, this could be the entire length inserted into the navigated airway, or the entire length up to a reference point (e.g., a reference point established along the trachea).

[0111] Methods for instrument-to-anatomical coordinate registration:

[0112] The lungs comprise a complex, tree-like airway hierarchical system. This complexity presents potential challenges for video navigation; particularly in the periphery of the lungs, where airways become smaller and may appear identical, and the number of bifurcations increases exponentially. Furthermore, image quality can be degraded due to factors such as the presence of mucus in the lungs, making video-based navigation difficult. As is known in the art, surgically useful registration methods can be implemented at multiple stages using multiple information sources, between probe locations and the anatomical structures they traverse.

[0113] Rough registration: In some NB (navigational bronchoscopy) systems, registration is performed between the 3-D localization coordinate (LOC) system and the patient's preoperative CT or MRI. This registration helps manage these difficulties. In this article, "localization system" refers to the coordinate system in which the probe's position is initially described (or "localized"). This position is referred to in this article as being described by the probe's "localization coordinates".

[0114] CT or MRI images contain 3D information about the lungs, particularly the precise location and shape of the patient's airway. The airway is resolved above a certain diameter, which itself depends on the resolution of the CT or MRI scan. In effect, the CT or MRI imager maps the airway, the "road" along which interventional instruments may travel. This map is provided to a navigation system (NB) that uses it to provide navigation instructions to the physician: for example, which turn to take, how far to travel, etc. These navigation instructions can be presented as an overlay on the patient's static preoperative CT scan ("map") and can be presented along with a path showing the route to the target.

[0115] To visualize the true anatomical location of the positioned interventional device within the lung, a corresponding precise registration (3D warp) is applied to map the measured 3D interventional device positioning coordinates to a CT-derived model of the patient's anatomy. One method for constraining and / or calibrating this warp parameter involves attaching reference sensors to the patient at a fixed anatomical location and using these sensors to achieve a coarse registration between the anatomy and the positioning system. The reference sensors move with the patient, so the system knows the body's position in the positioning coordinates. In some systems, it may be unknown how the reference sensors are actually positioned relative to the anatomy, and the reference sensors are used to calculate the LOC to the anatomy registration relative to some individually determined baseline registration.

[0116] Refining rough registration: To produce registrations that help distinguish adjacent airways, rough registration can be enhanced with additional information. For example, the tip of an interventional device within the airway can be used, along with markings from several anatomical references, to refine the registration.

[0117] One approach to this is as follows: During initial video-assisted measurements, the physician visually identifies the major predetermined airway bifurcations and marks them with the tip of a local interventional instrument. The locations of these marks are then collected in LOC coordinates and matched with their anatomical (e.g., CT or MRI image) coordinate counterparts. From this, LOC-to-anatomical registration is calculated. This can be highly accurate in the vicinity of the reference point involved.

[0118] In another approach: the physician can perform unsupervised measurements within the main airways of the lungs using interventional devices. Through these measurements, the physician collects the location of the interventional device in LOC coordinates. Unsupervised registration algorithms, such as those using energy-minimizing optimization methods, match these to the "paths" known from the airway map.

[0119] Once sufficient samples are collected, a unique solution can be found to achieve initial LOC-to-anatomical registration. This provides high accuracy, especially in the lung region being measured (typically the central region). LOC coordinates can be fixed relative to the bed (e.g., at the antenna mounting location) or moved with the patient using a standard reference sensor compensation method.

[0120] Dynamic registration: Understandably, while these methods provide initial registration between LOC coordinates and airway maps, they cannot resolve real-time registration inaccuracies caused by the flexibility and dynamic characteristics of the lungs.

[0121] The lungs change their shape due to reasons such as breathing, changes in body posture, and / or forces exerted on the lungs by interventional instruments moving within the lungs (e.g., bronchoscopes or other endoscopes, catheters, tools used with them, and / or similar surgical instruments). These (and possibly others) cause a discrepancy between the airway anatomy and its initial registration and real-time state relative to the positioning system.

[0122] In some systems, registration is updated based on history. For example, instead of using only a single 6-DOF measurement of the interventional device tip from the current time frame, the path of the interventional device is constructed over a short time window (e.g., a few seconds). The path is then searched within the graph to match several possible “roads.” As a result, instead of simply positioning the tip of the interventional device within the airway, the longer path the interventional device has traversed over time is matched to the graph. For example, if the interventional device “turns right” at a bifurcation (e.g., as seen in the path of the interventional device constructed from its sample history), it is known that the final position of the interventional device cannot be on the left airway; it must be on the right.

[0123] One aspect of some embodiments of this disclosure relates to a parametric lung deformation model that is adjusted by modifying parameters of associated airway segments that affect the lung model, while constraining the modification of those parameters so that the airway segments move within anatomically based constraints.

[0124] The inventors have discovered that by assigning parameters (e.g., up to about 50, 100, or 150; and far fewer than the number of voxels in a CT representation, and optionally even far fewer than the total number of bifurcations in the airway skeleton), the model is generally sufficient to describe the deformations the lung may undergo due to, for example, breathing, changes in patient posture, and / or forces applied by a bronchoscope or other endoscope, catheter, instruments used therewith, and / or similar interventional devices. The position and orientation of the interventional device along its body (i.e., its position and orientation at multiple points along the body of the interventional device when inserted into the lung) can be used as a data source to update those parameters as the lung moves and vice versa. In some embodiments, the interventional device body is traced from its tip to at least the trachea.

[0125] In some implementations, the model itself includes a skeletonized representation of the lung airways. For example, this skeletonized representation can be generated by performing a skeletonization operation on airway segmentation performed on CT or MRI images of the lungs. Deformability is introduced into the lung model by parameterizing the branches of the skeletonized representation; for example, the relative translation and orientation of the segments connected at each bifurcation are parameterized. Optionally, adjustments to the bifurcation parameters in larger (proximal) airways are allowed to propagate to more distant bives, simulating the relative movement of lung lobes relative to each other.

[0126] In some embodiments, constraints are added to the degrees of freedom for adjusting the bifurcation parameters to simulate mechanical characteristics that limit the range of rearrangements that can occur at a single bifurcation. In some embodiments of this disclosure, the constraints provided include one or more of the following:

[0127] • Limit how many segments at a bifurcation can change their orientation (angle) relative to each other. This can include allowing relatively free movement for small angle changes, with increasing resistance to change as the relative angle increases. In some implementations, resistance to angle changes is also specified as a function of airway diameter, airway wall thickness, and / or branch order to reflect the different relative stiffness at different bifurcations. The effect of this constraint on the model can include, for example, forcing some segments to rearrange and bend along their length, rather than bending at the bifurcation itself. In cases involving multiple bifurcations, the model may "prefer" (due to the constraint) to bend more at some bifurcations and less at others.

[0128] • Anchoring is applied from the distal side of the airway segments—that is, away from the side of the airway branching structures that anchor them to the trachea. Some regions of the lung may be partially anchored in place by connections, crossings, and / or constraints that are not part of the airway branching hierarchy; for example, the pleura and diaphragm. The effect of such constraints on the model may be, for example, causing airway segments to deflect their orientation relative to their proximal anchoring, so that they continue to point more or less toward their initial distal position.

[0129] • Airway segments are modeled as lateral mechanical interconnections via the lung parenchyma (i.e., not via the branching structures themselves). This connection can be, for example, transmitted through some airways that are separated from each other proximally by more than one bifurcation within the branching structures. The effect of this constraint on the model could be, for example, a tendency to reduce the airway angles at two or more separate bifurcations, to change their angular orientation, which would cause the segments to either stretch out too much or push them together too much.

[0130] Modeling airway segments along their length offers flexibility. This constraint mitigates strain caused by variations in segment angle bifurcation. It may also help reduce errors, as the curvature of interventional devices along the airway segment can be accommodated by the corresponding parameters of the airway segment in the model.

[0131] It should be noted that the representation of anatomical constraints allows the model to propagate deformations beyond the location of direct measurement. In the context of more comprehensive changes in lung shape, the measured shape changes are perhaps most reasonable (in terms of model configuration) to extend to segments and / or bifurcations away from the current location of the interventional device, propagating to the effects of the constraints.

[0132] Another approach to applying anatomical constraints to the model is to use the results of a machine learning algorithm that learns from inputs representing the anatomically realistic configuration of the lung airways. The inputs can be based on actual lung images, appropriately skeletonized. The arrangement of the airway skeleton in the model is determined based on whether it matches the arrangement used to train the machine learning algorithm.

[0133] In some implementations, machine learning is combined with parametric (e.g., error reduction-based) judgments of the anatomical rationality of the airway skeleton. For the training set, multiple skeletal models of the lung are perturbed (e.g., shapes imposed and / or sensed by models of interventional devices). A high-fidelity error reduction model (defining numerous constraints, e.g., most or all of the types just listed) is applied to the perturbed models. This result may be computationally intensive, but error reduction can be performed offline, so computation time is less critical. Machine learning is then applied to the newly generated training set, e.g., by training methods known in the art. The results of the machine learning can then optionally be used as a method for performing real-time judgments of lung airway shape, possibly encapsulating the results of more computationally intensive methods in a less computationally intensive form. Machine learning can be performed at the level of the entire lung skeleton (e.g., modeling as up to 3, 4, 5, or more branching orders) and / or on selected portions of the lung skeleton, e.g., a single branch occupied by an interventional device, preferably performed together with the modeled interventional device encountering and traversing the next branch.

[0134] This type of representation offers potential advantages for realistic modeling of dynamically changing lung geometry and / or reducing the computational burden involved in updating the model lung geometry to match measurements of the lung itself. In some implementations, updates occur in real time, allowing the model to be used, for example, during dynamic tracking of lung shape during guided bronchoscopy.

[0135] In identifying dynamic parameters of a lung model with bifurcation locations and angles, the model also exhibits potential synergies with lung geometry measurement methods, which themselves readily provide bifurcation angle information. Specifically, as the interventional device advances into the lung, it traverses the bifurcation. Changes in the orientation of the interventional device at distances corresponding to the bifurcation locations of the model provide information about the bifurcation locations and angles.

[0136] If measurements are taken using only the tip of the interventional device, the measured tip position and / or orientation information as an indicator of lung shape may become outdated as the device continues to advance distally. If subsequent lung deformation causes an angular rearrangement at the bifurcation near the current distal position of the interventional device, it may be unclear which bifurcation(s) should be adjusted to bring the lung model back into alignment with the newly moved tip position of the interventional device.

[0137] In some embodiments of this disclosure, this ambiguity is addressed by simultaneously collecting positional data from the longitudinally extending region of the interventional device. Optionally, the longitudinally extending region extends proximally and at least posteriorly from the tip of the interventional device to the first bifurcation of the lung traversed by the interventional device. In this case, the problem of dynamically identifying which bifurcations are deformed and the degree of deformation is limited by data collected in real time from the portions of the interventional device occupying the locations of these bifurcations.

[0138] One aspect of some embodiments of this disclosure relates to a two-stage process of registering a deformable model of a static lung shape to a dynamically changing lung shape, according to some embodiments of this disclosure.

[0139] In some implementations: In the initial deformable registration step, unsupervised measurements (preferably using a fully tracked interventional device) are performed to collect samples in the LOC (location coordinates of the spatially tracked interventional device). The deformable airway map is then fitted to the samples using an energy minimization optimization method. This is an example of a method for providing initial deformable registration between the LOC and anatomical structures.

[0140] Optionally, in the presence of a reference sensor, the instantaneous respiratory phase is assigned to each collected sample during registration. This allows deformable registration algorithms to fit two or more models (e.g., one for "inspiration" and one for "exhalation") to the collected samples. The resulting 4-D position samples (3-D position + breathing phase) are collected in a respiratory interpolation airway map. Optionally, another dimension, such as 7-D, is used: three position degrees of freedom, three directional degrees of freedom, and breathing.

[0141] Following initial registration, a dynamic deformation tracking algorithm adjusts the deformation model in real time to adapt to the deformable lung, providing realistic deformation tracking. Preoperative CT or MRI airway maps are correspondingly deformed and placed in the positioning coordinates to represent the actual state of the airway during surgery.

[0142] The model is fitted to the lung using an energy minimization optimization method. The energy function combines shape constraints (which restrict the solved deformation to a reasonable configuration, determined by a set of constraints based on theoretical and / or observational constraints on how the lung moves) and an objective (which requires one or more localized interventional devices to be located within the deformed airway).

[0143] Deformation algorithms use inputs from any combination of sources from the positioning system used as its target to perform fitting: for example, one or more fully tracked or single-sensor interventional devices, with or without historical data. However, providing deformation tracking algorithms with the real-time, fully tracked position and orientation of one or more interventional devices has specific potential advantages. If that location within the anatomy itself moves, the historical position of the interventional device tip within that anatomy may become outdated. However, the complete shape of the interventional device, "updated" with the anatomy, provides information particularly useful as a constraint on lung dynamics.

[0144] Using an initial registration and a deformable model with dynamic deformation tracking, the airway can then be displayed in its true state during NB surgery. The modeled lungs "breathe" in real time in sync with the patient, otherwise dynamically deforming with the application of additional forces. This allows interventional instruments to be displayed within the airway in their true deformable anatomical position, potentially improving the overall system accuracy.

[0145] It can be noted that modeling deformations occurring at the lung bifurcation may be less constrained (due to the lack of direct, real-time measurement data) than those deformations that are (especially those relatively far from the area through which the interventional device is currently traversing.) However, the effect of this ambiguity may be reduced to a negligible level as long as the lung model is used to guide navigation to a target at the end of the interventional device's current trajectory. For example:

[0146] • Some movements of the lung region away from the navigation trajectory may be irrelevant because they have no impact on the target location.

[0147] Other such movements can affect the target position by applying a deformable force to a bifurcation on (or near) the current trajectory of the interventional device. In this case, a new measurement will detect the change and allow at least the target-containing portion of the lung to be repositioned as needed through appropriate deformation of the model.

[0148] A third possibility is that this deviation from the intended path in lung deformation affects the geometry of the portion of the airway that has not yet been traversed on the way to the target. However, these changes are usually revealed as the interventional device continues to advance, allowing for the continuous detection of geometric changes at more distal airway bifurcations, thus enabling successful navigation to the target. This can be understood to apply more generally to distal deformations.

[0149] In principle, parts of the third possibility could still introduce ambiguity in navigation by deforming the distal bifurcation so much that by the time the interventional device arrives, one branch has already "replaced" the other—that is, the position occupied by one branch is consistent with the current model representation of the other. However, it is foreseeable that mechanical constraints on local tissue deformation will tend to spread such severe deformation over a relatively large longitudinal range of the airway trajectory, thus allowing the assumption that the incremental deformation experienced between two adjacent bifurcations is small enough to avoid confusion between the two bifurcation branches.

[0150] It can be noted that, in any case, even if a fork deforms to the extent that it becomes confused with (and supersedes) the location of other forks, the opposite is unlikely to occur. This is roughly equivalent to a 180-degree twist along a single segment connecting the two forks, which is biomechanically improbable. Therefore, in situations where there is a perceptible risk of ambiguity, alternating between the two branches can help avoid this risk.

[0151] One aspect of some embodiments of this disclosure relates to converting 3-D image data surfaces into a dynamic display corresponding to the current location of the anatomical features they represent, based on real-time updated measurements of the shape of the indicated anatomical features.

[0152] In some implementations, 3D images are acquired at a time prior to the interventional procedure. These images include images of the anatomical region along the path to be traversed to reach the interventional target, and optionally, the target itself. In particular, if the path is defined by the lumen of a branching anatomical structure such as the lung, the path can be swirled and curved in shape through 3D space so as not to be limited by any single plane in the space.

[0153] The shape of the path can also be dynamic; as long as it changes between the imaging time and the time of the interventional procedure itself, and / or even during the procedure. For example, in the case of the lungs, the lungs may change position over time for any of the reasons mentioned above. In particular, during the procedure, respiratory circulation causes continuous changes in the shape of the organ, and interventional instruments can apply forces that cause changes in organ movement and / or deformation.

[0154] Earlier-acquired 3D image data can include features relevant to the context of the ongoing surgery, such as the size, shape, and / or location of the target; and the relative location of potentially vulnerable structures such as blood vessels. For physicians, it is potentially advantageous to show the current and dynamically updated shape of any of these or other structures relative to the path the interventional instrument is traveling and / or is planned to travel.

[0155] In some embodiments of this disclosure, a method for displaying features along a spiral path includes calculating a surface along a given path that defines the positions of data elements in a 3-D image. Data from this surface is positionally transformed to record the current (and dynamically changing) shape of the path. The current shape of the path is determined by measurements indicating its shape, such as the shape of an interventional device occupying the path or a portion thereof, a measurement indicating the current lung expansion state, or another measurement.

[0156] The surface can be displayed (in its transformed 3D shape) to the limits of the available image data, or it can be confined to strips of data locations on either side of the path. In addition to the surface, the path itself can also be indicated. Another indication may be displayed along with the surface, such as a 3D grid indicating the shape of the lumen along which the channel extends (e.g., the shape of the airway, or another lumen anatomy such as a blood vessel, digestive tract, urinary tract, or another anatomical structure).

[0157] In some embodiments, the path is converted into a planar and / or linear path, and image data on either side of the path on the defined surface is shown together with the path in a flat image display. The image data may optionally be taken from locations along a series of lines perpendicular to the path definition at several locations. However, there is no particular limitation that the surface must be flat along the direction perpendicular to the path; it may also be curved in that direction. In some embodiments, the surface is defined as a result of constraints estimated to optimally and collectively satisfy the path shape, such constraints including, for example, portions of important structures (e.g., targets of interventional procedures) and / or potential hazards along the path (e.g., blood vessels).

[0158] Before explaining at least one embodiment of this disclosure in detail, it should be understood that the application of this disclosure is not necessarily limited to the details of the structures and components set forth in the following description and / or shown in the accompanying drawings, and the arrangements and / or methods. Features described in this disclosure (including those of this disclosure) can have other embodiments, or can be practiced or implemented in various ways.

[0159] Deformation Model

[0160] Now for reference Figure 1 This is a schematic flowchart illustrating a method for generating a lung deformation model according to some embodiments of the present disclosure.

[0161] In general, lung imaging data is transformed into a base model (preferably a skeletal representation of the lung airways). For this model, parameters describing how the base model deforms to match the specific condition of the actual lung are applied.

[0162] In some implementations, the deformation model (e.g., as described below) is defined as the result of defining the different possible deformation mechanisms described above using a relatively small number of parameters. Potential advantages of defining the deformation model using a few parameters include avoiding overfitting the model to the observed data and accelerating the optimization process.

[0163] Lung deformation can be viewed as a distributed characteristic, as long as the deformation of one airway is not significantly different from that of its adjacent airways. Global parameters can be defined to describe deformation features, such as: how the lungs expand or contract globally during respiration, how one lung deforms compared to another due to the forces applied by the bronchoscopy, and / or how the two lungs undergo some kind of global stretching or shearing (e.g., due to different patient postures). These deformation features are also important for the overall accuracy of the navigation system. These features are inherently global and can be described with just a few parameters and constraints.

[0164] In some embodiments of this disclosure, the following model is used as the basis for implementing deformation modeling. The inventors have derived it from experimental results using rigid and semi-rigid lung models, animal models, and recorded human data. It should be understood that deformation tracking algorithms can use different models, and are not limited to the specific parameterizations described below. Deformation tracking algorithms are generally independent of the specific parameterizations of the model used; however, good model selection can reduce the risk of overfitting and significantly improve the performance of the tracking algorithm.

[0165] In some implementations, the model is based on airway skeletonization, which provides the centerline of the lung airways. For example, an airway skeleton was generated after preparing the following data:

[0166] At box 102, in some embodiments, during the initialization phase prior to NB surgery, preoperative or early surgical CT, MRI, or other 3-D images of the patient are loaded into the system.

[0167] At box 104, in some embodiments, CT or MRI images are processed. Segmentation methods, such as adaptive region growing, deep learning 2-D / 3-D CNNs (convolutional neural networks), a combination of both, or any other suitable method, are used to segment and label 3-D voxels belonging to the airways (e.g., labeling in computer memory). Optionally, other anatomical features (e.g., blood vessels) are also segmented, although not navigated during NB surgery, but can provide information about the shape of the lung and / or its freedom to change shape in different ways. For example, blood vessels can restrict the bending or twisting of lung lobes in a certain direction or beyond a certain degree. It should be understood that reference is made here to CT or MRI images, which are a more general example of any detailed source of anatomical information about a particular lung. While CT or MRI images can be considered the current gold standard for providing such information, it is not excluded that current and / or future developments in the field of lung imaging may allow the use of another source of anatomical information, such as information based on future advancements in X-ray imaging, magnetic resonance imaging, or other data collection methods.

[0168] In some implementations, the airway segments generated by the operation of block 104 represent binary 3-D volumes, where each voxel is 1 if it belongs to an airway and 0 otherwise. As described above, this provides an initial representation of the airway map information used by the NB boot system.

[0169] At block 108, in some embodiments, skeletonization is performed as a step to determine the tree structure of the airway from the segmented 3-D binary volumes generated from block 104.

[0170] Skeletonization transforms the original airway segment volumes into a tree structure that encodes the airway map. As a result of skeletonization, the 3-D segment volumes of the tubes are converted into graphics with centerlines having radii.

[0171] At box 110, in some embodiments, the skeletonization produced at box 108 is further transformed into a graphical data structure comprising nodes connected by edges. Each node represents a segmented individual tube (a centerline segment, whether straight or curved, along which it has a constant or variable radius). Each edge (oriented or unoriented) connecting a pair of nodes represents the connectivity between the proximal end of one tube and the distal end of another. Thus, the graph of box 110 represents the lung as a tree-like (non-cyclic) graph of tubes.

[0172] The root of the tree diagram is defined as the trachea. The first branch is the carina, while the rest of the airway hierarchy serves as children of the tree. The integrity of the skeleton can be assessed by viewing it as a compressed data representation of airway segments, from which the volume of the segments can be recovered, and a reverse skeletonization transformation to segments can be performed. A good match with the original segments (as found in some cases) means that the skeleton encodes most of the information about the volume of the segments. Due to its much smaller size and greater ease of traversal and analysis, the skeleton offers potential advantages as a representation of airway diagrams.

[0173] Now for a brief reference Figure 2 The diagram schematically illustrates a lung airway framework 200 according to some embodiments of the present disclosure. The framework 200 is shown in terms of thickness (black area 201) and centerline (light ray 202 along the center of the black segment).

[0174] Back Figure 1 At box 112, in some implementations, deformation parameters are defined for the tree diagram. These parameters provide flexibility to the model. The tree diagram deformation parameters are defined for each branch node in the skeletonization, but not necessarily for all such nodes in the tree diagram. In some implementations, nodes are defined for branching sequences up to approximately three or four levels. Nodes can be appropriately defined for higher levels, for example, along the path traversed by the interventional instrument.

[0175] In some implementations, each bifurcation of the skeleton is assigned a coordinate system described by a 4×4 3-D rigid transformation matrix from local bifurcation coordinates to CT coordinates (or in any other suitable coordinate system). Alternatively, the coordinate system can be described using 4-D quaternions and 3-D positions, or another suitable method.

[0176] refer to Figure 3 The diagram schematically illustrates a bifurcated coordinate system 304 (also referred to herein as coordinate system T) according to some embodiments of the present disclosure. i ) and bifurcation plane 305. The mother airway 301 is drawn out from the trachea and divides into sub-branch airways 302.

[0177] Coordinate system 304, for example, located at the bifurcation r i The vertex, its Y-axis y i A plane perpendicular to bifurcation 305. In some embodiments, the plane of bifurcation 305 is defined as containing both the bifurcation point r and the plane perpendicular to ... i It is also the plane that is best aligned with all the airway branches connected to the bifurcation. Its Z-axis z i It can be defined as pointing to r i The direction of the airway is the endpoint, and it is also forced to lie on the bifurcation plane, making Z... i ⊥y iThen the remaining X-axis can be defined as x. i =y i ×z i To form a right-handed coordinate system.

[0178] In addition, the first (root) vertex in the tree is also assigned a coordinate system T0, where the Z-axis points in the direction of the trachea's origin. The Y-axis can be chosen arbitrarily, but for simplicity, it can be chosen as the nearest vector perpendicular to Z and aligned with the Y-axis of the trachea's end.

[0179] As described above, a coordinate system {T} with a specific orientation selection i The specific structure of the branch facilitates the representation of each branch, but the deformation model is invariant to the specific structure and can use any coordinate system centered on the branch, or more generally, it reflects the deformation state of the branch and fork.

[0180] Each 3D transformation can be derived from the position vector r. i And 3 Euler angles (α, β, γ), position vector and 3×3 rotation matrix R i And position vectors and 4-D quaternions q i This is used to represent the transformation. The set of all coordinate systems forms the hierarchy of transformations. To represent the deformation of the airway diagram, the proposed model uses bifurcation as control points (or joints). It describes the deformation applied at the airway bifurcation and uses interpolation methods to calculate the deformation within the airway. Let... Where p(i) is the parent branch of i (or the root of the tree), and U0 = T0 is defined. i Let {T} represent the i-th bifurcation coordinate system in its parent bifurcation coordinate system. Then we can write {T} i}:T i =U0...U p(p(i)) U p(i) U i The chain multiplication rule is the product of all relative transformations from the i-th branch to the root node between a branch and its parent branch. This is achieved through U... i When deformation is applied, all offspring branches are also affected in a tree-like manner.

[0181] Now for reference Figure 4 This schematically illustrates, according to some embodiments of the present disclosure, the lung airway skeleton before (left 410) and after (right 411) deformation between bifurcations 402 and 401. This deformation is achieved by modifying the corresponding U... i A rotation is applied, altering the positioning at fork 401 and all its offspring. Other forks specifically shown include fork 403, which is unaffected by deformation.

[0182] Use U iThe applied deformation also affects all offspring of the modified fork 401. This makes sense in most cases where deformation is applied to the lung. For example, in the case of a slight rotation of one lung relative to another, this can be achieved by rotating a single U... i To reflect, for example, Figure 4 As shown.

[0183] In some cases, it may be necessary to apply the transformation only to a single branch without affecting its offspring. This can be achieved by directly modifying the corresponding T. i This can be easily achieved. However, in the following description, U is used. i The deformation is applied because the inventors have experimentally found that tree-like behavior better captures the deformability of the lungs. Furthermore, this change can be encoded with fewer parameters. Then, the remaining crosspoints {T} are calculated using the chain multiplication rule described above. i The changes in}.

[0184] U i，0 This represents the original bifurcation transformation relative to its parent node (which can be from the CT or from the initial state calculated from the initial deformation). U i，t This represents the deformation bifurcation transformation at time t.

[0185] To model lung deformation, the deformation transformation U can be performed in the following way. i，t Applied to Figure 4 Modify the bifurcation coordinate system represented in the state of 410 on the left side: U i，t =ΔU i，t U i，0 Then use ΔU i，t Describe the deformation in the parent coordinate system. If ΔU i，t =I4 (identity matrix), then no deformation is applied to the i-th bifurcation (relative to the initial state). However, for example, if ΔU i，t =R ω (around the axis of rotation) (Rotation matrix of |ω| degrees), then U i，t The i-th branch (and all its offspring) is deformed by rotating ω relative to p(i), such as... Figure 4 As shown.

[0186] Using the bifurcation coordinates ΔU i，t The deformation is applied and forced to zero or a small offset, producing a deformed model where the distance between bifurcations and the angle between branches are almost completely preserved. For example... Figure 4 As shown, deformation is applied to the model along the length of the airway without affecting the total distance and angle at the bifurcation. If offset is not allowed, ΔU can be described using three Euler angles. i，t If offsets are allowed, a total of 6 parameters are used to describe it.

[0187] In some implementations, interpolation of points inside the branch is performed in the following manner: Let v represent the curve describing the initial shape of the i-th branch, which starts at the bifurcation p(i) and ends at the bifurcation i (for each bifurcation, there is a one-to-one correspondence between bifurcations and branches by considering the branches that end at the bifurcation). i (σ) is given in CT coordinates. v i (0) = r p(i) Let v represent the initial position of the bifurcation p(i). i (1) = r i This indicates the initial position of the branch i.

[0188] Given the bifurcation transformation {T} i，t In the case of}, it is necessary to calculate v. i，t (σ) represents the deformation curve of the i-th branch in the CT coordinates at time t. This represents the deformation at time t, from CT to the i-th bifurcation of the deformed CT coordinates.

[0189] Based on these definitions, the following interpolation formula is applied in some implementations:

[0190] v i，t (σ)=interp(ΔT p(i)，t ΔT i，t ,σ)(v i (σ))

[0191] Where "interp" can be any interpolation function between two 3-D rigid transformations. For example, "interp" can break down a transformation into rotation and translation, and process each part separately. Interpolation for rotation can be implemented using quaternion SLERP (spherical linear interpolation) or any other suitable method, while interpolation for translation can be done by a simple weighted average.

[0192] It can verify v i，t (0) = r p(i)，t The deformation position of the p(i)th bifurcation; v i，t (1) = r i，t The deformation position of the i-th bifurcation. This indicates that the interpolation formula treats the start and end points of the branch curve as expected values.

[0193] In short, Figure 1-4 The deformable model described in the paper assigns coordinate systems to the bifurcations of the skeletal lung airway model, with each bifurcation represented as {T}. i Then, it uses {T}. i，t} Describes the deformed bifurcation at time t. The bifurcation can be arranged in a tree hierarchy and the deformed {U} can be used with the chain multiplication rule. i} and its variant counterpart {U i，t} to calculate. ΔU i，t It provides a convenient representation of the deformation of a bifurcation in the coordinates of its parent branch.

[0194] After branching, the branch curves are interpolated using the start and end points of the bifurcation. Using only a few parameters, the model describes most of the natural deformations that occur in the lungs during NB surgery. This model can be used for real-time dynamic deformation tracking and respiratory compensation, as shown below.

[0195] Deformable registration

[0196] Now for reference Figure 5 This is a schematic flowchart illustrating a method for performing deformable registration between the location of an interventional device in the pulmonary airway and a branching model of the airway, according to some embodiments of the present disclosure.

[0197] At box 502, in some embodiments, as described above, a physician performs unsupervised measurements using one or more single-sensor or fully tracked interventional instruments. Samples are captured in LOC coordinates; their location within the anatomical structure is unknown and will be determined through a registration process. The location data 505 generated by the measurements can be used for operations at box 504, particularly those at box 504A.

[0198] The baseline airway map 503 consists of at least a partial representation of some previously measured lung geometry, for example, as per [reference to...]. Figure 1 It is generated as described. Although the lungs measured in box 502 and represented by baseline airway map 503 are the same lungs, there may be geometric variations between the time when the data for baseline airway map 503 was acquired and the measurement of box 502.

[0199] At box 504, in some implementations, a rigid registration is determined between unsupervised measurement data and airway maps (e.g., generated from previous imaging, such as using CT). Box 504 is divided into boxes 504A and 504B, which can be performed simultaneously or sequentially, optionally iteratively. In box 504A, the airway map is modified to improve its similarity to the current anatomical state. Box 504B represents a rigid registration (e.g., translation and rotation).

[0200] Algorithms for registration in box 504B are known in the art. They perform rigid registration determination using energy-minimization optimization methods, typically starting with unsupervised measurements performed with a single sensor. However, without additional modifications (as provided in box 504A), airway maps generated from preoperative CT may not accurately reflect the anatomy at the time of surgery: distortions are likely to exist between the airway map from preoperative CT and the lung state during actual surgery. Potential reasons include, for example: preoperative CT (or other preoperative imaging procedures) are typically performed days or even weeks before surgery; the patient is examined in a posture altered relative to the CT scan (or other baseline imaging procedure); and / or forces applied to the lungs by a bronchoscope or other endoscope, catheter, tools used with it, and / or similar surgical instruments that alter their shape.

[0201] Therefore, in some embodiments of this disclosure, other operations, indicated by box 504A, adjust the deformable model to better suit finding rigid registration between the LOC and the anatomical structure. The anatomical structure is initially represented by a deformable model with a “zero” deformable model applied relative to CT (and the state of the lungs is represented as baseline airway map 503 during CT). During registration, the deformable model is modified by selecting deformations to compensate for changes in lung state between the time of CT or other 3-D imaging and the time of NB surgery.

[0202] Box 506 represents the initial registration state of the deformable model (airway map), which can continue to change during surgery as the lungs undergo further dynamic changes.

[0203] In some embodiments of this disclosure, the selection of variations in the airway mapping parameters (modifications) used in block 504A is based on position and orientation sensing data obtained from one or more fully tracked interventional devices positioned in and / or moving within the lung. Position data from the tracked devices is provided to block 504A from block 505. For example, fully tracked interventional devices provide more data, particularly data that more directly indicates the longitudinal extent of the interventional device's position at the current moment (e.g., a timestamp), compared to interventional devices that are tracked only at the tip.

[0204] Because the lungs move during surgery, the interpretation of old position data can be ambiguous or misleading, making it less useful than data that clearly indicates the current position of all or most of the interventional devices. One type of problem with old data is that once the lungs have moved, the assumption that the interventional device position indicated in the old data is still within the original airway is invalid (and at least questionable). To correct this, some information about how the lungs have moved is needed. However, in the case of only tip tracking, no sensor can provide this information. On the other hand, for fully tracked interventional devices, the movement of the portion of the interventional device near the tip can faithfully represent the movement of the airway itself. Furthermore, the connection between old and new data is evident because the interventional device itself constitutes a constant "ruler." Therefore, comparing old and new data can directly indicate what type of lung motion may have occurred.

[0205] Now for a brief reference Figure 6 This schematically illustrates lung model states 602 and 602A, which are registered differently from measurement data 603, 603A, and 603B according to some embodiments of the present disclosure. In this example, samples of measurement data 603 are collected using a fully tracked interventional device. During the transition from state 602 to state 602A, regions 608 and 609 have, for example, changed their positions relative to the rest of the lung model.

[0206] Panel 610 (left) shows the original measurement location 603 roughly overlaid on the lung model 602, resulting in a significant degree of misalignment with the measurement data 603. Panel 620 (middle) shows the result of rigid registration of the measurement data 603A with the lung model 602. This is similar to the result of performing box 504B without prior processing of the model via box 504A. This results in errors, particularly in the more distal portions of the lung. Panel 630 (right) shows the lung model 602A, which has been transformed by deformation guided by the measurement data 603B, thus correcting the rigid registration errors seen in panel 620.

[0207] Back Figure 5 Discussion: In some embodiments of this disclosure, the baseline airway map 503 is constructed as about Figure 1 An example of a deformable model is described. To capture the deformation and rigidity registration between the LOC and the anatomical structure, for registration purposes, {T} is used. i It is convenient to consider} as a bifurcated coordinate system in the LOC coordinate system. In this sense, U0 = T0 encodes the rigid transformation from the anatomical structure to the LOC (root node), while {U i} i＞0This represents the local transformations and deformations between bifurcations, as previously described. This single model encodes the rigid anatomical structures to LOC registration (U0) and local CT deformations.

[0208] In some implementations, the optimization process for box 504 is configured to fit a model that performs the following operations: (a) placing all collected interventional device (LOC) samples within deformed and altered airways (essentially performing the operation of box 504B), and (b) describing the reasonable deformation of the lungs that allows for such operation (the operation of box 504A). These two conditions—the target constraint and the shape constraint—are optionally encoded in a single energy function. Alternatively, multiple energy functions are used in an iterative alternation to perform the fitting.

[0209] To prevent overfitting and make convergence more robust, generational bifurcations larger than a certain threshold (e.g., larger than the third generation) can optionally be omitted from the model. Optionally, one or more larger bifurcation generations are included in the fitting calculation only after the initial fit with lower generational bifurcations has converged to a solution. Optionally, interventional device location measurements (samples acquired by measurements) are preprocessed or weighted to balance their contributions to the fit.

[0210] To encode the rigid portion of the transformation from the locally deformed airway map in CT coordinates to the LOC coordinates, U0 can be assigned across six parameters (6-DOF). This is used as part of the operation in box 504B. Optionally, the parameter values ​​are restricted to a limited (reasonable) search region. In some implementations, U0 is searched in an initial global search step, as is done in other such algorithms. The relative scaling of the location data 505 and the baseline airway map 503 can be well characterized from the conditions of data acquisition; optionally, a scaling factor is included in the transform fitting.

[0211] Now for reference Figure 7A This schematically illustrates the determination of deformation transformations of a deformation model according to some practical methods of this disclosure to match available data and the elements used in further dynamic modeling. Regarding Figure 7A The description outlines the use of Figure 5 More specific implementations of some operations are described, and boxes are added to illustrate how the results of this implementation can be further used to model the dynamics of the lungs during surgery.

[0212] In some implementations, via {ΔU i} i＞0 Local deformation is performed on the baseline airway chart (e.g., by the operation in box 504A) and also on the baseline airway chart (box 503, as in box 504A). Figure 5(As shown). Optionally, only 3-DOF (if offset is not allowed) or 6-DOF (if offset is allowed) can be used for definition. The parameters of the model can be collected into a single state vector x = (U0, ΔU1, ΔU2, ..., ΔU...). N In the ), the matrix will be represented in its compact 3-DOF or 6-DOF form, and i = 0, 1, ..., N are the branches involved in the optimization process.

[0213] Box 706 represents the shape of the bifurcation, which is at least in the form of the rotational state assumed by the branches of the bifurcation in the model. Box 708 represents the shape constraints. Optionally, these constraints are defined for each bifurcation to limit and / or guide how the shape of box 706 is modified. For the i-th bifurcation (i > 0), the shape constraints can be simply encoded using the following energy function: This limits the optimization to finding only small and localized deformations between bifurcations. Alternatively, shape constraints can be specified in another way, for example, by applying a set of machine learning scoring weights to a specific model configuration to score its energy function.

[0214] Furthermore, the bifurcation can be determined based on its generation or its characteristic radius. The allocation of weights makes it more or less susceptible to distortions in the fitting process. For example, a deeper bifurcation with a smaller radius can make the main ridge harder.

[0215] above The definition includes simplification by mixing rotation matrices and translations of arbitrary units. Alternatively, in some implementations, ΔU is... i Divide into a 3×3 rotation matrix R and a translation vector t = (x, y, z). Each can be compared with I3 and 0 respectively, and each difference is assigned its own weight.

[0216] Alternatively, consider R-I3: In some implementations, the rotation is in quaternion form q = q w +iq x +jq y +kq z Representation and analysis. Then, the shape energy function will be read accordingly. Force (q) x q y q z The rotation approaches 0, i.e., zero rotation. Optionally, in ΔU... i Different types of rotations can be distinguished. For example, this can also be achieved by analyzing q and assigning different weights to different types of rotations (e.g., making model segments easier to bend at bifurcation points but harder to twist). To reflect this distinction, [the text abruptly ends here]. Figure 1 The coordinate system described {T i The specific construction of} may be very suitable. Optionally, fitting weights are chosen to favor offsets in one direction and / or prevent offsets in other directions. From these examples, it can be understood that... The implementation can take any of several different forms, imposing shape constraints on the deformation applied to a specific bifurcation to reflect the actual lung deformation configuration.

[0217] The optimization process is independent of the choice of a specific energy function. Then, the overall shape energy function is:

[0218]

[0219] To achieve the two objectives of box 504 together—namely, to find rigid transformation (box 504B) and local deformation (box 504A) such that position data 505 (e.g., from an interventional device sample collected during the measurement movement of the interventional device) is within the deformed and transformed airway—in some embodiments, the distance between the interventional device sample collected on one side and the airway tube deformed and transformed on the other side (in LOC coordinates) is taken into account.

[0220] When a sample from location data 505 is located exactly inside the airway duct defined by the airway map (baseline airway map 503, or another modified airway map), it should not introduce any error into the optimization. However, if the sample is located outside the boundary defined by the branch airway model, the model and / or rigid registration should be corrected to minimize this error.

[0221] Depend on This represents the collected interventional device position samples of position data 505, which are transformed from local interventional device or sensor coordinates to location coordinates (LOC) through a 4×4 3-D transformation. Alternatively, the interventional device position samples may be described as 4-D quaternions and 3-D position vectors, or in any other suitable representation.

[0222] Depending on the type of interventional device location data available, the use of the collected interventional device location samples can be optionally varied. For example, when implemented using a 5-DOF tracking system, each S i The first and second axes of the rotation matrix in the image may not exist. When used with a single-sensor interventional device, each S... i Both represent the tip of the interventional device at a certain point in the registration measurement. Using a fully tracked interventional device positioning system, the length of the system is continuously visible. In this case, S can be generated by dividing the fully tracked interventional device along its length into a series of discrete transformations (e.g., using splines or other interpolation methods). kTo generate many S at each moment (timestamp) k This optimization process is independent of the structure of the interventional device position measurement (single sensor vs. full tracking) and the number of interventional devices being tracked. It only requires a set of {S} k} as input. k This represents each S k The translation (position) vector. For a given sample S k The sample energy is defined as follows:

[0223]

[0224] Among them W k The weights are assigned to a specific k-th sample; Skel(x) represents the airway map skeleton deformed by the deformable model state vector x, as described in detail in Section (3); Γ(v, Skel) is a function of the shortest distance or offset between the output position vector v and the skeleton model Skel. When x represents the true deformation and transformation, for each k, The expected value is 0. W can be selected in various ways. k For example, using fully tracked interventional devices, at each time stamp, the fully tracked interventional device is divided into multiple discrete {S}. k Their corresponding W k The tip of the interventional device can be assigned a larger weight relative to its tail to allow the optimization to focus on deeper airways when searching for deformable registrations. For the choice of Γ, a simple distance function between v and the deformable skeleton Skel(x) can be used, but the optimization will then attempt to fit all collected samples to the centerline of the skeleton, which is not necessarily the case. To avoid this, the radius of the deformable skeleton needs to be considered. A normalized distance function can be used, which finds the closest normalized (dimensionless) distance by dividing the distance by the corresponding radius to the nearest point on the skeleton; then the normalized distance is measured in 1-radius, 2-radius instead of [mm], [cm], etc. Normalized distances less than 1 can then be considered as points inside the airway. Therefore, radius-normalized distances can undergo a transfer function such as SmoothReLU, whose offset treats all radius-normalized distances less than 1 as close to 0, and for radius-normalized distances greater than 1 as linearly increasing. For samples off-center, the optimization will still have some small errors, but these are negligible. More generally, a normalized distance can be defined... Here, Γ now sees the full 3D transformation of the k-th sample and finds the shortest distance between the full 3D transformation and the deformed skeleton. This formula allows Γ to include the directional distance between the interventional device sample and the deformed tree, for example, requiring the orientation of the interventional device to align with the orientation of the nearest skeleton point. Furthermore, unlike simply outputting the shortest normalized distance between the interventional device sample and the deformed tree, Γ can output a full difference vector in 3D, or two difference vectors in 3D—one for position and the other for orientation. It should be understood that optimization with Γ and The specific selection of these parameters is irrelevant; they should only be selected such that the airway will be drawn to the sample, preferably with a normalization dependent on the airway radius and a nearly uniform cost within the airway duct. The total sample energy function is:

[0225]

[0226] The overall energy function is:

[0227] E(x)=(W smp E smp (x), W shape E shape (x))

[0228] Among them W smp and W 形状 A balance is provided between sample airway deformation and satisfying shape constraints, and the choice is made experimentally. E(x) is represented as a multidimensional error vector, where each component describes a specific type of error. The goal of the optimization process is to find the state vector x that minimizes the total error ||E(x)||. Various nonlinear methods can be used for optimization, such as gradient descent, Nelder-Mead, trust regions, Levenberg-Marquardt (LM), etc. Some optimization methods (e.g., LM) utilize the representation of the error function as a multidimensional vector of error to accelerate and improve the robustness of their convergence process. For example, LM optimization uses the Jacobian matrix of the error vector E(x) to achieve convergence. The Jacobian matrix of E(x) can be calculated numerically or analytically and used in LM or other types of nonlinear optimization algorithms.

[0229] The output of the deformable registration process is a state vector x (box 710), which describes the transformation and deformation model {ΔU} that can be used as the initial state for NB surgery. i} i≥0The parameters. Using fully tracked interventional devices, it is possible to collect a sufficiently large sample to fit a flexible deformable model onto the collected interventional device sample, obtaining an improved fit, for example, as regarding... Figure 6 As stated above.

[0230] Dynamic deformation tracking

[0231] Figure 7A Deformable registration output set (Transformation state vector 710), its encoding is from anatomical structure to Rigid transformation and CT in its initial state Local deformation below. During NB surgery, the baseline airway map is transformed / deformed to reflect the updated (e.g., current and real-time) state of the airway map 720 in the LOC coordinates.

[0232] In some implementations, the updated registration airway map 720 is generated by... Description: That is, the total transformation and deformation of box 710, along with the additional dynamic transformation state vector 712 encoding dynamic changes, are applied to the baseline airway diagram 503. More specifically: Static portion Derived from deformable registration, and describes the airway map transformation in its initial deformable state during surgery. Dynamics section. The data is updated dynamically during surgery to account for the dynamic forces that deform the lung.

[0233] In some implementations, breathing deformation is removed from Excluded from the middle and instead integrated into (For example, as discussed in the section on respiratory deformation).

[0234] At the start of the NB procedure, after generating the initial registration airway map 506, and That is, the baseline airway diagram 503 is transformed only by the transformation state vector 710. This is achieved by comparing it with existing... Combining adds additional deformations to the model. It can also be written as... To indicate from It begins, and during surgery, local deformation occurs over time.

[0235] The dynamic deformation tracking algorithm implemented by the operation of box 720 is similar to the deformable registration implemented by the operation of box 504.

[0236] At each of the multiple time points t, a new set of interventional instrument samples provided by the 505A as location and orientation data is available.

[0237] In the case of single-sensor interventional devices, K = 1. For fully tracked interventional devices, many S values ​​can be given. k For example, the representation of the interventional device can be further decomposed into a series of discrete transformations along its length by sampling each sensor along the interventional device and / or by using splines or other interpolation methods.

[0238] Similar to the initial deformable registration that generates the initial registration airway map 506, the deformation tracking algorithm of box 720 is independent of the number of interventional devices used and the configuration of each device. The dynamic deformation tracking algorithm of box 720 is configured to calculate... The deformation model is updated such that for each time t, most samples from location data 505A will be inside the airway, and the applied deformation will describe a reasonable lung deformation as described above. In some implementations, this includes jointly minimizing two types of "energy" defined by the total distance between the sample and the airway and by the unreliability of a certain configuration, such as by the shape constraint used in box 720, which is similar to the shape constraint 708 used in box 504.

[0239] However, compared to the deformable registration algorithm implemented by box 504, the deformation tracker of box 720 does not have as many interventional instrument samples in its disposal.

[0240] In some implementations, deformation tracking uses only the current position of the fully invasive instrument at the current time t. In some implementations, deformation tracking also uses past samples from location data 505; for example, by adding them to {S}. k In this context, the weights decrease as the data ages.

[0241] Some embodiments of this disclosure aim to provide continuously updated deformation tracking at a fast (real-time) rate and low latency. This objective is achieved in part by incrementally updating the deformation model to match new data obtained from location data 505A. This incremental updating aligns with the goal of low latency, as the actual deformation rate of the lung is relatively slow compared to the sampling rate of location data 505A.

[0242] In some implementations, the deformation tracker of box 720 reuses the definition of the energy function from the deformable registration in box 504 to define the energy of the dynamically deformed portion of the model:

[0243]

[0244]

[0245]

[0246] in:

[0247] ·x t This is the model's state vector at time t, currently only including the dynamic part:

[0248]

[0249] ·x reg It is a deformable registration state vector.

[0250] · It is a shape energy function, which optionally now focuses only on the dynamic part; that is, it does not penalize the registered shape, but only the additional dynamic shape (although optionally, the initial shape can be considered).

[0251] · It is the energy function of the sample, which is a measure of the distance between the interventional device sample and the final transformed and deformed skeleton (including registration and dynamic deformation).

[0252] In some implementations, the dynamic deformation tracker assumes that the shape derived from the deformable registration is real; that is, it does not include it in its shape energy function. It only searches for local deformation corrections. It will approximate a sample of the interventional device and will not deviate too much in shape from the original shape defined by the deformable registration performed at frame 504.

[0253] There is a risk of overfitting when the information available in each iteration of a dynamic deformation tracking algorithm is limited. In any case, convergence and runtime should be fast to maintain real-time updates.

[0254] In some implementations, these criteria are helped to be met by limiting the attention of the tracker algorithm implemented at box 720 to the bifurcation near the interventional device.

[0255] For example, assuming the tracked interventional device is located inside the right lung, the goal is to minimize ||E dyn (x t Deformation of the bifurcation of the left lung would be futile, as it has no significant effect on total energy.

[0256] In some implementations, the optimizer used when recalculating the dynamic transformation state vector 712 only modifies x. t The bifurcation that may affect the energy function, i.e., the bifurcation that is sufficiently close to the interventional device being tracked. These may include the bifurcation that the interventional device is currently traversing, and optionally along with bifurcations selected based on some distance metric (e.g., branch distance and / or spatial distance).

[0257] Similarly, some implementations of the deformable registration algorithm implemented at box 504, x t It can be restricted to a branching point up to a certain generation. Alternatively, based on, for example, a generation or a feature radius, it can be... The optimization assigns weights to the forks.

[0258] The Γ distance function is used to correlate the sampling location of interventional devices with the transformed and deformed skeleton Skel(x) t x reg Matching them so that calculations can be performed. This can be computationally expensive. To avoid considering the entire tree, irrelevant parts of the tree (e.g., those far from the location of the interventional device sample) can be omitted from the search to achieve some speedup. For example, this can be achieved by considering skeleton points restricted to a certain radius from the interventional device sample and forming a sub-skeleton with them and all their ancestors for Γ distance calculation.

[0259] Now for reference Figure 7B It is illustrated schematically according to some embodiments of this disclosure. Figure 7A A flowchart of the implementation method for the block. When continuously acquiring interventional device position data, Figure 7B The operation is performed iteratively to maintain an updated lung model during surgery.

[0260] At box 730, in some implementations, the deformation tracker accesses a new set of interventional device samples {S}. k} and weight W k For example, the weights can be adjusted so that the distal tip of the interventional device has a stronger weight than its proximal portion.

[0261] At box 732, in some implementations, the tracker then bases its operation on {S}. k The proximity of the nodes is used to select a group of bifurcations and related sub-skeletons.

[0262] At box 734, in some implementations, the model state vector utilizes the data from its previous iteration t0: Initialize it using the state.

[0263] At box 736, in some implementations, for example, a nonlinear iterative optimization method (e.g., the Levenberg-Marquardt (LM) algorithm) is used to perform one or more convergence steps to update x. t To minimize the error function ||E dyn (x tThe number of convergence steps can be chosen to avoid overusing available computational resources. It can be quite low—even just one step. Since real-time operation of the algorithm helps ensure that there should be no significant changes between sampling times, the model should generally be close to its convergence state in any case.

[0264] At box 738, in some implementations, the resulting x can optionally be... t Filtering can be used to force the tracker to produce a smoother response; for example, by using:

[0265]

[0266] α controls the amount of filtering to be performed between the previous timestamp t0 and the deformed state of the new timestamp t (0 - no filtering, update immediately; 1 - full filtering, no update).

[0267] Alternatively, dynamic deformation can be limited by using an optional damping phase applied after each update step: β controls the amount of damping to be applied (0 - no damping, full update; 1 - fully damped, no update). By increasing the damping (β > 0), the dynamic deformation always relaxes and returns to zero. This only applies to interventional device samples {S}. k The deformation tracker will only maintain a non-zero local deformation transformation when the local deformation is compared to that of the deformable registration.

[0268] It should be noted that the shape energy function Dynamic deformation can also be implicitly pushed toward the identity matrix I4, unless While resistance is provided, the explicit damping stage can help accelerate and better control the damping process.

[0269] The remainder of this section deals with the details of the error minimization calculation for box 736.

[0270] Performing only the local convergence step can lead to the algorithm getting stuck in ET. dyn (x t The risk of local minima, where local convergence algorithms (such as LM) cannot easily recover.

[0271] This can happen, for example, when the interventional device is located near a deep and highly deformed bifurcation and / or its deformation is near a bifurcation that was not modeled during the deformable registration step.

[0272] From such a bifurcation point, it may not initially be entirely clear whether the interventional device turns right or left at the bifurcation. In the example described below, it is assumed that the interventional device turns right at the bifurcation point in a fuzzy model. A potential advantage of a fully tracked interventional device combined with the deformable model described herein is that, after entering a specific airway and traveling a short distance in parallel, the shape of the interventional device itself teaches the system whether the interventional device turns right or left, even in the case of significant deformation. For example, after turning right, ||E dyn (x 右 )||<||E dyn (x 左 )||, where x 右 x 左 This represents the state vector considering dynamic deformations of right turns and left turns respectively. However, since deformation tracking algorithms are inherently local, they may incorrectly oriented towards the x-axis during navigation. 左 Convergence, thereby incorrectly forcing the left airway to align with the interventional device by applying appropriate (excessive) dynamic deformation.

[0273] After the interventional device has traveled a short distance within the airway, select x 左 It's definitely not as good as choosing x. 右 However, for locally convergent algorithms, this requires traversing back through the local minimum x. 左 Escape and converge to the global minimum x. 右 It may be too late.

[0274] In some embodiments of this disclosure, global search is incorporated into the dynamic deformation tracking algorithm, which allows for escaping local minima.

[0275] An example of a global search implementation with small perturbations The form in the current state vector x t Nearby application search (e.g., random or grid search), where v i Modify x in some directions t The goal is to find a state vector with a smaller total error. Each perturbation v i The error function can be estimated in parallel within the search algorithm, which can easily utilize multiple computing processing units, such as graphics processing units (GPUs).

[0276] Another option is to use a multiple hypothesis approach. At a specific bifurcation, the skeleton can be divided into two sub-skeletons: one containing left turns and all its offspring (but excluding right turns), and the other containing right turns and all its offspring (but excluding left turns). The tracker runs two separate optimizations, one using... One optimizer uses the "left" skeleton, while the other uses the "right" skeleton. The optimizer using the "right" skeleton will be forced to converge to a right turn because the sub-skeleton lacks a left turn. Initially, the "left" optimizer may show a smaller error; however, after the interventional device has traveled a short distance, the "right" optimizer will win, and the "left" hypothesis will be discarded.

[0277] The method is formalized as follows: For each iteration time t, the search skeleton path {P} j The skeletal path is located at a distance of {S} from the interventional device sample. k Within a certain distance of}. It can be assumed that the interventional device must fit one of these pathways. Then local convergence is performed to find that makes ||E dyn (x, P) j )||(here) Using P in its Γ distance function j Minimize x j Each x j Both describe the model's state vector, which is based on the interventional device being located at path P. j The assumptions within the function are used to minimize the energy function.

[0278] E j =||E dyn (x j P j || represents the minimum state vector x j The error at that point. Then j0 can be chosen such that... if This means that the optimizer has found a way to improve local convergence x. t The state vector, and the optimizer can choose to set x. t ←x j This allows them to escape local minima. The global multi-hypothesis search can be run on every dynamic deformation tracking step. Alternatively, each hypothesis can be tested on a separate step to reduce the required computational power. The intuition behind this idea is as follows: This means that, although by E dyn (x t The convergence surface defined by E can contain many local minima (each E) dyn (x t P j (the sum of all local minima of P), but these P j Each of the given convergence surfaces is much simpler in itself, and therefore easier to search.

[0279] Similar to deformable registration, dynamic deformation tracking algorithms can be formalized as an error minimization optimization problem. The total dynamic deformation error function is ||E dyn (xt This results in a complex convergence surface. The solver can employ global minimization, multiple sub-methods (with several starting points), multiple hypothesis methods, or any other suitable local / global method to minimize the error function in real time while avoiding local minimization. t It can be filtered over time to produce a continuously updated dynamic deformable experience, and can be damped to accelerate convergence (“relaxation”) back to the initial deformable registration state.

[0280] Together with deformable registration, by The described total deformation transformation explains the deformation and transformation between preoperative CT and the anatomical structures at the start of surgery (in LOC coordinates), as well as the additional dynamic deformation of the lungs during NB surgery. A key difference between the deformable registration algorithm and the dynamic deformation tracker is that deformable registration sees many interventional device samples and produces a single deformation and transformation state x. reg This could take several seconds to compute; while the dynamic deformation tracker is real-time and only based on a few observed values ​​at each timestamp. k To provide rapidly evolving x t .

[0281] Breathing deformation

[0282] In some embodiments of this disclosure, in Figure 5 and Figure 7A The rigid registration defined at frame 504 is a cyclically varying registration. More specifically, in some implementations, the registration changes according to the movement of the respiratory cycle.

[0283] One challenge in NB surgery is overcoming the deformation and inaccuracies caused by breathing. This movement can be particularly important for peripheral lung targets that are further from the relatively fixed and / or rigid structures of the trachea and large bronchi.

[0284] The solution is to use a reference sensor attached to the patient's chest to create a generalized LOC coordinate system that moves with the body and chest. Movement of the lungs and interventional devices caused by breathing is partially compensated for by changes in the reference sensor's position, as these two movements are often correlated.

[0285] However, lung deformation caused by respiration is complex. For example, one airway can deform in one direction, while another airway deforms in the opposite direction. This reduces the accuracy of respiration compensation algorithms based on simple reference sensors.

[0286] In partial compensation for this complication, reference sensor readings can be interpreted based on the respiratory phase. For example, the respiratory phase φ∈[0,1] can be calculated based on the periodic motion of the reference sensor. The phase can be represented, for example, as a number between 0 and 1, where φ=0 represents a complete exhalation state and φ=1 represents a complete inhalation state. In this representation, the respiratory phase oscillates between these two extreme values. Alternatively, φ can be understood as a circumference. The true phase on the circumference, where φ = 0 ≈ 1 represents a complete exhalation state, and φ = 0.5 represents a complete inhalation state. Treating φ as the phase on the circumference can distinguish between the transition from exhalation to inhalation and the transition from inhalation to exhalation. These two movements do not necessarily follow the same motion pattern as the function output by the reference sensor.

[0287] After calculating the respiratory phase, a function relating respiratory motion and respiratory phase can be applied. In summary, this function takes the respiratory phase as a parameter for the predicted output, which estimates the respiratory motion at the location of an interventional device. This function can, for example, use a respiratory model adapted to the patient's motion; it can be learned through offline imaging and / or measurements (before interventional device-guided surgery) and / or during NB surgery.

[0288] By applying this function in real time, respiratory motion of interventional devices can be compensated, for example, by keeping them stationary relative to a static airway map, even though both the interventional device and the mapped airway actually deform with respiration. However, if lung deformation causes it to no longer match the periodic predictions of the respiratory model, motion compensation will be downgraded.

[0289] In some embodiments of this disclosure, as the interventional device navigates through the lungs, the anatomical structures and the interventional device are dynamically displayed; as well as their measured and / or estimated true shape / positional state, including deformations due to breathing or other causes. Thus, the respiratory motion of the interventional device is preserved, rather than eliminated through correction.

[0290] In some implementations, to model respiration, the respiratory phase φ is calculated, for example, based on one or more reference sensors attached to the patient's chest. Standard signal processing methods can be used to calculate the respiratory phase, for example, based on the up-and-down movement of the tracked reference sensors associated with the patient's periodic respiratory movements. At the highest sensor position, the patient is estimated to be in full inspiration; at the lowest sensor position, the patient is estimated to be in full expiration.

[0291] A high-pass filter can be used to filter out other body movements of the patient. Additionally or alternatively, multiple reference sensors can be used, with the position of each sensor calculated in the sensor's PCA coordinates. In some implementations, instead of using the absolute or relative height of the reference sensors, a triangular region, for example, between three reference sensors can be examined. For example, it can be assumed that the patient is in full inspiration when the area of ​​this triangle is at its maximum, and in full expiration when the area of ​​this triangle is at its minimum. The respiratory phase φ can be calculated using a method similar to one of the techniques described above or by any other suitable method.

[0292] In some embodiments of this disclosure, φ is used in the initial deformable registration stage (e.g., Figure 5 (Operation of box 504). A single deformable registration state x reflecting the lung deformation state at the start of surgery. reg Replace it with a time-varying registration that considers lung deformation based on the respiratory phase φ. Therefore, the deformable registration algorithm is modified to find the lung. The initial deformable registration is φ-dependent. Based on this result, the system can then predict lung deformation caused by respiration in real time. The total correction of the system will be... That is, the dependence of initial registration on the respiratory phase φ is introduced.

[0293] For example, Figure 7A The dynamic deformation tracker for box 720 uses data from box 506. replace As its input. The dynamic deformation tracker can be like a computation... Same calculation and phase compensation registration The differences may exist, but the error may be small. For example, the filtering described with respect to box 738 does not need to be adjusted to allow movement through the breathing cycle, thus potentially improving the algorithm's artifact suppression characteristics.

[0294] Even without any interventional devices in the airway, one or more reference sensors attached to the patient can be used to optionally calculate the respiratory phase, allowing the airway map to continue “breathing” in sync with the patient’s actual breathing.

[0295] In some implementations using respiratory phase φ∈[0,1]: It is a state of extreme values and The measurements (for the deformable registration states used for exhalation and inhalation, respectively) are calculated. Other states are interpolated between these measured states. This uses the respiratory phase φ to interpolate between the "exhalation" and "inhalation" states. Optionally, extreme states are calculated based on individual registration measurements, separating measurements taken at any extreme point into different datasets.

[0296] In some implementations, the definitions of the deformable registration state vector and energy function are modified as follows:

[0297]

[0298]

[0299]

[0300] The state vector now includes two models: exhalation and inhalation. Optionally, the shape energy function can be modified so that similar shape constraints are imposed on both models.

[0301] For further customization, different shape constraints can be assigned to each model, or at least weighted differently. For example, weighting can be based on the knowledge that the preoperative CT scan indicates full inspiration, thus expecting more pronounced deformations in the "expiratory" model compared to the "inspiratory" model. This can be achieved, for example, by introducing a phase change term E. smp To handle weighted correction.

[0302] Instead of matching each interventional device sample with a single deformable model x reg Instead of matching, each sample is paired with a respiratory phase φ k The respiratory skeleton is matched, where φ k It belongs to the category of interventional device samples S k The breathing phase. More formally:

[0303]

[0304] This will be two models Samples were drawn toward the interventional device, with samples collected at φ≈0 primarily affecting the "exhalation" model, samples collected at φ≈1 primarily affecting the "inhalation" model, and samples with φ in [0,1] affecting both models.

[0305] To fit these two models to the collected sample of interventional devices {S} k The number of available samples K should be from {φ}. k Sampling is performed differently within the range of [blank]. In some implementations, the control system is configured to ensure that sufficiently diverse samples are collected from both lungs and all respiratory phases to prevent overfitting. Because a large number of interventional device samples are collected for each time stamp, it may become much easier to use interventional devices with complete tracking.

[0306] It should be noted that the two-key-phase breathing model can optionally be extended to any number of key breathing phases. For example, it can be defined as: In this scenario, three registration states are used to describe the three corresponding critical respiratory phases: expiration, expiration-inspiration, and inspiration. Interpolation is then performed between the three critical phases. The energy function is defined as described above. For a circle... For a complete respiratory cycle, at least three key respiratory phases are preferably used to simulate the complete respiratory phase. For simple phases (φ∈[0,1]), two key phases (φ0=0, φ1=1) may be sufficient. Adding more key respiratory phases can increase the accuracy of the respiratory model, but it may also cause overfitting in the respiratory model if there are not enough interventional device samples available.

[0307] although Most respiratory movements were modeled, but due to the inaccuracies of the proposed expiratory-inspiratory interpolation model, there may be residual unmodeled respiratory deformation errors. A dynamic deformation tracker can utilize its... To eliminate at least some of these residual errors. The accuracy of the deformable respiratory registration model is important for providing a good starting point for the dynamic deformation tracker to search for local deformation and / or for lung respiratory modeling (by relying solely on the respiratory phase regardless of whether an invasive instrument is inserted into the lung). The dynamic deformation tracker can then be used to... Imposing stronger shape constraints, focusing on Look for relatively small and / or slowly developing deformations nearby. This can improve its overall performance.

[0308] Deformation and breathing view

[0309] In some embodiments of this disclosure, the system is configured to display a view of a deformable lung model, optionally together with a view of the location of one or more interventional devices located within the lung modeled by the deformable lung model. Figure 1 Showing. Now refer to. Figure 8A This is a schematic flowchart outlining a method for generating a deformable lung model view according to some embodiments of the present disclosure.

[0310] A 3D mesh describing the smooth boundary surfaces of airway segmentation can be used to represent airway maps used during guided bronchoscopy. The 3D mesh consists of a set of vertices and triangular faces that connect to form surfaces. Alternatively, the airway 3D mesh can be generated directly from the airway skeleton (centerline and radius) during the skinning process to form a “synthetic” 3D mesh representing the airway map. Additionally or alternatively, gradient information obtained directly from CT volumes or from segmented airway volumes is used to capture fine airway textures. This may generate a more “realistic” mesh.

[0311] Gradient information obtained directly from the CT volume or from the segmented airway volume was used to capture the fine texture of the airway. This potentially produces a more “realistic” mesh.

[0312] In some embodiments of this disclosure, the current deformable state of the deformable lung model is preferably applied to a view of any lung-associated 2-D / 3-D features (view objects) displayed by the system (i.e., not just the model's skeleton). This has the potential advantage of realistically displaying anatomical structures in real-time.

[0313] Examples of such view objects include, for example: 3D meshes of airways, target representations such as spheres or real target meshes, waypoints within the lungs, and / or CT slices of lung imaging.

[0314] Now go to Figure 8A Box 810: In some embodiments of this disclosure, the determination of the deformable state location of a view object begins by associating the view object with the airway of the skeleton of a deformable lung model.

[0315] In some implementations, this association involves assigning one or more vertices of the displayed object (defined in a pre-distortion space, e.g., the space of the original CT image used to define the model skeleton) to the corresponding airway and a normalized distance σ ∈ [0, 1] along the airway. For example, in the case of a 3-D airway mesh, each vertex of the mesh originates from a skeletal branch along that branch at a distance σ. The 3-D mesh can be divided into a branch hierarchy identical to the hierarchical structure of the airway skeleton, where each sub-mesh corresponds to a branch of the skeleton, and each vertex on the branch is assigned a value of σ.

[0316] At box 812, in some implementations, a deformable model {T} is used. i Apply the appropriate interpolation transformation to each vertex v to deform the 3D mesh (or other representation of the view object). For example:

[0317] v 变形 =interp(ΔT) p(i)，t ΔT i，t ,σ)(v)

[0318] Where v belongs to the i-th branch at a distance σ along this branch in CT (or other 3-D image) coordinates, p(i) is the i-th parent branch, and ΔT i，t and ΔT i，t These are variations of the forks at the beginning and end of a branch.

[0319] Alternatively, at box 814, the lighting model of the 3D mesh used for rendering the deformation can be improved by applying a similar deformation to the vertex normals (e.g., reducing the appearance of faceting):

[0320] n 变形 =interp(ΔR) p(i)，t ΔR i，t ,σ)(n)

[0321] Where n is the normal vector corresponding to vertex v, ΔR i，t ΔR p(i)，t It is a deformation ΔT i，t ΔT p(i)，t The corresponding rotation matrix, ignoring the offset.

[0322] Optionally, the deformation of boxes 812 and 814 is applied by the central processing unit (CPU), using its appropriate deformation transformation to transform all vertices (and normals). Alternatively, the deformation can be applied via a graphics processing unit (GPU), for example, using a 3D shader. A shader is a program that runs on the GPU for each specific segment of the graphics pipeline. The vertex shader program is responsible for processing each vertex in the graphics pipeline and can be used to change the geometry of the object before it is displayed.

[0323] Decomposing a 3D mesh into a hierarchical structure of branches (sub-meshes) facilitates the application of deformable models using vertex shaders. Before drawing each branch (sub-mesh), the vertex shader is assigned a matrix ΔT. i，t ΔT p(i)，t Then, a draw command containing branch geometry information (vertices, normals, texture coordinates, etc.) is issued. The distance σ along the branch is given as additional per-vertex data, for example, using unused texture coordinate channels so that they are accessible to the vertex shader program. Inside the shader program, a vertex and its distance σ are input. A pre-configured ΔT is used. i，t ΔT p(i)，t Calculate interpolated deformation transformations for specific vertices and apply them to the vertex positions and normals. The deformation model can also be applied using general-purpose GPU computing toolkits such as CUDA or any other suitable method.

[0324] To apply the deformation model to small targets, such as 3D target spheres, guide arrows, and text, it was observed that the deformation model is smooth and exhibits little variation in small neighborhoods. Therefore, it can be assumed that the small targets undergo uniform deformation transformations. To apply the deformation model, a specific branch and a constant distance σ are assigned to the small object from the airway diagram (e.g., the nearest branch), indicating its distance along the branch (or its centroid distance). Then, as described above, its vertices (and normals) will deform.

[0325] Now for reference Figures 8B-8CThe diagram schematically illustrates the rendering of the 3D mesh before and after the deformation of a single model branch 802 and its offspring, according to some embodiments of this disclosure. An indication of target 801 is also shown.

[0326] Applied to Figure 8B The model below is converted to Figure 8C The deformation of the model below and such Figure 4 The single-branch deformation of the fork 802 shown is the same. However, as mentioned above, the deformation is now applied to the vertices of the 3-D mesh (and normals) to display a smoothly deformed 3-D mesh. The parameter adjustment is an adjustment to the fork itself. Figure 8C The increased curvature is due to the interpolation of the airway between the altered structure at bifurcation 802 and the bifurcation closer to it. The interpolation used can be a uniform curve, but not necessarily; for example, in some embodiments, stiffness is limited to a higher level near the proximal bifurcation where the airway is larger and / or the airway walls are thicker. Additionally or alternatively, when available, positional measurements from the interventional device across the airway region connecting the two bifurcations can be used as a constraint. The curvature may optionally be limited by the curvature present in the baseline state of the lung.

[0327] Target 801 illustrates a small target (e.g., approximately 1 cm in diameter) deformed by being assigned to a branch with a corresponding distance σ. Sometimes, it may seem possible to assign multiple airways to a single small object; for example, when a small 3-D target is located at a bifurcation or even above a skeleton. In such cases, the choice of which branch to bind the small target to the skeleton may be irrelevant; since the deformation model is smooth and σ is used, the deformation interpolation formula will produce similar results for different (closer) airway assignments. However, when the target is far from the airway map, for example, if the target is very peripheral compared to the segmented airways, or if a full CT slice needs to be deformed, more refined deformation extrapolation techniques, as described below, are required. In another case, the target object may be primarily associated with non-lung structures (e.g., the pleural cavity), where the lungs are used as a pathway. Especially in this case, good absolute accuracy in representing the lung location is crucial, as it cannot be assumed that the target will move along with the airways nearby.

[0328] In other cases, the location of target deformation can be predicted based on deformable tracking airways. This can be achieved, optionally, by using a trained model (e.g., a final element model) that predicts how deformation propagates through tissue (as shown in CT or MRI scans). In this case, the target will not be associated with the airway or any other specific anatomical structure. Instead, its location will be deformed by the propagation of airway tracking deformation into the tissue surrounding the target.

[0329] The deformation model describes how to use a set of deformation transformations {T} i This is used to deform the bifurcation. These transformations can be interpolated along the branches to describe how each point on the curved centerline of the airway deforms according to the model. It can be assumed that points not exactly on the centerline but close enough (e.g., within two radii of the airway) deform in the same way as their nearest centerline point. However, it is relatively difficult to determine how points far from any airway deform based on the deformation of the airway. For example, features implemented in some embodiments of this disclosure include, in a dedicated or combined view (with airway) Figure 1 The NB system displays CT slices showing the current intrapulmonary position of the interventional instruments. In some implementations of the deformation tracking algorithm, the NB system displays anatomical features at their true spatial positions, and this may optionally include displaying CT slices with appropriate deformation. Without distorting the CT slices, the voxel data in the slices will reflect the preoperative anatomical state without applying any deformation (and misalignment with other displayed objects).

[0330] Extending a deformation model from the airway to its external regions is an extrapolation problem. The deformation model is known at certain locations in three-dimensional space, and needs to be extended outwards to other points. Extrapolation can be achieved, for example, using thin-plate splines (TPS) or, more generally, another radial basis function (RBF) or a universal basis method. Additionally or alternatively, machine learning methods can be applied, based on training on a sample training set, to predict the most probable deformation propagation within lung tissue; and / or finite element simulation models can be used to predict the propagation of deformation through tissues, such as those represented by CT or MRI scans.

[0331] In some implementations, for example, a set of basis functions (e.g., TPS) is trained on known control points of the deformable model, such that it sends bifurcation or even full-length centerline points to their known destinations (according to the deformable model). Extrapolation models use basis functions that can sample anywhere. Because they have high accuracy in approximating known control points, it is assumed that they will also send external points (not belonging to the airway) to reasonable destination locations. These models can be combined with smoothness measurements suitable for the lungs.

[0332] Another example of extrapolation methods uses the K-nearest neighbors (KNN) method: a point p not located within the airway is deformed. Then, the nearest neighbors of p are searched in the undeformed skeleton, and these points are represented by v. k They are represented, and their distances are used to represent weights. For example: w k =dist(p, v k ) -PWhere P > 0, the nearest neighbor of p is assigned a stronger weight. By increasing P, the nearest point becomes more advantageous relative to other points. Using the K nearest neighbors and their weights, the deformation offset of p can be calculated as follows:

[0333]

[0334] Alternatively, the full deformable transformation at p can be computed using a variant of the "interp" function, which is capable of computed multiple transformations {T}. i The weighted average of}.

[0335] More complex deformation extrapolation can be achieved instead of using simple extrapolation methods based solely on smooth extension control points. For example, ribs can be assumed to be static as long as they do not deform significantly during NB surgery. This information can be added to the extrapolation function based on additional control points: points can be generated on the surface of the segmented pleura and those points can be forced to remain static during deformation. This provides boundary conditions for the extrapolation function, forcing it to decay closer to the pleura. Alternatively, the lung can be modeled not only using its airways but also using blood vessels, fissures, pleura, and other anatomical features. These features can then be fed into a skeletal model (similar to the deformation model described above) and the total lung deformation can be predicted based on some known control points using a method similar to that described above. In some implementations, the physical properties of the lung are simulated; for example, by applying finite element simulation, better predictions of lung deformation between control points are provided. In some implementations, trained machine learning models are used to predict the propagation (or extrapolation) of deformation from known control points to other lung tissues; for example, understanding derived from observed lung tissue behavior is used instead of traditional finite element simulation.

[0336] Now for reference Figures 9A-9B It schematically illustrates some embodiments according to this disclosure, where a single branch and all its offspring are before deformation ( Figure 9A ) and afterwards ( Figure 9B The 16×16 slice interpolation mesh of skeleton 901. The bifurcation 900 of skeleton 901 is at... Figure 9B The deformation also led to the movement of lung region 902.

[0337] To deform a complete CT slice, each voxel in the CT slice can optionally be deformed using one of the extrapolation methods described above. However, this can be computationally expensive because CT slices typically contain a large number of voxels (e.g., 512×512), and the extrapolation formulas are computationally intensive. Deformation of a CT slice can be simplified by constructing a low-resolution uniform grid consisting of, for example, 16×16 blocks covering the entire CT slice. The deformation is calculated at each corner point, and then the deformation between grid points is calculated using simple interpolation between adjacent corner points, such as 2-D bilinear interpolation, bicubic interpolation, spline interpolation, or any other suitable method. In terms of computational resources, using interpolation methods instead of direct per-pixel deformation calculations is preferable, and is almost as accurate as direct deformation methods due to the reasonable smoothness of the deformation model. When finer results are required, a denser grid (e.g., 32×32 blocks) can be used.

[0338] Now for reference Figure 9C The illustration schematically depicts a CT section 910 extending along a surface defined as a curved geometric cross-section, according to some embodiments of the present disclosure, such that it includes a region of the original 3-D image of the anatomical structure, corresponding to a location along a spiral path 914 traversing the anatomical structure. In some embodiments, path 914 is a path through an airway. In some embodiments, path 914 is defined by the shape and position of the interventional device. In some embodiments, path 914 is defined using anatomical features such as therapeutic targets, blood vessels, or other structures of interest as waypoints. Path 914 may be defined based on conditions during the procedure (e.g., the current position of the interventional device) and / or include predefined portions, such as a planned path to the target of the interventional device.

[0339] The CT portion 910 corresponds to data from a source CT image located at a position where it intersects with a surface extending through the volume of the 3-D image, the surface itself including positions along path 914.

[0340] In some embodiments, path 914 is presented as corresponding to a 3-D shape that corresponds to the measurement shape of its corresponding interventional device, and CT portion 910 is deformed from its original coordinates to maintain region correspondence. In some embodiments, interventional device path 914 and CT portion 910 are displayed together in a composite 3-D scene that also includes a rendered mesh view of deformable model 912.

[0341] Using a single-sensor interventional device, deformed CT slices 910 can be displayed, for example using the low-resolution deformable mesh described above, reflecting the total deformation of the lung and centered on the tip of the interventional device. The CT slice 910 can be global (reaching the CT boundary) or local (around a specific path along the airways passing through the lung, e.g., as per...) Figures 10A-10B (As described). When using a fully tracked interventional instrument, the displayed CT slice may be geometrically curved, intersecting all or most of the curves (not just the tip) of the interventional instrument, making it non-planar. Such curved CT slices will display deformed anatomical features along the entire length of the interventional instrument, providing valuable information for NB procedures (e.g., location of vessels, fissures, septa, and / or pleura). Curved CT slices can also pass through specific important anatomical features. For example, a curved CT slice may pass through the next navigation bifurcation in an airway map, such that the cross-section of the bifurcation is displayed in a slice in front of the interventional instrument. A potential advantage of using curved CT slices is that it compresses the (deformed) 3-D CT volume into a 2-D view, encoding most of the valuable anatomical features for navigation within the area of ​​the interventional instrument.

[0342] Since not all important anatomical features always lie on smooth 2D surfaces, the specific shape of the curved CT slice can be solved during the optimization process. This involves searching for 2D surfaces that approximate a set of important features (e.g., interventional instrument path 914 and / or target 916) in location and orientation, with weights describing the importance of each feature. The optimization process can include weights to penalize excessive distortion; for example, optimization can penalize deep and / or sharp angles of curvature, and / or penalize extreme curvatures based on the amount of increased surface area they contribute. The end result is a curved CT slice that compresses a deformed 3D volume into a compact 2D CT view.

[0343] Despite Figure 9C The text describes CT images as an example, but it should be understood that the display method may optionally be performed using another type of 3-D image, such as MRI images, ultrasound images, and / or positron emission tomography (PET) scan images. Optionally, slices from two or more 3-D images (optionally having different imaging modalities) may be combined.

[0344] Now for reference Figures 10A-10B It schematically illustrates some embodiments according to this disclosure, including before modification ( Figure 10A ) and afterwards ( Figure 10B The CT strips 1002 and 1003 along the path in the composite-D scene of the rendered mesh view of the deformed model.

[0345] Figures 9A-9BThe described low-resolution 2D deformable mesh allows for the rapid calculation of deformation at any pixel between mesh points via interpolation, enabling real-time display of deformable CT slices. The same technique can be generalized to create 3D deformable meshes covering the entire volume of a CT (or other 3D image). The deformation is calculated for each control point of the mesh, and then the deformation can be interpolated onto voxels between the mesh points. 3D deformable meshes enable rapid calculation of deformation anywhere within the imaging volume. This allows for real-time display of deformable 3D views of the lungs, such as deformable virtual bronchoscopy or deformable virtual fluoroscopy, both of which use ray-projection rendering techniques. In a standard virtual bronchoscopy view, a virtual camera is positioned inside the airway. Each pixel on the screen represents a single virtual ray that travels through the imaging voxels to accumulate the final shadow of that pixel. The end result is a realistic image that looks similar to an image from a real bronchoscope inside the airway. While standard virtual bronchoscopy algorithms sample the static CT volume in their ray-tracing step, 3D deformable meshes can be used in dynamic virtual bronchoscopy algorithms to create ray-traced images of deformable CT volumes. Instead of sampling voxels from a static 3-D image, at each ray-tracing step, a 3-D deformable mesh is used to transform the position of each ray into its deformed position within the initial, undeformed 3-D image, thereby effectively sampling real-time deformed CT.

[0346] During NB surgery, the physician guides the instrument (e.g., an interventional device) along a path from the trachea 1000 to the target 1005. This path can be displayed as a 3-D centerline within an airway map. As described above, raw CT data around the interventional device can be presented to the physician using a local CT slice centered on the interventional device or its tip, or the system can present a complete CT strip along the path in 3-D (optionally, a strip extending approximately equidistant on either side of the path), for example, as... Figures 10A-10B As shown. When the lung is deformed (e.g., as indicated by measuring the position of interventional devices and / or another parameter), CT data, along with other anatomical deformity features, is displayed at the location of the deformity. To deform a CT slice, its spatial position can be deformed to the deformed location (while retaining its original CT data), or its spatial position can be fixed while deforming its CT data. Because 3D CT strips naturally integrate with the airway, they deform in space along with the airway, as... Figures 10A-10B As shown. Therefore, the 3-D CT strips retain their original CT data, but their spatial position is smoothly deformed along with the airway, similar to how a 3-D airway map is deformed.

[0347] Now for reference Figure 11It shows a flat strip 1100 of CT image data extracted along a 3-D path following the interventional device 1101, according to some embodiments of the present disclosure.

[0348] Figures 10A-10B The same 3-D CT strip shown can optionally be used to generate a 2-D path view, where the CT strip 1100 is displayed as a static 2-D background image, and the interventional instrument 1101 and / or target 1102 are projected on top of the static image in path coordinates. Similar to the 3-D CT strip, this view potentially encodes the most important anatomical information for navigating along the path to a specific target, but it is projected in a 2-D view that displays the entire path from start to finish as a flat surface, independent of the 3-D camera position and unaffected by 3-D occlusion that may occur in the 3-D CT strip view, such as... Figures 10A-10B As shown. A potential advantage of 2-D path views is their locality, making them unaffected by deformation. When deformation is applied, the CT strip follows the deformed airway (as described above) and therefore does not change, and interventional devices typically maintain their position within the path (e.g., in path coordinates).

[0349] Applying the deformation model to all displayed 2D / 3D objects, including 3D airway meshes and CT slices, allows for the real-time generation of complex deformation scenes. For example, during a patient's breathing process, a 3D breathing airway map, overlaid with a breathing CT slice, can be displayed, perfectly synchronized with the patient's breathing. As another example: during manipulation, an interventional device can apply force to the airway it navigates within. The system can then display how the 3D airway deforms in real-time in response to the manipulation of the interventional device (based on the deformation model), and display a corresponding deformed version of a local CT slice near the navigation airway. Anatomical data of the real-time deformation is then shown to the physician, containing the original features of the navigation airway and the anatomical deformation from the CT volume, valuable for guidance, biopsies, and treatment.

[0350] Now for reference Figure 12 The illustration schematically depicts a system for tracking the movement of an interventional device within the lung airway, according to some embodiments of the present disclosure.

[0351] Figure 12 The system's hardware components include a memory 1201, a processor 1202, and an optional display 1203. Optionally, an interventional device 1204 equipped with one or more position sensors 1205 is provided with the system, optionally together with a position sensing controller 1206. The position sensors 1205 may be located, for example, at the tip of the interventional device 1204, and / or at a location along the body of the interventional device 1204.

[0352] Optionally, all or part of a subsystem including interventional device 1204 and position sensing controller 1206 may be provided separately.

[0353] exist Figure 12 The selected data structures used and / or generated in the operation of the system are shown as boxes within memory 1201.

[0354] These include:

[0355] • Instruction 1212, whose instruction processor 1202 executes the methods described herein (e.g., regarding...). Figure 1 , 5 The processor 1202 may perform any one or more operations as described in 7A-7B and 8A. The processor 1202 may generate and / or access any other data structures according to instruction 1212.

[0356] • Pre-deformation model 1210, for example, corresponding to Figure 1 The model and / or generated at the middle frame 110 Figure 5 Baseline airway diagram 503.

[0357] • Initial registration 1214, which corresponds to, for example, by Figure 5 Box 504 and / or Figure 7A The registration applied to the transformed state vector 710.

[0358] • Interventional device position measurement 1218, which corresponds to, for example Figure 5 Location data 505.

[0359] • Dynamic registration 1216, which corresponds to, for example Figure 7A The dynamic transformation state vector 712.

[0360] Instruction 1212 may instruct processor 1202 to generate one or more views 1220 representing a modified (e.g., initial and / or dynamically registered) version of the pre-deformed model 1210 and provide them to display 1203. Box 1220 illustrates the views as data streams fed to display 1203. Data structures representing these views and / or intermediate data structures used in generating these views may also be generated by processor 1202 and stored in memory 1201 (not explicitly shown). Views 1220 can be generated, for example, as per [reference to...]. Figure 8A-11 As described.

[0361] general

[0362] As used in this article, the term “about” refers to a quantity or value within ±10%.

[0363] The terms “comprises,” “comprising,” “includes,” “including,” “having,” and their cognates (conjugates) mean “including but not limited to.”

[0364] The term "consisting of" means "including and limited to".

[0365] The term "consisting essentially of" means that a composition, method, or structure may include additional ingredients, steps, and / or portions, provided that the additional ingredients, steps, and / or portions do not substantially alter the fundamental and novel characteristics of the claimed composition, method, or structure.

[0366] As used herein, the singular forms “a” and “the” include the plural unless the context clearly indicates otherwise. For example, the terms “a compound” or “at least one compound” can include multiple compounds, including mixtures thereof.

[0367] The terms “example” and “exemplary” are used herein to mean “served as an example, instance, or illustration.” Any implementation described as “example” and “exemplary” is not necessarily to be construed as being more preferred or advantageous than other implementations, and / or excluding combinations of features of other implementations.

[0368] The term "optionally" is used herein to mean "provided in some embodiments but not in others." Any particular embodiment of this disclosure may include several "optional" features unless these features conflict with each other.

[0369] As used herein, the term "method" refers to the manner, means, techniques, and procedures used to accomplish a given task, including but not limited to those manner, means, techniques, and procedures known to or readily developed from known manner, means, techniques, and procedures by practitioners in the fields of chemistry, pharmacology, biology, biochemistry, and medicine.

[0370] As used in this article, the term “treating” includes abandoning, substantially suppressing, slowing or reversing the development of a condition, substantially improving the clinical or aesthetic symptoms of a condition, or substantially preventing the occurrence of the clinical or aesthetic symptoms of a condition.

[0371] Throughout this application, various embodiments of this disclosure may be presented in the form of ranges. It should be understood that the range descriptions are merely for convenience and brevity and should not be construed as immutable limitations on the scope of this disclosure. Therefore, the range descriptions should be considered as having specifically disclosed all possible subranges and individual numerical values ​​within those ranges. For example, a description of a range such as "1 to 6" should be considered as having specifically disclosed subranges such as "1 to 3," "1 to 4," "1 to 5," "2 to 4," "2 to 6," "3 to 6," etc., and individual numbers within those ranges, such as 1, 2, 3, 4, 5, and 6. This applies regardless of the width of the range.

[0372] Whenever this document refers to a range of numbers (e.g., “10 to 15 (10-15 or 10 to 15)” or any pair of numbers connected by such another range indication), it means that any referenced number (fraction or integer) within the indicated range, including the range boundaries, is included, unless the context explicitly specifies otherwise. The expressions “range between the first and second indicator numbers” and “range from the first indicator number to the second indicator number (or another such range indication term)” are used interchangeably herein and mean that the first and second indicator numbers, and all numbers and integers in between, are included.

[0373] While this disclosure has been described in conjunction with its specific embodiments, it will be apparent to those skilled in the art that many substitutions, modifications, and variations will be readily apparent. Therefore, this disclosure is intended to cover all such substitutions, modifications, and variations that fall within the spirit and scope of the appended claims.

[0374] It should be understood that, for clarity, certain features of this disclosure described in the context of individual embodiments may also be provided in combination in a single embodiment. Conversely, for brevity, various features of this disclosure described in the context of a single embodiment may also be provided individually, or in any suitable sub-combination, or in a suitable manner as in any other described embodiment of this disclosure. Certain features described in the context of various embodiments should not be considered essential features of those embodiments unless the embodiment does not function without these elements.

[0375] The applicant intends that all publications, patents, and patent applications mentioned in this specification are incorporated herein by reference in their entirety, as if each individual publication, patent, or patent application were specifically and individually incorporated herein by reference. Furthermore, any reference or designation in this application should not be construed as an admission that such reference is prior art to this disclosure. The section headings used should not be construed as necessary limitations. In addition, the entire contents of any one or more priority documents of this disclosure are incorporated herein by reference.

Claims

1. A system for modeling the deformed state of a branch anatomical structure via interventional instrument navigation, the system comprising a memory storing instructions and a processor, wherein the memory instructs the processor to: Access a baseline model constructed using the imaging shapes of the branches and bifurcations of the said branching anatomy; While inserting the interventional device into the branch anatomical structure, multiple measurements of the shape and position of the interventional device are accessed using the interventional device, including at least two measurements of shape and position respectively when the interventional device is located in at least two corresponding branches proximally connected to the same bifurcation; The modifications to the baseline model are calculated based on the multiple measurements, including modifications to the modeling branches representing the at least two corresponding branches; as well as A deformation model is generated by modifying the baseline model according to the deformation, thereby registering the interventional device to the deformed state of the branch anatomy. And among them: The processor uses an error function to calculate the deformation, the error function defining the error relative to the predetermined target and shape constraints; The predetermined target is characterized by the deviation of the deformable model from the shape of the branch anatomy, defined by multiple measurements of error increase as a function of the deviation from the shape of the branch anatomy; and The shape constraint is characterized by changes in the anatomical constraint, which manifest as an increase in error as a function of the deviation from the baseline model after the deformation model is weighted.

2. The system according to claim 1, wherein: The processor, based on further measurements of the intervening shape and position, uses a second error function to calculate and generate a dynamic deformation model that modifies the deformation model. This second error function is a shape constraint error function that defines shape constraints in terms of error increment, which is a function of the deviation from the deformation model. The processor reduces the second error function in the following way: From each of the multiple initial states, convergence is achieved towards the respective convergence objective, minimizing the error function value; and Based on the value of the minimized error function, a selection is made from the convergence objective to define the modification of the deformed model.

3. The system according to claim 1, wherein, Each of the multiple measurements of shape and position is associated with a corresponding respiratory phase.

4. The system of claim 3, further comprising using the measurements of shape and position associated with the modeled respiratory phase to determine deformation parameters of the modeling branch in each of a plurality of modeled respiratory phases.

5. The system according to claim 1, wherein, Calculating the deformed model includes selecting measurements from the multiple measurements taken during a selected respiratory phase, and using the selected measurements to calculate a respiratory phase-dependent modification of the baseline model; and the generation includes modifying the baseline model to represent the respiratory phase-dependent shape of the deformed model.

6. The system according to claim 1, wherein, Generating the deformable model involves calculating the modeling shape of the branch anatomy for at least one respiratory phase by combining the modeling shapes of at least two other respiratory phases.

7. The system according to claim 6, wherein, The modeling shape of the branch anatomy for calculating at least one respiratory phase includes interpolation between the at least two other respiratory phases.

8. The system according to claim 1, wherein: The deformation model includes a skeletonized model, which represents the branches and bifurcations of the branch anatomy as segments connected at the modeling bifurcations; The deformation model models the error based on the movement and bending of the segment relative to the deformation model; and The processor uses the modeled error to limit at least one of the following: The relative angle changes of segments that share a common bifurcation. Based on the variation of the anchoring force applied from the distal side of at least some of the segments, Based on the variation of curvature along the length of at least some of the said segments, Based on the variation of segments of intervals that are not directly connected through the branching anatomical structures. Segments that form a total shape that are inconsistent with the range of total shapes represented by a dataset comprising multiple total shapes, and Form segments of branch shapes that do not conform to the range of branch shapes represented by a dataset containing multiple branch shapes.

9. The system according to any one of claims 1 to 8, wherein, The shape of each of the multiple measurements during the access is measured as multiple position measurements along the location of the interventional device, each of which is encoded in a probe positioning coordinate system comprising at least three position axes measured by sensors at the corresponding locations.

10. The system according to claim 9, wherein: The deformation model calibrates the probe positioning coordinate system to the imaging shape of the branch and the fork by modifying the baseline model.

11. The system according to any one of claims 1 to 8, wherein, Each of the baseline model and the deformed model includes a skeletal model, wherein the branches and bifurcations of the branch anatomy correspond to the modeling branches in each model, represented as segments connected at model bifurcations.

12. The system according to any one of claims 1 to 8, wherein, Each measurement of the shape and position of the interventional device describes the shape and position of the curve of the interventional device.

13. The system according to any one of claims 1 to 8, comprising the interventional device configured to be inserted into the branching anatomy.

Images