Systems and methods for endoscope localization

Abstract

Systems, methods, computer-readable media, and techniques for locating a scope tip of an endoscope within a patient's organ include: (a) acquiring (i) a biological model of the patient's organ and (ii) electromagnetic (EM) data generated by the endoscope within the organ; (b) generating a plurality of scope tip location hypotheses based on the biological model and the EM data; (c) generating a plurality of transformations, each transformation of the plurality of transformations respectively mapping each of the plurality of scope tip location hypotheses to the EM data; (d) determining a scope tip location from the plurality of scope tip location hypotheses, the location corresponding to an assumed transformation of the plurality of transformations that satisfies a threshold; and (e) presenting the scope tip location on a graphical display.

Description

[Technical Field]

[0001] cross reference This application claims priority to U.S. Provisional Patent Application No. 63 / 477,890, filed December 30, 2022, which is incorporated herein by reference in its entirety. [Background technology]

[0002] Early diagnosis of lung cancer is crucial. Lung cancer is the deadliest form of cancer, killing more than 150,000 people annually. Compared with CT-guided TTNA (CT-TTNA), navigation bronchoscopy has a favorable safety profile (reduced risk of pneumothorax, life-threatening bleeding, and hospital stay) and can determine mediastinal stage, but is associated with a lower diagnostic yield. Endoscopy (e.g., bronchoscopy) may involve accessing and visualizing the inside of a patient's internal cavity (e.g., airway) for diagnostic or therapeutic purposes. During the procedure, a flexible tubular tool, such as an endoscope, may be inserted into the patient's body, and instruments can be delivered through the endoscope to identified tissue sites for diagnosis or treatment.

[0003] Robotic bronchoscopy systems have attracted interest for use in biopsy of peripheral lung lesions. Robotic platforms offer superior stability, distal articulation, and visualization compared to conventional pre-curved catheters. Some conventional robotic bronchoscopy systems utilize shape-sensing technology (SS) for guidance. SS catheters may have embedded fiber-optic sensors that measure the catheter's shape hundreds of times per minute. Other conventional robotic bronchoscopy systems incorporate direct visualization, optical pattern recognition, and geographic location sensing (OPRGPS) for guidance. Both SS and OPRGPS systems utilize pre-planned CT scans to create electronically generated virtual targets. However, SS and OPRGPS systems are prone to CT-to-body discrepancy (CT2BD). CT2BD is the mismatch between the electronic virtual target and the actual anatomical location of a peripheral lung lesion. CT2BD can occur for a variety of reasons, including atelectasis, anesthesia-induced neuromuscular weakness, tissue distortion by the catheter system, bleeding, ferromagnetic interference, and anatomical disturbances such as pleural effusion. Neither the SS system nor the OPRGPS platform includes intraoperative real-time correction of CT2BD. The combination of CT2BD with other deformations, such as respiratory motion of the lungs, affects the localization accuracy of the electromagnetic sensor at the endoscope tip.

[0004] Existing solutions for localization, such as nearest-point searches and Bayesian filters, are based on rigid lung models that do not account for deformation, which can increase the length of the procedure, frustrate the operator, and ultimately lead to a non-diagnostic procedure. Furthermore, localization measurements obtained by existing methods can be inaccurate due to noise, such as Gaussian noise, and can be biased by tissue deformation. Summary of the Invention

[0005] Provided herein are systems, methods, computer-readable media, and techniques for locating the scope tip of an endoscope within a patient's organ, taking into account deformation, CT2BD, and respiratory motion. Better scope tip localization results can be achieved when deformation is properly accounted for. In some cases, the systems, methods, computer-readable media, and techniques generate an estimate of one or both of: (i) the current position and orientation of the scope tip within the patient frame; or (ii) the path of the scope tip through the patient within the patient frame. This estimate may be used to guide or drive the scope (e.g., via a user) toward a pre-operatively planned target location to point the scope at the target.

[0006] In some cases, localization may be performed by acquiring EM data from an electromagnetic (EM) sensor system, registering the EM data to CT scan (patient data) coordinates, and using the EM sensor data to calculate the position of the scope tip within the patient's lung airway. Based on this localization, information such as the distance to a lesion may be determined. However, conventional methods map point locations based on EM data to points (e.g., closest points) in a CT scan-based airway model, which can result in inaccurate location results due to noise (e.g., Gaussian noise) in the EM sensor data and biases due to tissue deformation. The localization method herein beneficially improves localization accuracy by treating the airway path as a deformable object (instead of a rigid model as in conventional methods) and identifying a best-fit deformable path among multiple deformable hypothetical paths that best matches the path based on EM data (instead of finding a best-fit hypothetical point using conditional probability estimation). The algorithm herein advantageously avoids Gaussian noise in the EM sensor data by finding a best-fit path (instead of a point), and eliminates errors introduced by tissue distortion by treating the airway as a deformable object. In some embodiments, the methods herein may identify a deformable best-fit path and determine the position of the endoscope tip by solving a nonlinear optimization problem.

[0007] In some cases, the systems, methods, computer-readable media, and techniques may be implemented in conjunction with tomosynthesis. Tomosynthesis (sometimes referred to as "tomo") is narrow-angle tomography, as opposed to full-angle (e.g., up to 180-degree) tomography. However, tomosynthesis reconstructions do not have uniform resolution. For example, resolution is often lowest in the depth direction. The standard method of displaying a 3D volume data set in three orthogonal planes (e.g., axial, sagittal, and coronal) may not be useful because two of the orthogonal planes have low resolution. A common way to view a tomosynthesis volume is to scroll in the depth direction, where each slice has good resolution. In the case of pulmonology, this can be identified by viewing it in the coronal plane and scrolling in the anterior-posterior (AP) direction. However, this makes it difficult to determine the spatial relationship of structures in the depth direction. In particular, it can be difficult to determine whether a tool (e.g., a biopsy needle) is within a lesion in the AP direction of a tomosynthesis reconstruction of the chest.

[0008] There is a need for methods and systems that can localize an endoscope tip (e.g., determine whether a tool is within a target, such as a lesion) with improved precision and accuracy. The present disclosure addresses this need by providing systems, methods, computer-readable media, and techniques for localizing the scope tip of an endoscope within a patient's organ. In some cases, the systems, methods, and computer-readable media may perform operations including: (a) acquiring (i) a biological model of the patient's organ and (ii) electromagnetic (EM) data generated by an endoscope within the organ; (b) generating multiple scope tip location hypotheses based on the biological model and the EM data; (c) generating multiple transformations, each of the multiple transformations mapping each of the multiple scope tip location hypotheses to the EM data; (d) determining a scope tip location from the multiple scope tip location hypotheses, wherein the location corresponds to an assumed transformation of the multiple transformations that meets a threshold; and (e) presenting the scope tip location on a graphical display.

[0009] In one aspect, a method for locating an endoscope within a patient's body portion is provided, the method including: (a) acquiring a sequence of electromagnetic (EM) data; (b) generating an EM data-based path based at least in part on the sequence of EM data; (c) identifying one or more path hypotheses based at least in part on data points from the sequence of EM data, the one or more path hypotheses having a shape based on a biological model of the patient's body portion; (d) generating one or more deformed paths by mapping the one or more path hypotheses to the EM data-based path using an optimization algorithm; and (e) selecting a deformed path from the one or more deformed paths based at least in part on a probability associated with each of the one or more path hypotheses, and determining a position of the tip of the endoscope based on the selected deformed path.

[0010] In another but related aspect, a system for locating an endoscope within a patient's body portion includes a memory storing computer-executable instructions and one or more processors in communication with the endoscope, the one or more processors configured to execute the computer-executable instructions to: (a) acquire a sequence of electromagnetic (EM) data; (b) generate an EM data-based path based at least in part on the sequence of EM data; (c) identify one or more path hypotheses based at least in part on data points from the sequence of EM data, where the one or more path hypotheses have a shape based on a biological model of the patient's body portion; (d) generate one or more deformed paths by mapping the one or more path hypotheses to the EM data-based path using an optimization algorithm; and (e) select a deformed path from the one or more deformed paths based at least in part on a probability associated with each of the one or more path hypotheses, and determine a position of the tip of the endoscope based on the selected deformed path.

[0011] In some embodiments, prior to (c), the path based on the EM data is transformed into the coordinate frame of the biological model. In some embodiments, the biological model includes one or more airways. In some cases, the EM data is acquired while navigating the tip of the endoscope forward or backward along the one or more airways.

[0012] In some embodiments, the EM data-based path is generated by applying binning filtering and / or temporal filtering to the EM data. In some cases, the EM data-based path includes a sequence of points that are evenly distributed in both the spatial and temporal domains and represents a driving trajectory of the tip of the endoscope.

[0013] In some embodiments, the body part is a lung and the endoscope is a bronchoscope. In some embodiments, the deformation parameter is indicative of deformation of the body part due to patient movement. In some embodiments, the biological model of the patient's body part is generated based on a computed tomography (CT) scan. In some embodiments, the deformation parameter is indicative of deformation due to CT-body dissociation.

[0014] In some embodiments, the one or more path hypotheses are within a predetermined position range from the most recent data point in the sequence of EM data, are within a measurement error or deformation threshold, or exceed a likelihood threshold, hi some embodiments, each of the one or more path hypotheses includes at least a portion of the centerline of the airway of the biological model.

[0015] In some embodiments, the method further includes dynamically adding or removing path hypotheses while the tip of the endoscope is being driven through the body part. In some embodiments, mapping the one or more path hypotheses to the EM data-based path includes applying an optimization algorithm to determine a deformation and shape match between the one or more path hypotheses and the EM data-based path. In some embodiments, a deformed path selected from the one or more deformed path paths is associated with a highest probability. In some cases, the highest probability is determined based at least in part on normalized probabilities associated with each of the one or more path hypotheses.

[0016]

[0013] Further aspects and advantages of the present disclosure will become readily apparent to those skilled in the art from the following detailed description, wherein only illustrative embodiments of the present disclosure are shown and described. As will be realized, the present disclosure is capable of other and different embodiments, and its several details are capable of modifications in various obvious respects, all without departing from the disclosure. Accordingly, the drawings and description are to be regarded as illustrative in nature, and not as restrictive.

[0017] Incorporation by Reference All publications, patents, and patent applications mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent, or patent application was specifically and individually indicated to be incorporated by reference. To the extent that the publications, patents, or patent applications incorporated by reference conflict with the disclosure contained herein, the present specification is intended to supersede or take precedence over such conflicting material. [Brief explanation of the drawings]

[0018] The novel features of the invention are set forth with particularity in the appended claims. A better understanding of the features and advantages of the present invention will be obtained by reference to the following detailed description that sets forth illustrative embodiments, in which the principles of the invention are utilized, and the accompanying drawings (also referred to herein as "Figure" and "FIG.").

[0019] [Figure 1] 1 illustrates an exemplary localization process using deformable path mapping. [Figure 2] 1 illustrates an exemplary batch localization process using deformable path mapping. [Figure 3] 1 is an example of a robotic bronchoscopy system according to some embodiments of the present invention. [Figure 4] 1 shows an example of a fluoroscopic (tomosynthesis) imaging system. [Figure 5] 1 shows an example of a flexible endoscope. [Figure 6] 1 shows an example of a flexible endoscope. [Figure 7] 1 illustrates an example of an instrument drive mechanism that provides a mechanical interface to the handle portion of a robotic bronchoscope. [Figure 8] 1 shows an example of the distal tip of an endoscope. [Figure 9] 1 illustrates an exemplary distal portion of a catheter with an integrated imaging and illumination device. [Figure 10] 1 shows an example of a biological model of the lung including EM data and multiple localization hypotheses. [Figure 11] 1 shows an example of a biological model of the lung including EM data and multiple localization hypotheses. [Figure 12] 1 illustrates a computer system that is programmed or otherwise configured to perform the methods presented herein. [Figure 13] 1 illustrates an example method for presenting one or both of a tomosynthesis reconstruction or an enhanced fluoroscopy overlay. [Figure 14] 1 illustrates an exemplary diagram of an EM sensor state machine. [Figure 15] 1 shows an exemplary diagram of a registration state machine. [Figure 16] 1 illustrates an exemplary diagram of a location state machine. DETAILED DESCRIPTION OF THE INVENTION

[0020] While the exemplary embodiments are primarily directed to tomosynthesis, endoscopy (e.g., bronchoscopy), and the like, this is not intended to be limiting, and those skilled in the art will understand that the systems, methods, and techniques described herein may be used for other therapeutic or diagnostic procedures and may be used in other anatomical regions of a patient's body, such as the digestive system, including, but not limited to, the esophagus, liver, stomach, colon, urinary tract, and various others.

[0021] The embodiments disclosed herein can be combined in one or more of numerous ways to provide improved diagnosis and treatment for patients. The disclosed embodiments can be combined with existing methods and devices to provide improved treatment, such as in combination with known methods of lung disease diagnosis, surgery, and surgery on other tissues and organs. It should be understood that any one or more of the structures and steps described herein can be combined with any one or more additional structures and steps of the methods and devices described herein, and the figures and supporting text provide a description according to the embodiments.

[0022] Although the treatment plans and definitions of diagnostic or surgical procedures described herein are presented in the context of pulmonary diagnosis or surgery, the methods and devices described herein can be used to treat any tissue of the body and any organ and duct of the body, such as the brain, heart, lungs, intestines, eyes, skin, kidneys, liver, pancreas, stomach, uterus, ovaries, testes, bladder, ear, nose, mouth, soft tissue such as bone marrow, adipose tissue, muscle, glandular and mucosal tissue, bone marrow and nerve tissue, cartilage, hard biological tissue such as teeth and bone, and body cavities and passageways such as the sinuses, ureters, colon, esophagus, pulmonary passageways, blood vessels, and throat.

[0023] As used herein, a processor encompasses one or more processors, such as a single processor or multiple processors, for example, in a distributed processing system. A controller or processor as described herein generally includes a tangible medium that stores instructions for performing process steps, and a processor may comprise, for example, one or more of a central processing unit, programmable array logic, gate array logic, or field programmable gate array. In some cases, the one or more processors may be a programmable processor (e.g., a central processing unit (CPU) or microprocessor), a digital signal processor (DSP), a field programmable gate array (FPGA), or one or more Advanced Risk Machines (ARM) processors. In some cases, the one or more processors may be operably coupled to a non-transitory computer-readable medium. The non-transitory computer-readable medium may store logic, code, or program instructions executable by one or more processor units to perform one or more steps. The non-transitory computer-readable medium may include one or more memory units (e.g., removable media or external storage devices, such as SD cards or random access memory (RAM)). One or more of the methods or operations disclosed herein may be implemented in hardware components or a combination of hardware and software, such as, for example, an ASIC, a special purpose computer, or a general purpose computer.

[0024] As used herein, the terms distal and proximal may generally refer to a location indicated from a device and may be the opposite of an anatomical indication. For example, a distal location of a bronchoscope or catheter may correspond to a proximal location of a patient's elongated portion, and a proximal location of a bronchoscope or catheter may correspond to a distal location of a patient's elongated portion.

[0025] The systems described herein include an elongated portion or member, such as a catheter. The terms “elongated member,” “catheter,” and “bronchoscope” are used interchangeably throughout this specification, unless the context indicates otherwise. The elongated member can be placed directly within a body lumen or cavity. In some embodiments, the system may further include a support device, such as a robotic manipulator (e.g., a robotic arm), to drive, support, position, or control the movement or motion of the elongated member. Alternatively or additionally, the support device may be a handheld device or other control device that may or may not include a robotic system. In some embodiments, the system may further include peripheral devices and subsystems, such as an imaging system, that assist or facilitate navigation of the elongated member to a target site within the subject's body. Such navigation may require a registration process, as described later in this specification.

[0026] In some embodiments of the present disclosure, a robotic bronchoscopy system is provided to perform improved surgical or diagnostic procedures at a low cost. For example, the robotic bronchoscopy system may include a steerable catheter that can be entirely disposable. This may advantageously reduce the need for sterilization or disinfection, which can be costly, difficult to operate, and ineffective. Furthermore, one challenge in bronchoscopy is navigating through the airways to reach the upper lobes of the lungs. In some cases, the provided robotic bronchoscopy system may be designed with the capability to navigate through airways with small curvatures autonomously or semi-autonomously. Autonomous or semi-autonomous navigation may require a registration process. Alternatively, the robotic bronchoscopy system may be navigated by an operator via a control system using visual guidance.

[0027] The typical lung cancer diagnosis and surgical treatment process can vary greatly depending on the techniques, clinical protocols, and clinical settings used by medical institutions. Inconsistent processes can result in delayed early lung cancer diagnosis, high costs to the healthcare system and patients for lung cancer diagnosis and treatment, and a high risk of clinical and procedural complications. The robotic bronchoscopy system herein can utilize integrated tomosynthesis to improve lesion visualization and intralesional tool confirmation, and can utilize enhanced fluoroscopy to enable real-time navigation updates and guidance throughout the lung, thereby enabling standardized early lung cancer diagnosis and treatment.

[0028] In one aspect of the present disclosure, a method for locating a scope tip of an endoscope within a body part of a patient is provided, the method comprising: (a) acquiring (i) a biological model of the body part of the patient, and (ii) electromagnetic (EM) data generated by the endoscope; The method includes (b) generating an EM data-based path based on the sequence of EM data; (c) identifying one or more path hypotheses based on a biological model, each of the one or more path hypotheses including deformation parameters; (d) determining a probability associated with each of the one or more path hypotheses by mapping the one or more path hypotheses to the EM data-based path; and (e) determining a best-fit path from the one or more path hypotheses as a position of the scope tip based at least in part on the probability associated with each of the one or more path hypotheses. The patient's body part may include any tissue or organ of the patient, such as the lungs and / or one or more airways. The EM data-based path indicates a trajectory of the scope tip, and the EM data is acquired while navigating the scope tip along the one or more airways.

[0029] In some cases, the EM data-based path is generated by applying binning filtering and / or temporal filtering to the EM data, or other suitable processing may be employed such that the EM data-based path / trajectory includes a sequence of points that are evenly distributed in both the spatial and temporal domains.

[0030] 1 illustrates an exemplary localization process 100 using deformable path mapping. In some cases, process 100 may include localizing the scope tip of a bronchoscope within a body part (e.g., lungs) of a patient. The Deformable Path Mapping Localization (DPM Localization) algorithm or process 100, which performs localization using a bronchoscope in the lungs, can be applied in any number of ways, such as with other types of endoscopes in other organs or cavities of the body (e.g., humans, animals).

[0031] Process 100 may optionally begin with initialization (or configuration) at 105. During initialization, various configuration parameters may be obtained. For example, configuration parameters for one or more of the following may be obtained: EM sensors, registration, localization, localization executor, or navigation. During initialization 105, various example configuration parameters may be obtained.

[0032] Inputs to the DPM algorithm may include EM sensor data acquired as the scope is traversed / navigated within the patient's body. The EM sensor data may be acquired at a frequency (e.g., 30 Hz) where noise, such as Gaussian noise, may be present in the sensor data, as described above. Outputs from the DPM algorithm may include the position of the scope tip, located corresponding to the last input to the EM sensor data.

[0033] The DPM algorithm may be implemented by multiple functional units. In some cases, process 100 may begin by initializing (105) the parameters or configurations of the functional units. For example, parameters related to the acquisition of EM sensor data may be configured (e.g., the rate at which tracking data is broadcast, the rate at which to poll for new hardware status, the rate at which to attempt to reconnect to the hardware, the accepted field generator model identifier, a delay to account for latency in receiving a scope connection contract, etc.), and parameters related to the registration of EM sensor data into the patient CT scan coordinate system (e.g., the state machine smRegistration) may be configured, including, for example, the delay between registration update operations, a scalar that specifies the degree of smoothing of received tracking signals, the minimum allowable distance between EM tracking points, the percentage of tracheal length to use in processing the cloud of registration points, the distance from the target at which EM tracking points are discarded from the cloud of registration points, and the parameter used in positioning the field generators. Parameters related to the DPM localization and optimization algorithms may be configured, including, for example, the maximum number of threads to use for DPM localization optimization, the maximum number of hypotheses to maintain for the DPM localization process (e.g., the maximum number of hypotheses may be set to 2, 3, 4, 5, 6, etc.), a penalty weighting basis for the localization switching branching scheme, the maximum EM data samples per bin for bin filtering of pre-localization data, the maximum number of EM data samples that can be processed in a batch for batch processing of DPM localization, and various other parameters.

[0034] In some cases, process 100 may include loading a lung model at 110. The lung model may be generated based on C-arm video data or imaging data obtained using an imaging device such as a C-arm imaging system. For example, the lung model may be a 3D model generated based on a CT scan. The C-arm imaging system may include a source (e.g., an X-ray source) and a detector (e.g., an X-ray detector or X-ray imager). In some cases, a single C-arm source may provide the video or imaging data at 110. In some cases, different C-arm sources may provide the video or imaging data at 110. While any model of an organ or body cavity may be loaded, a lung model is loaded as shown. The lung model may correspond to a patient of interest. The lung model may include an airway tree model 115. Because the lung's most basic function is to bring air and blood into close proximity for gas exchange, the airway tree develops adjacent to the arterial vasculature and branches into tightly packed alveoli, which may be 100-200 μm in diameter. In some cases, the airway tree model may include a geometric model of the patient's lung airway tree, including centerline points, bifurcation information, and airway surface meshes.

[0035] Optionally, process 100 may confirm that the lung model was successfully loaded at 120. If the lung model was not successfully loaded at 120, process 100 may return to other operations of process 100, such as initialization at 105, or may display a flag indicating that the airway tree model was not successfully created or loaded. If the model was successfully loaded at 120, process 100 may proceed to acquiring and preprocessing EM data at 125 through EM data pre-localization 130. EM data pre-localization 130 may include continuously adding EM sensor data to an eventQueue. The EM data in the eventQueue may be filtered, transformed with EM-CT registration, and added to the DPM localization process.

[0036] In one example, EM data may be acquired at 125. The EM data may be acquired as a future instance at 135 or as real-time batch EM data at 140. The EM data may be acquired at multiple instances or on a recurring or recurring basis. For example, the EM data may be acquired at a particular frequency, such as 10-60 Hz. When the EM data is acquired at a particular frequency, the EM data may be queued. For example, the EM data may be queued for pre-processing. In another example, the EM data may be queued for localization. The queued EM sensor data may be processed as a batch input to a DPM algorithm. For example, a maximum number of EM data samples may be processed in a batch for batch processing of DPM localization. The number of batch values ​​may be 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, and any number equal to or less than 5.

[0037] The EM data may include position data corresponding to the tip of the scope. For example, the EM data may include a sequence of data points, each corresponding to the position of the scope tip, thereby forming a trajectory of the scope's current and previous positions in the EM data. In some cases, acquiring EM data from one or more EM sensors related to the scope tip may be performed by a registration state machine (e.g., smEmSensor in FIG. 14 ). The smEmSensor may be responsible for monitoring and broadcasting information provided by any connected hardware related to EM tracking. This may include the system control unit (SCU), system interface unit (SIU), magnetic field generators (FG), and EM sensors. The smEmSensor may expose hardware status information along with EM tracking signal data for the scope and reference sensors. The tracking signal may include a position (represented as a vector) and orientation (represented as a quaternion) describing the sensor orientation in EM space (the global space created by the connected magnetic field generators).

[0038] The EM data acquired at 125 and 135 may also be preprocessed. Preprocessing the EM data may include registering the EM data to a CT frame. The transformation that transfers the EM sensor data to the CT coordinate frame is sometimes referred to as EM-To-CT registration. EM-To-CT registration may be published at dynamic intervals. The registration of the EM sensor coordinates to the CT scan coordinates of the patient's lungs may be published as registration transformation data. In some cases, registration may include the use of a tomosynthesis board. Registration may include the use of a modified iterative closest point (ICP) algorithm. At a high level, the registration process may be illustrated by the registration state machine (smRegistration) in FIG. 15. smRegistration may be responsible for registering the EM sensor coordinates to the CT scan coordinates of the patient's lungs and then publishing the registration transformation data. The registration transformation data may then be used by smLocalization (e.g., as shown in FIG. 16) to transform the EM data to CT scan coordinates. smRegistration sends new updates while the user is in driving mode away from the lesion.

[0039] The EM data may be further processed to generate an EM data-based trajectory or path. In some cases, the EM data-based path may include evenly distributed points. For example, to better represent the path of the scope during the procedure, the EM data may be processed by applying binning filtering and temporal filtering to the EM data to distribute the points evenly in space and time. Alternatively, the points of the EM data-based path may not be evenly distributed.

[0040] In some cases, the trajectory / path based on the EM data may correspond to the current position of the scope tip. For example, the EM data may be bin-filtered to form a path from the trachea to the location where the last measurement (e.g., EM sensor data) was processed. In some cases, EM data beyond the current scope tip position may be omitted, so that if the scope is withdrawn, EM data paths beyond it may not be considered.

[0041] In some cases, process 100 includes performing optimization and post-processing on the EM data acquired at 125 and 135. The optimization can generate a localization result consisting of an estimated scope tip position. Obtaining the estimated scope tip position may include applying a deformable path mapping (DPM) algorithm to align the scope tip drive trajectory (the sequence of EM data) with the lung airway path. The DPM algorithm takes into account CT-body deviation and respiratory deformation by modeling the airway path as a deformable assumption and applies nonlinear optimization to estimate the deformation and find the best-fit path. To ensure that the DPM algorithm does not block the data flow and event processing of the state machine smLocalization (e.g., as shown in FIG. 16 ), the computationally intensive DPM process may occur in a separate thread.

[0042] In some cases, the DPM algorithm may operate on multiple localization hypotheses or path hypotheses. In some cases, each localization hypothesis may include one or more airway centerline points, in order from the trachea to possible scope tip positions. In some cases, one or more localization hypotheses may be identified based at least in part on the current scope tip position. For example, one or more localization hypotheses corresponding to one or more airways may be identified within a predetermined range of the current position of the scope tip or the EM data-based trajectory, and other hypotheses outside that range may be excluded. The identified one or more path hypotheses may be selected as those corresponding to the EM data-based trajectory for subsequent mapping operations.

[0043] In some cases, each hypothetical path may be dynamically updated as the EM data is updated or as the scope tip moves forward or backward. In some cases, one or more localization hypotheses may be dynamically added or removed from the hypothesis pool as the scope tip is driven through the airway. For example, when the scope tip reaches an airway bifurcation, another localization hypothesis is created representing a further path along which the scope tip could potentially traverse. In another example, infeasible hypotheses may be removed from the hypothesis pool. For example, hypothetical airway paths that are far beyond the range of measurement error and deformation, or hypotheses with probabilities that are too small after normalization, may be removed from the hypothesis pool.

[0044] In some cases, each localization hypothesis is mapped to the EM data path and solved by computing a deformable path mapping through nonlinear optimization (e.g., via the software package GMM_Reg). In some cases, the result is a deformed hypothetical path having deformation parameters 145. For example, the path hypotheses may be deformed to fit the EM data path by applying a nonlinear optimization algorithm to minimize the displacement and bending energies between each path hypothesis and the EM data path. In some cases, the deformed path with the highest probability may be the final result. For example, if the scope moves forward, a distal end point on the deformed path may be designated as the position of the scope tip, and if the scope moves backward, a proximal end point on the deformed path may be designated as the position of the scope tip. As an example, the highest probability may be calculated using a softmax algorithm or the like. The deformed path with the highest probability may be referred to as the most likely path. The deformed path may correspond to deformation parameters that may be related to tissue or organ deformation due to patient movement (e.g., breathing), deformation due to CT-body misalignment, and / or various other deformation factors. By transforming the path assumption into an EM-based path, the transformed path advantageously avoids the challenges of real-time CT-body dissociation or real-time movement of the patient by fitting to the EM-based path.

[0045] In some cases, mapping one or more deformable path hypotheses to an EM-based trajectory / path may include determining a match / fit between the deformable path hypotheses and the EM-based path by solving an optimization problem. For example, for each hypothesis, the average displacement energy and bending energy may be modeled as Equation 1 and Equation 2, respectively:

[0046]

number

[0047]

number

[0048]

number

[0049]

number

[0050]

number

[0051]

number

[0052]

number

[0053] In post-processing at 150, one or more hypotheses that meet a threshold or the path hypotheses with the highest probability may be selected as the most likely path map, and one or more corresponding path points that map to the last EM data location may be output as the location result at 155. In some cases, meeting a threshold may include selecting one hypothesis (e.g., the hypothesis with the highest probability) as the most likely path map. Additionally, in some cases, hypotheses are sorted by probability value from most to least, and any remaining hypotheses beyond the maximum number of hypotheses to retain are pruned.

[0054] As an example of implementing the above algorithm, a localization state machine, smLocalization (e.g., as shown in FIG. 16 ), may take EM data as input, apply an EM-CT registration transformation, and then estimate the most likely location of the scope tip within the patient's lung airway. The localization results may be published at 160 and used to generate virtual views to estimate the distance of the scope tip to a lesion. By way of example, data generated by the localization state machine may include scope tip position data (e.g., including unlocalized and localized positions). The unlocalized positions are the positions of the scope working channel, which are EM sensor data converted to CT coordinates by an EM-CT transformation. The localized positions may correspond to the best estimate of the scope camera's location within the airway, calculated by the localization algorithm to guide scope navigation as described above.

[0055] 2 shows an example batch localization process 200 using deformable path mapping. Process 200 may be similar to process 100. While process 100 provides an example of DMP localization, process 200 provides an example of batch DPM localization, including batch pre-processing optimization and batch post-processing. Some operations of process 200 may be the same as or similar to operations of process 100.

[0056] For example, newly acquired EM data is added to a queue at 205, such as by pre-localizing the EM data as described above. For example, EM data may be continuously added to the eventQueue until a maximum number of points is reached. The EM data in the eventQueue may be filtered, converted with EM-CT registration, and added to the DPM localization process. If the DPM localization is slow (by using a slower processor or due to debugging), the maximum queued size may be set to a number (e.g., 15 by default), meaning that the DPM localization updates the position at least once for every maximum (e.g., 15) of input EM sensor data. If the DPM localization process is fast enough, the queued size may be 1 in most cases. The process may generate an output of localized tip positions for all such EM sensor data input points.

[0057] The process may generate deformed EM data 210 based on deformation parameters 255 obtained from the most probable hypothesis 280. The deformed EM data may then be bin-filtered 215 to generate an EM-data-based path that includes evenly distributed points in both the time and spatial domains. Preprocessing the EM data may further include mapping the deformed EM-based path to a deformed lung model at 220. The process may then determine whether the scope tip is moving backward, forward, or not at 225. If no movement is detected, the result timestamp may be updated directly. If the scope tip is detected moving backward at 235, the localization result may be updated for each path hypothesis at 240. Also, at 240.1, the path hypotheses can be updated by adding or removing new hypotheses or pruning infeasible hypotheses; if it is determined at 230 that the scope tip is moving forward, the process can proceed to determine for each hypothesis at 230.1 whether the scope tip is passing through a bifurcation; if so, a new hypothesis can be added at 230.2, and the hypothesis pool is updated at 230.3.

[0058] Next, for each path hypothesis at 230.3, an optimization operation (e.g., gmm-reg) is performed at 260.1 to map the deformable hypothetical path to an EM-based path. The results of the optimization operation may include deformation parameters and deformation energy (e.g., bending energy) for each hypothesis at 265. The probability of each path hypothesis can then be determined based on the deformation parameters and energy at 270, and the probabilities can be normalized among all hypotheses at 275 to determine the most probable hypothesis at 280. The most probable path hypothesis can be output as a localization result at 285 and provided to a UI for display at 290.

[0059] Robotic Bronchoscopy System 3 illustrates an example of a robotic bronchoscopy system 300, 330, according to some examples. The robotic bronchoscopy system can implement the systems, methods, computer-readable media, and techniques described above. For example, the bronchoscopy system 300 can use a scope tip that can be located using the systems, methods, computer-readable media, and techniques described herein.

[0060] As shown in FIG. 3 , a robotic bronchoscopy system 300 may include a steerable catheter assembly 320 and a robotic support system 310 for supporting or carrying the steerable catheter assembly. The steerable catheter assembly may be a bronchoscope. In some embodiments, the steerable catheter assembly may be a single-use robotic bronchoscope. In some embodiments, the robotic bronchoscopy system 300 may include an instrument drive mechanism 313 attached to an arm of the robotic support system. The instrument drive mechanism may be provided by any suitable controller device (e.g., a handheld controller), which may or may not include a robotic system. The instrument drive mechanism may provide a mechanical and electrical interface to the steerable catheter assembly 320. The mechanical interface may allow the steerable catheter assembly 320 to be removably coupled to the instrument drive mechanism. For example, a handle portion of the steerable catheter assembly may be attached to the instrument drive mechanism via a quick-attach / release means, such as a magnet, a spring-loaded level, or the like. In some cases, the steerable catheter assembly may be manually coupled to or released from the instrument drive mechanism without the use of tools.

[0061] The steerable catheter assembly 320 may include a handle portion 323, which may include components configured to process image data, provide power, or establish communication with other external devices. For example, the handle portion 323 may include circuitry and communication elements that enable electrical communication between the steerable catheter assembly 320 and the instrument drive mechanism 313 and any other external system or device. In another example, the handle portion 323 may include circuit elements such as a power source for powering the endoscope's electronics (e.g., camera and LED light). In some cases, the handle portion may be in electrical communication with the instrument drive mechanism 313 via an electrical interface (e.g., a printed circuit board), and image / video data and / or sensor data may be received by a communication module of the instrument drive mechanism and transmitted to other external devices / systems. Alternatively or additionally, the instrument drive mechanism 313 may provide only a mechanical interface. The handle portion may be in electrical communication with a modular wireless communication device or any other user device (e.g., a portable / handheld device or controller) to transmit sensor data or receive control signals. More details about the handle portion are provided later in this specification.

[0062] The steerable catheter assembly 320 may include a flexible elongate member 311 coupled to a handle portion. In some embodiments, the flexible elongate member may include a shaft, a steerable tip, and a steerable portion. The steerable catheter assembly may be a single-use robotic bronchoscope. In some cases, only the elongate member may be disposable. In some cases, at least a portion of the elongate member (e.g., shaft, steerable tip, etc.) may be disposable. In some cases, the entire steerable catheter assembly 320, including the handle portion and elongate member, may be disposable. The flexible elongate member and handle portion are designed to allow the entire steerable catheter assembly to be disposable at low cost. More details about flexible elongate members and steerable catheter assemblies are provided later in this specification.

[0063] In some embodiments, the provided bronchoscopy system may also include a user interface. As shown in exemplary system 330, the bronchoscopy system may include a therapy interface module 331 (user console side) or a therapy control module 333 (patient and robot side). The therapy interface module may allow an operator or user to interact with the bronchoscope during a surgical procedure. In some embodiments, the therapy control module 333 may be a handheld controller. The therapy control module may optionally include a proprietary user input device and one or more add-on elements removably coupled to an existing user device to enhance the user input experience. For example, a physical trackball or roller may be provided with functionality similar to the graphic element it replaces, thereby replacing or supplementing the functionality of at least one virtual graphic element displayed on a graphical user interface (GUI) (e.g., a navigation arrow displayed on a touchpad). Examples of user devices may include, but are not limited to, a mobile device, a smartphone / cell phone, a tablet, a personal digital assistant (PDA), a laptop or notebook computer, a desktop computer, a media content player, etc. More details about user interface devices and user consoles are provided later in this specification.

[0064] The user console 331 may be mounted on the robotic support system 310. Alternatively or additionally, the user console or a portion of the user console (e.g., the treatment interface module) may be mounted on a separate mobile cart.

[0065] The present disclosure provides a robotic endoluminal platform with integrated intralesional tool tomosynthesis technology. In some cases, the robotic endoluminal platform may be a bronchoscopy platform. The platform may be configured to perform one or more operations consistent with the methods described herein. FIG. 4 illustrates an example of a robotic endoluminal platform and its components or subsystems according to some embodiments of the present invention. In some embodiments, the platform may include a robotic bronchoscopy system and one or more subsystems that can be used in combination with the robotic bronchoscopy system of the present disclosure.

[0066] In some embodiments, one or more subsystems may include an imaging system, such as a fluoroscopy imaging system, for real-time imaging of the target region (e.g., including the lesion). Multiple 2D fluoroscopy images may be used to generate tomosynthesis or cone-beam CT (CBCT) reconstructions for better visualization and to provide 3D coordinates of anatomical structures. FIG. 4 illustrates an example of a fluoroscopy (tomosynthesis) imaging system 400. For example, the fluoroscopy (tomosynthesis) imaging system may perform precise lesion location tracking or intralesional tool verification before or during a surgical procedure, as described above. In some cases, lesion location may be tracked based on positional data related to the fluoroscopy (tomosynthesis) imaging system / station (e.g., a C-arm) and image data captured by the fluoroscopy (tomosynthesis) imaging system. The lesion location may be registered to the coordinate frame of the robotic bronchoscopy system.

[0067] In some cases, the position, pose, or motion of the fluoroscopic imaging system may be measured / estimated to register the coordinate frame of an image to the robotic bronchoscopy system or to construct a 3D model / image. The pose or motion of the fluoroscopic (tomosynthesis) imaging system may be measured using any suitable motion / position sensors 410 disposed on the fluoroscopic (tomosynthesis) imaging system. The motion / position sensors may include, for example, an inertial measurement unit (IMU), one or more gyroscopes, velocity sensors, accelerometers, magnetometers, position sensors (e.g., Global Positioning System (GPS) sensors), visual sensors (e.g., imaging devices capable of detecting visible, infrared, or ultraviolet light, such as cameras), proximity or distance sensors (e.g., ultrasonic sensors, lidar, time-of-flight, or depth cameras), altitude sensors, attitude sensors (e.g., compasses), or field sensors (e.g., magnetometers, electromagnetic sensors, radio wave sensors). In some cases, one or more sensors for tracking the movement and position of the fluoroscopy (tomosynthesis) imaging station may be located on the imaging station or may be located remotely from the imaging station, such as a wall-mounted camera 420. A C-arm fluoroscopy (tomosynthesis) imaging system at various (rotational) poses while taking images of a subject. The various poses may be captured by one or more sensors as described above.

[0068] In some embodiments, the signal processing unit 430 may be used to segment the location of a lesion within image data captured by the fluoroscopy (tomosynthesis) imaging system. One or more processors of the signal processing unit may be configured to further overlay a treatment location (e.g., a lesion) on the real-time fluoroscopy image / video. For example, the processing unit may be configured to generate an augmentation layer that includes augmentation information, such as the location of the treatment location or target site. In some cases, the augmentation layer may also include a graphic marker indicating a path to the target site. The augmentation layer may be a substantially transparent image layer that includes one or more graphic elements (e.g., boxes, arrows, etc.). The augmentation layer may be overlaid on an optical view of the optical images or video stream captured by the fluoroscopy (tomosynthesis) imaging system or displayed on a display device. The transparency of the augmentation layer allows a user to view the optical image with the graphic elements overlaid. In some cases, both the segmented lesion image and an optimal path along which the elongated member has navigated to reach the lesion may be overlaid on the real-time tomosynthesis image. This may allow the operator or user to visualize the exact location of the lesion as well as the planned path of bronchoscope travel. In some cases, segmented and reconstructed images (e.g., CT images as described elsewhere) provided prior to operation of the systems described herein may be overlaid on the real-time images.

[0069] In some embodiments, one or more subsystems of the platform may include one or more therapeutic subsystems, such as a manual or robotic instrument (e.g., a biopsy needle, a biopsy forceps, a biopsy brush) or a manual or robotic therapeutic instrument (e.g., an RF ablation instrument, a cryoinstrument, a microwave instrument, etc.).

[0070] In some embodiments, one or more subsystems of the platform may include a navigation and localization subsystem. The navigation and localization subsystem may be configured to construct a virtual airway model based on preoperative images (e.g., preoperative CT images or tomosynthesis). The navigation and localization subsystem may be configured to identify segmented lesion locations within the 3D-rendered airway model, and based on the location of the lesion, the navigation and localization subsystem may generate an optimal path from the main bronchus to the lesion with a recommended approach angle to the lesion for performing a surgical procedure (e.g., biopsy).

[0071] In a registration step prior to driving the bronchoscope to the target site, the system may align the rendered virtual view of the airway with the patient's airway. Image registration may consist of a single registration step or a combination of a single registration step and real-time sensory updates to the registration information. The registration process may include finding a transformation that aligns objects (e.g., airway model, anatomical regions) between different coordinate systems (e.g., EM sensor coordinates and patient 3D model coordinates based on preoperative CT imaging). More details about registration are provided later in this specification.

[0072] Once registered, all airways can be aligned with the pre-procedure rendered airways. The position of the bronchoscope within the airways can be tracked and displayed while the robotic bronchoscope is being driven toward the target site. In some cases, a positioning sensor can be used to track the position of the bronchoscope relative to the airways. Also, using sensor fusion techniques, other types of sensors (e.g., cameras) can be used instead of or in conjunction with the positioning sensor. A positioning sensor, such as an electromagnetic (EM) sensor, can be embedded in the distal tip of the catheter, and an EM field generator can be positioned next to the patient's torso during the procedure. The EM field generator can identify the EM sensor's location in 3D space, or the EM sensor's location and orientation in 5D or 6D space. This can provide a visual guide to the operator while driving the bronchoscope toward the target site.

[0073] In real-time EM tracking, an EM sensor, consisting of one or more sensor coils embedded in a medical instrument (e.g., the tip of an endoscopic tool) at one or more locations and orientations, measures variations in an EM field generated by one or more static EM field generators positioned at locations near the patient. The position information detected by the EM sensor is stored as EM data. The EM field generator (or transmitter) can be placed near the patient to generate a low-intensity magnetic field that can be detected by the embedded sensor. The magnetic field induces small currents in the sensor coils of the EM sensor, which can be analyzed to determine the distance and angle between the EM sensor and the EM field generator. These distances and orientations can be registered to the patient's anatomy (e.g., a 3D model) during surgery to determine a registration transformation that aligns a single location in a coordinate system with a location in a pre-operative model of the patient's anatomy.

[0074] In some embodiments, the platforms herein can utilize a fluoroscopic imaging system to determine the position and orientation of medical instruments and patient anatomy within the coordinate system of the surgical environment. In particular, the systems and methods herein may employ mobile C-arm fluoroscopy as a low-cost, portable, real-time quantitative assessment tool. Fluoroscopy is an imaging modality that acquires real-time moving images of patient anatomy and medical instruments. The fluoroscopy system may include a C-arm system that provides positional flexibility and allows circular, horizontal, and / or vertical movement via manual or automated control. Fluoroscopic image data from multiple perspectives in the surgical environment (i.e., with the fluoroscopic imager moved between multiple positions) can be compiled to generate two-dimensional or three-dimensional tomographic images. When using a fluoroscopic imager system including a digital detector (e.g., a flat-panel detector), the generated and compiled fluoroscopic image data may enable planar image sectioning in parallel planes according to tomosynthesis imaging techniques. The C-arm imaging system may include a source (e.g., an X-ray source) and a detector (e.g., an X-ray detector or X-ray imager). The X-ray detector can generate an image representing the intensity of the received x-rays. The imaging system can reconstruct a 3D image based on multiple 2D images acquired from a wide range of angles. In some cases, the rotation angle range may be at least 120 degrees, 130 degrees, 140 degrees, 150 degrees, 160 degrees, 170 degrees, 180 degrees, or more. In some cases, the 3D image may be generated based on the pose of the X-ray imager.

[0075] The bronchoscope or catheter may be disposable. FIG. 5 illustrates an example of a flexible endoscope 500 according to some embodiments of the present disclosure. As shown in FIG. 5, the flexible endoscope 500 may include a handle / proximal portion 509 and a flexible elongate member that is inserted into a subject. The flexible elongate member may be the same as those described above. In some embodiments, the flexible elongate member may include a proximal shaft (e.g., insertion shaft 501), a steerable tip (e.g., tip 505), and a steerable portion (active bending portion 503). The active bending portion and proximal shaft portion may be the same as those described elsewhere herein. The endoscope 500 may also be referred to as a steerable catheter assembly, as described elsewhere herein. In some cases, the endoscope 500 may be a single-use robotic endoscope. In some cases, the entire catheter assembly may be disposable. In some cases, at least a portion of the catheter assembly may be disposable. In some cases, the entire endoscope may be disengaged from the instrument drive mechanism and be disposable. In some embodiments, the endoscope may include varying degrees of stiffness along the shaft to improve functional operation.

[0076] The endoscope or steerable catheter assembly 500 may include a handle portion 509, which may include one or more components configured to process image data, provide power, or establish communication with other external devices. For example, the handle portion may include circuitry and communication elements that enable electrical communication between the steerable catheter assembly 500 and instrument drive mechanism (not shown) and any other external systems or devices. In another example, the handle portion 509 may include circuit elements such as a power source for powering the endoscope's electronics (e.g., camera, electromagnetic sensors, and LED lights).

[0077] One or more components located in the handle may be optimized to allow for expensive and complex components to be allocated to the robotic support system, handheld controller, or instrument drive mechanism, thereby reducing costs and simplifying the design of the single-use endoscope. The handle or proximal portion may provide an electrical and mechanical interface to enable electrical and mechanical communication with the instrument drive mechanism. The instrument drive mechanism may include a motor set actuated to rotatably drive a pull wire set of the catheter. The handle portion of the catheter assembly may be mounted on the instrument drive mechanism such that its pulley / capstan assembly is driven by the motor set. The number of pulleys may vary based on the configuration of the pull wires. In some cases, one, two, three, four, or more pull wires may be utilized to articulate the flexible endoscope or catheter.

[0078] The handle portion may be designed to make the robotic bronchoscope low-cost and disposable. For example, conventional manual and robotic bronchoscopes may have cables at the proximal end of the bronchoscope handle. Often, the cables include illumination fibers, camera video cables, and other sensor fibers or cables, such as electromagnetic (EM) sensors or shape-sensing fibers. Such complex cables can be expensive and increase the cost of the bronchoscope. The provided robotic bronchoscope may have an optimized design that allows for simplified structures and components while retaining mechanical and electrical functionality. In some cases, the handle portion of the robotic bronchoscope may adopt a cable-less design while providing a mechanical / electrical interface to the catheter.

[0079] An electrical interface (e.g., a printed circuit board) allows image / video data or sensor data to be received by the communication module of the instrument drive and transmitted to other external devices / systems. In some cases, the electrical interface can establish electrical communication without cables or wires. For example, the interface may include pins soldered onto an electronics board such as a printed circuit board (PCB). For example, a receptacle connector (e.g., a female connector) may be provided on the instrument drive as a mating interface. This may advantageously allow the endoscope to be quickly plugged into the instrument drive or robotic support without utilizing extra cables. This type of electrical interface may also function as a mechanical interface, so that both a mechanical and electrical coupling are established when the handle portion is plugged into the instrument drive. Alternatively or additionally, the instrument drive may only have a mechanical interface. The handle portion may be in electrical communication with the modular wireless communication device or any other user device (e.g., a portable / handheld device or controller) to transmit sensor data or receive control signals.

[0080] In some cases, the handle portion 509 may include one or more mechanical control modules, such as luers 511, for interfacing to an irrigation / aspiration system. In some cases, the handle portion may include levers / knobs for articulation control. Alternatively, the articulation control may be located in a separate controller attached to the handle portion via the instrument drive mechanism.

[0081] The endoscope may be attached to a robotic support system or a handheld controller via an instrument drive mechanism. The instrument drive mechanism may be provided by any suitable controller device (e.g., a handheld controller), which may or may not include a robotic system. The instrument drive mechanism may provide a mechanical and electrical interface to the steerable catheter assembly 500. The mechanical interface may allow the steerable catheter assembly 500 to be removably coupled to the instrument drive mechanism. For example, a handle portion of the steerable catheter assembly may be attached to the instrument drive mechanism via a quick attachment / release means, such as a magnet, a spring-loaded level, or the like. In some cases, the steerable catheter assembly may be manually coupled to or released from the instrument drive mechanism without the use of tools.

[0082] In the illustrated example, the distal tip of the catheter or endoscope shaft is configured to articulate / bend with two or more degrees of freedom to provide a desired camera view or to control the direction of the endoscope. As shown in this example, an imaging device (e.g., a camera) and a position sensor (e.g., an electromagnetic sensor) 507 are located at the tip of the catheter or endoscope shaft 505. For example, the camera's line of sight may be controlled by controlling the articulation of the active bending section 503. In some cases, the angle of the camera may be adjustable to adjust the line of sight without or in addition to articulating the distal tip of the catheter or endoscope shaft. For example, the camera may be angled (e.g., tilted) relative to the axial direction of the endoscope tip using suitable components.

[0083] The distal tip 505 may be a rigid component that allows for positioning of sensors such as electromagnetic (EM) sensors, imaging devices (e.g., cameras), and other electronic components (e.g., LED light sources) embedded in the distal tip.

[0084] In real-time EM tracking, an EM sensor, consisting of one or more sensor coils embedded in a medical instrument (e.g., the tip of an endoscopic tool) at one or more locations and orientations, measures fluctuations in an EM field generated by one or more static EM field generators positioned at locations near the patient. The position information detected by the EM sensor is stored as EM data. The EM field generator (or transmitter) can be placed near the patient to generate a low-intensity magnetic field that can be detected by the embedded sensor. The magnetic field induces a small current in the sensor coil of the EM sensor, which can be analyzed to determine the distance and angle between the EM sensor and the EM field generator. For example, the EM field generator can be positioned near the patient's torso during a procedure to identify the EM sensor location in 3D space, or the EM sensor location and orientation in 5D or 6D space. This can provide a visual guide to the operator while navigating the bronchoscope toward the target site.

[0085] The endoscope may have a unique design for the elongate member. In some cases, the active bending section 503 and proximal shaft of the endoscope may consist of a single tube that includes a series of cuts (e.g., reliefs, slits, etc.) along its length to allow for increased flexibility, desired stiffness, and anti-slip features (e.g., the ability to define a minimum bend radius).

[0086] As discussed above, the active flexion section 503 may be designed to bend (e.g., articulate) in more than two degrees of freedom. The unique structure of the active flexion section allows for even greater degrees of flexion, such as 180 degrees and 270 degrees (or other articulation parameters for clinical indications). In some cases, a variable minimum bend radius along the axial axis of the elongate member may be provided such that the active flexion section can have two or more different minimum bend radii.

[0087] Articulation of the endoscope may be controlled by applying force to the distal tip of the endoscope via one or more pull wires. One or more pull wires may be attached to the distal tip of the endoscope. In the case of multiple pull wires, pulling on the wires one at a time can redirect the distal tip to swing up, down, left, right, or in any desired direction. In some cases, the pull wires may be fixed to the distal tip of the endoscope, extend through a bend, and enter a handle where they connect to a drive component (e.g., a pulley). This handle pulley may interact with an output shaft from the robotic system.

[0088] In some embodiments, the proximal end or proximal portion of one or more pull wires may be operably coupled to various mechanisms (e.g., gears, pulleys, capstans, etc.) in the handle portion of the catheter assembly. The pull wires may be metallic or polymeric wires, cables, or threads. The pull wires may also be made of natural or organic materials or fibers. The pull wires may be any type of suitable wire, cable, or thread capable of supporting various types of loads without deformation, significant deformation, or breakage. The distal end / portion of one or more pull wires may be fixed to or integral with the distal portion of the catheter, and the control unit may operate the pull wires to apply force or tension to the distal portion, thereby steering or articulating at least the distal portion (e.g., flexible portion) of the catheter (e.g., up, down, pitch, yaw, or any direction in between).

[0089] The pull wires may be made of any suitable material, such as stainless steel (e.g., SS316), a metal, an alloy, a polymer, a nylon, or a biocompatible material. The pull wires may be wires, cables, or threads. In some embodiments, different pull wires may be made of different materials to vary the load-bearing capabilities of the pull wires. In some embodiments, different portions of the pull wire may be made of different materials to vary stiffness and / or load-bearing capacity along the pull. In some embodiments, the pull wires may be utilized for the transmission of electrical signals.

[0090] The proximal design may improve device reliability without incurring extra costs, enabling a low-cost, single-use endoscope. In another aspect of the present invention, a single-use robotic endoscope is provided. The robotic endoscope may be a bronchoscope and may be the same as the steerable catheter assembly described elsewhere herein. Conventional endoscopes can be complex in design and are typically designed to be reused after procedures, which requires thorough cleaning, disinfection, or sterilization after each procedure. Existing endoscopes are often designed with complex structures to ensure they can withstand cleaning, disinfection, and sterilization processes. The provided robotic bronchoscope may be a single-use endoscope, which may advantageously reduce cross-contamination between patients and infectious diseases. In some cases, the robotic bronchoscope may be delivered to a physician in a pre-sterilized package and is intended to be disposed of after a single use.

[0091] As shown in FIG. 6 , the robotic bronchoscope 610 may include a handle portion 613 and a flexible elongate member 611. In some embodiments, the flexible elongate member 611 may include a shaft, a steerable tip, and a steerable / active bend. The robotic bronchoscope 610 may be the same as the steerable catheter assembly described in FIG. 5 . The robotic bronchoscope may be a single-use robotic endoscope. In some cases, only the catheter may be disposable. In some cases, at least a portion of the catheter may be disposable. In some cases, the entire robotic bronchoscope may be disposable by disengaging it from the instrument drive mechanism. In some cases, the bronchoscope may include varying degrees of stiffness along its shaft to improve functional operation. In some cases, the minimum bend radius along the shaft may vary.

[0092] The robotic bronchoscope can be detachably coupled to an instrument drive mechanism 620. The instrument drive mechanism 620 may be mounted on an arm of a robotic support system or any of the actuated support systems described elsewhere herein. The instrument drive mechanism may provide a mechanical and electrical interface to the robotic bronchoscope 610. The mechanical interface may allow the robotic bronchoscope 610 to be detachably coupled to the instrument drive mechanism. For example, a handle portion of the robotic bronchoscope can be attached to the instrument drive mechanism via a quick installation / release means such as a magnet and a spring-loaded level. In some cases, the robotic bronchoscope may be manually coupled to or released from the instrument drive mechanism without the use of tools.

[0093] 7 shows an example of an instrument drive mechanism 700B that provides a mechanical interface to a handle portion 713 of a robotic bronchoscope. As shown in this example, the instrument drive mechanism 700B may include a motor set that is actuated to rotatably drive a pull wire set of a flexible endoscope or catheter. The handle portion 713 of the catheter assembly may be mounted on the instrument drive mechanism such that its pulley assembly or capstan is driven by the motor set. The number of pulleys may vary based on the configuration of the pull wires. In some cases, one, two, three, four, or more pull wires may be utilized to articulate the flexible endoscope or catheter.

[0094] The handle portion may be designed to make the robotic bronchoscope low-cost and disposable. For example, conventional manual and robotic bronchoscopes may have cables at the proximal end of the bronchoscope handle. Often, the cables may include illumination fibers, camera video cables, and other sensor fibers or cables, such as electromagnetic (EM) sensors or shape-sensing fibers. Such complex cables can be expensive and increase the cost of the bronchoscope. The provided robotic bronchoscope may have an optimized design that allows for simplified structures and components while retaining mechanical and electrical functionality. In some cases, the handle portion of the robotic bronchoscope may employ a cable-less design while providing a mechanical / electrical interface to the catheter.

[0095] FIG. 8 shows an example of a distal tip 800 of an endoscope. In some cases, the distal portion or tip of the catheter 800 may be substantially flexible so that it can be steered in one or more directions (e.g., pitch, yaw). The catheter may include a tip section, a bending section, and an insertion shaft. In some embodiments, the catheter may have variable bending stiffness along its longitudinal axis. For example, the catheter may include multiple sections with different bending stiffnesses (e.g., flexible, semi-rigid, and rigid). The bending stiffness may be varied by selecting materials with different stiffness / rigidity, varying the structure in different segments (e.g., cuts, patterns), adding additional support components, or any combination thereof. In some embodiments, the catheter may have a variable minimum bend radius along its longitudinal axis. Selecting different minimum bend radii at different locations along the catheter can advantageously provide anti-dislodgement functionality while allowing the catheter to reach hard-to-reach areas. In some cases, the proximal end of the catheter does not need to be highly flexible, so the proximal portion of the catheter may be reinforced with additional mechanical structure (e.g., additional layers of material) to achieve greater bending stiffness. Such a design can provide support and stability to the catheter. In some cases, variable bending stiffness may be achieved by using different materials during extrusion of the catheter. This may advantageously allow for different degrees of stiffness along the catheter shaft during the extrusion manufacturing process without the need for additional fastening or assembly of different materials.

[0096] The distal portion of the catheter may be manipulated by one or more pull wires 805. The distal portion of the catheter may be made of any suitable material, such as a copolymer, polymer, metal, or alloy, so that it can be bent by the pull wires. In some embodiments, the proximal or terminal ends of the one or more pull wires 805 may be coupled to a drive mechanism (e.g., a gear, pulley, capstan, etc.) via a locking mechanism, as described above.

[0097] The pull wires 805 may be metallic wires, cables, or threads, or polymeric wires, cables, or threads. The pull wires 805 may also be made of natural or organic materials or fibers. The pull wires 805 may be any type of suitable wire, cable, or thread capable of supporting various types of loads without deformation, significant deformation, or breakage. The distal ends or distal portions of one or more pull wires 805 may be fixed to or integral with the distal portion of the catheter, and when the control unit operates the pull wires, a force or tension can be applied to the distal portion, causing at least the distal portion (e.g., flexible portion) of the catheter to be steered or articulated (e.g., up, down, pitch, yaw, or any direction in between).

[0098] The catheter may have dimensions that allow one or more electronic components to be integrated into the catheter. For example, the outer diameter of the distal tip may be approximately 4 to 4.4 millimeters (mm), and the diameter of the working channel may be approximately 2 mm, allowing one or more electronic components to be embedded in the wall of the catheter. However, it should be noted that, depending on different applications, the outer diameter may be within any range less than 4 mm or greater than 4.4 mm, and the diameter of the working channel may be within any range depending on the tool size or specific application.

[0099] The one or more electronic components may include an imaging device, an illumination device, or a sensor. In some embodiments, the imaging device may be a video camera 813. The imaging device may include optical elements and an image sensor to capture image data. The image sensor may be configured to generate image data according to wavelengths of light. Various image sensors, such as a complementary metal-oxide semiconductor (CMOS) or a charge-coupled device (CCD), may be employed to capture the image data. The imaging device may be a low-cost camera. In some cases, the image sensor may be provided on a circuit board. The circuit board may be an imaging printed circuit board (PCB). The PCB may include multiple electronic elements to process the image signal. For example, the circuitry of a CCD sensor may include an A / D converter and an amplifier to amplify and convert the analog signal provided by the CCD sensor. Optionally, the image sensor may be integrated with an amplifier and a converter to convert the analog signal to a digital signal without the need for a circuit board. In some cases, the output of the image sensor or circuit board may be image data (a digital signal), which may be further processed by the camera's camera circuitry or processor. In some cases, the image sensor may comprise an array of optical sensors.

[0100] The illumination device may include one or more light sources 811 positioned at the distal tip. The light sources may be light emitting diodes (LEDs), organic LEDs (OLEDs), quantum dots, or any other suitable light source. In some cases, the light sources may be miniaturized LEDs or dual-tone flash LED illumination for compact designs.

[0101] The imaging and illumination devices may be integrated into the catheter. For example, the distal portion of the catheter may be provided with appropriate structure matching at least the dimensions of the imaging and illumination devices. The imaging and illumination devices may be embedded in the catheter. FIG. 9 shows an exemplary distal portion of a catheter with an integrated imaging and illumination device. The camera may be positioned in the distal portion. The distal tip may have structure to accommodate the camera, illumination device, or position sensor. For example, the camera may be embedded within a cavity 910 at the distal tip of the catheter. The cavity 910 may be integrally formed with the distal portion of the cavity and may have dimensions matching the length / width of the camera so that the camera does not move relative to the catheter. The camera may be adjacent to the working channel 920 of the catheter to provide a close-up view of the tissue or organ. In some cases, the attitude or orientation of the imaging device may be controlled by controlling the rotational movement (e.g., roll) of the catheter.

[0102] Power for the camera may be provided by a wired cable. In some cases, the cable wires may be present in a wire bundle that provides power to the camera as well as lighting elements or other circuitry at the distal tip of the catheter. The camera or light source may be powered from a power source located in the handle portion via wires, copper wires, or any other suitable means extending through the length of the catheter. In some cases, real-time images or videos of the tissue or organ may be transmitted wirelessly to an external user interface or display. The wireless communication may be WiFi, Bluetooth, RF communication, or other forms of communication. In some cases, images or videos captured by the camera may be broadcast to multiple devices or systems. In some cases, image or video data from the camera may be transmitted down the length of the catheter via wires, copper wires, or any other suitable means to a processor located in the handle portion. Image or video data may be transmitted to an external device / system via wireless communication components in the handle portion. In some cases, the system may be designed so that wires are not visible or exposed to the operator.

[0103] In conventional endoscopes, illumination may be provided by a fiber optic cable that transmits light from a light source positioned at the proximal end of the endoscope to the distal end of the robotic endoscope. In some embodiments of the present disclosure, to reduce design complexity, a miniaturized LED light may be employed and embedded in the distal section of the catheter. In some cases, the distal section may include a structure 930 with dimensions that match those of the miniaturized LED light source. As shown in the illustrated example, two cavities 530 may be integrally formed with the catheter to accommodate two LED light sources. For example, the outer diameter of the distal tip may be approximately 4 to 4.4 millimeters (mm), and the diameter of the working channel of the catheter may be approximately 2 mm to accommodate two LED light sources embedded in the distal end. The outer diameter may be within any range less than 4 mm or greater than 4.4 mm, and the diameter of the working channel may be within any range depending on the dimensions of the tool or the specific application. Any number of light sources may be included. The internal structure of the distal section may be designed to accommodate any number of light sources.

[0104] In some cases, each LED may be connected to a power wire that may extend to the proximal handle. In some embodiments, the LEDs may be soldered to separate power wires that are later bundled together to form a single stranded wire. In some embodiments, the LEDs may be soldered to a pull wire that provides power. In other embodiments, the LEDs may be crimped or connected directly to a pair of power wires. In some cases, a protective layer, such as a thin layer of biocompatible adhesive, may be applied to the front of the LEDs to provide protection while still allowing light to be emitted. In some cases, an additional cover 931 may be placed on the front face of the distal tip to allow for precise positioning of the LEDs and sufficient space for the adhesive. The cover 931 may be made of a transparent material that matches the refractive index of the adhesive so that the illumination light is not obstructed.

[0105] Example of a location assumption 10 and 11 show example lung biomodels 1000 and 1100, respectively, including EM data and multiple localization hypotheses. Examples 1000 and 1100 may, in some cases, illustrate internal computational processes. Examples 1000 and 1100 may, in some cases, be presented in whole or in part on a graphical display. Examples 1000 and 1100 may, in some cases, represent processes 100 or 200. Examples 1000 and 1100 may, in some cases, represent hardware, software, and techniques described with respect to FIGS. 3-9 (e.g., endoscopy, bronchoscopy, etc.).

[0106] Examples 1000 and 1100 each include a lung model. The lung model may be generated based on a CT scan. EM data is also shown in examples 1000 and 1100. The EM data may be acquired using an EM sensor at the tip of a scope (e.g., a bronchoscope tip). As shown, the EM data may be registered, i.e., the EM data may be transformed to the same set of coordinates as the lung model in the CT scan. Examples 1000 and 1100 show multiple localization hypotheses, each corresponding to a different possible path down the lung model based on the EM data. For example, localization hypothesis 1000 or path hypothesis 1100 may have a shape based on a segment of the centerline of the airway, where the centerline is determined based on the CT scan, and the location of the path hypothesis may be based on registration. As shown, the multiple localization hypotheses are within a specific distance from the EM data. In particular, as shown, the EM data is within a specific distance from the most recent point of the EM data. By doing so, examples 1000 and 1100 can avoid assumption 1111, which is highly unlikely due to how far away they are from the most recent point in the EM data.

[0107] As shown in example 1100, deformable path hypotheses may be dynamically added and removed as the scope tip traverses along the airway. For example, as the scope tip reaches the first bifurcation 1102 (scope trajectory 1101), a new hypothesis 1103 may be added. As the scope tip reaches the second bifurcation 1104, a new hypothesis 1105 may be added. Other hypotheses 1111 that fall outside a predetermined distance range of the scope tip or EM data points may be removed.

[0108] At a high level, examples 1000 and 1100 illustrate (A) modeling the EM data as registered paths; (B) applying binning and temporal filtering to distribute the EM data evenly in space and time (e.g., to better represent the scope path during the procedure); (C) arranging all possible airway paths in the lung model within a certain range (e.g., distance) as possible matches with the EM data, sometimes referred to as localization hypotheses; (D) creating additional localization hypotheses as the scope moves further and the EM data is updated (e.g., when the scope reaches an airway bifurcation, another localization hypothesis is created to represent the two possibilities); and (E) mapping the airway path of each localization hypothesis with the EM data, where the mapping is an optimization problem that solves deformation and shape matching. A deformable path mapping (DPM) operation can be performed, which consists of (i) mapping the localization hypotheses, which is formed as a problem, (ii) solving an optimization problem to estimate the deformation and shape matching mapping for each of the localization hypotheses, (iii) estimating the probability of each of the localization hypotheses based on the deformation and shape matching mapping results (e.g., using the normalized probabilities of the existing hypotheses to determine the best mapping solution), (iv) outputting the localization hypotheses with the highest probability, (v) pruning infeasible localization hypotheses (e.g., localization hypotheses that are far beyond the measurement error and deformation range or whose normalized probabilities are below a threshold), and (v) passing the remaining (unpruned) localization hypotheses and incrementally continuing the evaluation using updated EM data acquired as the scope continues to move further through the patient. In some cases, the DPM operation is performed and solved in real time, so dynamic deformations such as respiratory motion may be taken into account. In some cases, the EM data of the lungs and airways may be modeled as a Gaussian mixture model. In some cases, DPM optimization may be solved as an optimization problem in combination with an appropriate regularization function. In some cases, the optimization is solved as a nonlinear optimization. In some cases, binning filtering of the EM data can be useful to obtain sharp and well-distributed paths.In some cases, DPM operations may take into account CT-body disparity. In some cases, DPM operations can be applied to localization using other sensors, such as scope shape sensing techniques.

[0109] Computer Systems The present disclosure provides computer systems programmed to implement the methods, computer-readable media, and techniques of the present disclosure. Figure 12 illustrates a computer system 1201 that is programmed or otherwise configured to operate the methods, computer-readable media, and techniques described herein (such as the methods for locating the scope tip of an endoscope within an organ of a patient described herein). For example, computer system 1201 may implement processes 100 or 200. In another example, user interface 1240 may present one or more of the example models described with respect to Figures 10 or 11.

[0110] The computer system 1201 can coordinate various aspects of the present disclosure, such as techniques for locating the scope tip of an endoscope within a patient's organ. The computer system 1201 can be a user's electronic device or a computer system located remotely relative to the electronic device. The electronic device can be a mobile electronic device.

[0111] Computer system 1201 includes a central processing unit (CPU, also referred to herein as a "processor" and "computer processor") 1205, which may be a single-core or multi-core processor, or multiple processors for parallel processing. Computer system 1201 also includes memory or storage locations 1210 (e.g., random access memory, read-only memory, flash memory), electronic storage 1215 (e.g., a hard disk), a communication interface 1220 (e.g., a network adapter) for communicating with one or more other systems, and peripheral devices 1225, such as cache, other memory, data storage, and / or an electronic display adapter. Memory 1210, storage 1215, interface 1220, and peripheral devices 1225 communicate with CPU 1205 via a communication bus (solid lines) such as a motherboard. Storage 1215 may be a data storage device (or data repository) for storing data. Computer system 1201 may be operably coupled to a computer network ("network") 1230 using communication interface 1220. Network 1230 can be the Internet, an Internet and / or an extranet, or an intranet and / or an extranet in communication with the Internet. Network 1230 is, in some cases, a telecommunications or data network. Network 1230 can include one or more computer servers that enable distributed computing, such as cloud computing. Network 1230 can, in some cases, implement a peer-to-peer network with the aid of computer system 1201, allowing devices coupled to computer system 1201 to operate as clients or servers.

[0112] The CPU 1205 can execute instructions on a computer-readable medium, which may be embodied in a program or software. The instructions may be stored in a memory location, such as the memory 1210. The instructions may be directed to the CPU 1205, which may then be programmed or otherwise configured to implement the methods of the present disclosure. Examples of operations performed by the CPU 1205 may include fetch, decode, execute, and writeback.

[0113] The CPU 1205 may be part of a circuit, such as an integrated circuit. One or more other components of the system 1201 may be included in the circuit. In some cases, the circuit is an application specific integrated circuit (ASIC).

[0114] The storage device 1215 can store files such as drivers, libraries, and saved programs. The storage device 1215 can store user data, such as user preferences and user programs. The computer system 1201 can optionally include one or more additional data storage devices external to the computer system 1201, such as located on a remote server that communicates with the computer system 1201 over an intranet or the Internet.

[0115] Computer system 1201 can communicate with one or more remote computer systems via network 1230. For example, computer system 1201 can communicate with a remote computer system of a user (e.g., a medical device operator). Examples of remote computer systems include a personal computer (e.g., a portable PC), a slate or tablet PC (e.g., an Apple® iPad, a Samsung® Galaxy Tab), a telephone, a smartphone (e.g., an Apple® iPhone, an Android-enabled device, a Blackberry®), or a personal digital assistant. A user can access computer system 1201 via network 1230.

[0116] Methods as described herein may be implemented by machine (e.g., computer processor) executable code stored in electronic storage locations of computer system 1201, such as memory 1210 or electronic storage 1215. The instructions may be code stored on a computer-readable medium and may be provided in the form of software. During use, the code may be executed by processor 1205. In some cases, the code may be retrieved from storage 1215 and stored in memory 1210 for immediate access by processor 1205. In some situations, electronic storage 1215 may be omitted, and machine-executable instructions are stored in memory 1210.

[0117] The code may be pre-compiled and configured for use on a machine having a processor adapted to execute the code, or it may be compiled at run-time. The code may be provided in a programming language that can be selected to allow the code to be executed in a pre-compiled or as-compiled manner.

[0118] Aspects of the systems and methods provided herein, such as computer system 1201, can be embodied in programming. Various aspects of the technology can be considered "products" or "articles of manufacture," typically in the form of computer-readable media storing instructions, such as code or associated data, which are held or embodied on one type of computer-readable medium. Machine-executable code can be stored in electronic storage devices, such as memory (e.g., read-only memory, random-access memory, flash memory) or a hard disk. "Storage" type media can include any or all of the tangible memory of a computer, processor, etc., or their associated modules, such as various semiconductor memories, tape drives, disk drives, etc., that may provide non-transitory storage for software programming at any time. All or portions of the software may be communicated over the Internet or various other telecommunications networks. Such communication may, for example, enable loading of software from one computer or processor to another, e.g., from a management server or host computer to an application server computer platform. Accordingly, other types of media that can hold software elements include light waves, radio waves, and electromagnetic waves used across physical interfaces between local devices via wired and optical landline networks, as well as various airlinks. The physical elements that carry such waves, such as wired or wireless links, optical links, etc., may also be considered media that carry software. As used herein, unless limited to non-transitory, tangible "storage" media, terms such as computer or machine "readable medium" refer to any medium or media that participate in providing instructions to a processor for execution.

[0119] Thus, a computer-readable medium such as a computer-executable code may take many forms, including, but not limited to, a tangible storage medium, a carrier wave medium, or a physical transmission medium. Non-volatile storage media include, for example, optical or magnetic disks, such as any of the storage devices of any computer, such as those that may be used to implement the databases, etc., shown in the figures. Volatile storage media include dynamic memory, such as the main memory of such a computer platform. Tangible transmission media include copper wire and fiber optics, including coaxial cables, i.e., the wires that comprise a bus within a computer system. Carrier-wave transmission media may take the form of electric or electromagnetic signals, or acoustic or light waves such as those generated during radio frequency (RF) and infrared (IR) data communications. Thus, common forms of computer readable media include, for example, a floppy disk, a flexible disk, a hard disk, magnetic tape, any other magnetic medium, a CD-ROM, a DVD, or a DVD-ROM, any other optical medium, punched cards, paper tape, any other physical storage medium with a pattern of holes, RAM, ROM, PROM and EPROM, FLASH-EPROM, any other memory chip or cartridge, a carrier wave carrying data or instructions, a cable or link carrying such a carrier wave, or any other medium from which a computer can read programming code or data. Many of these forms of computer readable media may be involved in carrying one or more sequences of one or more instructions to a processor for execution.

[0120] The computer system 1201 may include or communicate with an electronic display 1235, which may include a user interface (UI) 1240 for providing, for example, tomosynthesis (e.g., tomosynthesis reconstruction) or fluoroscopy (e.g., enhanced fluoroscopy) data, e.g., text, video, images, etc. Examples of a UI include, but are not limited to, a graphical user interface (GUI), a web-based user interface, or an application programming interface (API). The UI 1240 may also be used for input, in some cases, via touchscreen functionality.

[0121] The methods, systems, instructions, and techniques of the present disclosure may be implemented by one or more algorithms. The algorithm may be implemented in software when executed by the central processing unit 1205. The algorithm may, for example, (a) acquire (i) a biological model of a patient's organ and (ii) electromagnetic (EM) data generated by an endoscope within the organ, (b) generate multiple scope tip localization hypotheses based on the biological model and the EM data, (c) generate multiple transformations, each of the multiple transformations mapping each of the multiple scope tip localization hypotheses to the EM data, (d) determine a scope tip localization from the multiple scope tip localization hypotheses, the localization corresponding to an assumed transformation of the multiple transformations that satisfies a threshold, and (e) present the scope tip localization on a graphical display.

[0122] Exemplary Methods 13 illustrates an exemplary method 1300 for locating a scope tip of an endoscope within a patient's organ, consistent with certain examples of the present disclosure. The method may include acquiring (i) a biological model of a patient's body part and (ii) electromagnetic (EM) data generated by the endoscope (ACT 1305), generating an EM data-based path based on a sequence of the EM data (ACT 1310), identifying one or more path hypotheses based on the biological model, each of the one or more path hypotheses including deformation parameters (ACT 1315), determining a probability associated with each of the one or more path hypotheses by mapping the one or more path hypotheses to a path based on the EM data (ACT 1320), and determining a best-fit path from the one or more path hypotheses as a location of the scope tip based at least in part on the probability associated with each of the one or more path hypotheses (ACT 1325). Method 1300 may implement one or more of the systems, computer-readable media, techniques, etc. described herein.

[0123] In some cases, method 1300 may begin at block 1305 with obtaining (i) a biological model of a patient's organ and (ii) EM data generated by an endoscope located within the organ. In some cases, the organ is a lung and the endoscope is a bronchoscope. In some cases, the biological model of the patient's organ is generated by a computed tomography (CT) scan.

[0124] In some cases, method 1300 may include, at block 1310, generating multiple localization hypotheses of the scope tip based on the biological model and the EM data. In some cases, method 1300 may include, prior to block 1310, transforming the EM data into the coordinate frame of the biological model (e.g., as described with respect to smRegistration). In some cases, each of the localization hypotheses includes an assumed path of the scope tip through the organ (e.g., as shown in FIGS. 10 and 11 ). In some cases, each of the localization hypotheses includes an assumed position of the scope tip within the organ. In some cases, each of the localization hypotheses includes an assumed orientation of the scope tip within the organ. In some cases, each of the multiple hypotheses is within a range of a current data point of the EM data. In some cases, the range may be a predetermined number determined based on experimental data. In some cases, the range may be configurable by the user.

[0125] Optionally, method 1300 may include generating a plurality of transformations at block 1315, where each transformation of the plurality of transformations respectively maps each of a plurality of scope tip localization hypotheses to the EM data. Optionally, the plurality of transformations includes lung transformations due to patient breathing. Optionally, the plurality of transformations includes transformations due to CT-body dissociation.

[0126] In some cases, method 1300 may include, at block 1320, determining a scope tip location from a plurality of scope tip location hypotheses, the location location corresponding to an assumed deformation among the plurality of deformations that satisfies a threshold. In some cases, method 1300 may include, prior to block 1320, generating a plurality of probability scores corresponding to the plurality of deformations. In some cases, the threshold is based on each probability score of the plurality of probability scores corresponding to a respective deformation among the plurality of deformations. In some cases, the probability score of the assumed deformation corresponding to the scope tip location satisfies the threshold based on the probability score being greater than all other probability scores among the plurality of probability scores.

[0127] Optionally, method 1300 may include, at block 1325, causing the localization of the scope tip to be presented on a graphical display. Optionally, the localization of the scope tip, when presented on the graphical display, is overlaid on a biological model of the organ. Optionally, method 1300 may further include determining a distance of the scope tip to a lesion within the organ based on the localization. Optionally, method 1300 may further include (i) updating, by one or more processors, the electromagnetic (EM) data to include new EM data, where the new EM data is generated by the endoscope, and (ii) performing operations corresponding to blocks 1310, 1315, 1320, and 1325 using the updated EM data. This may continue iteratively as updated EM data is acquired. For example, if updated EM data is acquired at a frequency of 30 Hz, the operations corresponding to blocks 1310, 1315, 1320, and 1325 may be performed at a frequency of 30 Hz using the updated EM data. In another example, if updated EM data is acquired at a frequency of 30 Hz, the operations corresponding to blocks 1310, 1315, 1320, and 1325 may be performed using the updated EM data at a frequency less than 30 Hz (e.g., 5 times per second, 1 time per second, 10 times per minute, etc.).

[0128] Further considerations While preferred embodiments of the present invention have been shown and described herein, it will be obvious to those skilled in the art that such embodiments are provided by way of example only. It is not intended that the present invention be limited by the specific examples provided herein. While the present invention has been described with reference to the foregoing specification, the description and illustration of the embodiments herein are not intended to be construed in a limiting sense. Numerous variations, changes, and substitutions will now occur to those skilled in the art without departing from the invention. Furthermore, it is to be understood that all aspects of the present invention are not limited to the specific descriptions, configurations, or relative proportions set forth herein, depending upon a variety of conditions and variables. It is to be understood that various alternatives to the embodiments of the present invention described herein may be employed in practicing the invention. Therefore, it is contemplated that the present invention also encompasses any and all such alternatives, modifications, variations, or equivalents. The following claims define the scope of the invention, and it is intended that methods and structures within the scope of these claims and their equivalents be covered thereby.

[0129] While various embodiments of the present invention have been shown and described herein, it will be obvious to those skilled in the art that such embodiments are provided by way of example only. Numerous variations, changes, and substitutions may occur to those skilled in the art without departing from the invention. It will be understood that various alternatives to the embodiments of the invention described herein may be used.

[0130] Whenever the terms "at least," "greater than," or "greater than or equal to" are listed before the first number in a series of two or more numbers, the terms "at least," "greater than," or "greater than or equal to" apply to each and every number in the series. For example, 1, 2, or 3 or more is equivalent to 1 or more, 2 or more, or 3 or more.

[0131] Whenever the term "no more than," "less than," or "less than or equal to" appears before the first number in a series of two or more numbers, the term "no more than," "less than," or "less than or equal to" applies to each and every number in the series. For example, 3, 2, or 1 or less is equivalent to 3 or less, 2 or less, or 1 or less.

[0132] Any reference herein to the term "or" is intended to mean what is also known as "inclusive or" or "logical OR," and it should be understood that when used as a logical statement, the phrase "A or B" is true if either A or B is true, or if both A and B are true, and when used as a list of elements. The phrase "A, B, or C" is intended to include all combinations of the elements listed in the phrase, e.g., any element selected from the group consisting of A, B, C, (A,B), (A,C), (B,C), and (A,B,C), and so on, if additional elements are listed. It should also be understood that the indefinite article "a" or "an" and the corresponding related definite article "the" or "said," respectively, are intended to mean one or more, unless otherwise stated, implied, or physically impossible. Furthermore, it should be understood that the phrases "at least one of A and B, etc.", "at least one of A or B, etc.", "selected from A and B, etc.", and "selected from A or B, etc." are each intended to mean either any listed element individually or any combination of two or more elements of any element from the group consisting of, for example, "A," "B," and "A and B together," etc.

[0133] Certain inventive embodiments herein contemplate numerical ranges. When a range exists, it includes the endpoints of the range. Furthermore, all subranges and values ​​within the range exist as if explicitly written out. The term "about" or "approximately" can mean within an acceptable error range of a value, which depends in part on how the value is measured or determined, e.g., the limitations of the measurement system. For example, "about" can mean within one or more standard deviations, in accordance with the practice in the art. Alternatively, "about" can mean within a range of up to 20%, up to 10%, up to 5%, or up to 1% of a given value. When values ​​are described in this application and claims, unless otherwise specified, the term "about" can be assumed to mean within an acceptable error range of the particular value.

[0134] It should be noted that the various example or suggested ranges described herein are specific to those example embodiments and are not intended to limit the scope or reach of the disclosed technology, but rather merely provide example ranges of frequencies, amplitudes, etc. associated with their respective embodiments or use cases.

[0135] While preferred embodiments of the present invention have been shown and described herein, it will be obvious to those skilled in the art that such embodiments are provided by way of example only. It is not intended that the present invention be limited by the specific examples provided herein. While the present invention has been described with reference to the foregoing specification, the description and illustration of the embodiments herein are not intended to be construed in a limiting sense. Numerous variations, changes, and substitutions will now occur to those skilled in the art without departing from the invention. Furthermore, it is to be understood that all aspects of the present invention are not limited to the specific descriptions, configurations, or relative proportions set forth herein, depending upon a variety of conditions and variables. It is to be understood that various alternatives to the embodiments of the present invention described herein may be employed in practicing the invention. Therefore, it is contemplated that the present invention also encompasses any and all such alternatives, modifications, variations, or equivalents. The following claims define the scope of the invention, and it is intended that methods and structures within the scope of these claims and their equivalents be covered thereby.

Claims

1. 1. A method for locating an endoscope in a body part of a patient, comprising: (a) acquiring a sequence of electromagnetic (EM) data; (b) generating an EM data-based path based at least in part on the sequence of EM data; and (c) identifying one or more path hypotheses based at least in part on data points from the sequence of EM data, the one or more path hypotheses having a shape based on a biological model of the body part of the patient; (d) generating one or more deformed paths by mapping the one or more path hypotheses to a path based on the EM data using an optimization algorithm; and (e) selecting a variant path from the one or more variant paths based at least in part on a probability associated with each of the one or more path hypotheses, and determining a position of the tip of the endoscope based on the selected variant path; A method comprising:

2. The method of claim 1 , wherein prior to (c), the EM data-based path is transformed into the coordinate frame of the biological model.

3. The method of claim 1 , wherein the biological model includes one or more airways.

4. The method of claim 3 , wherein the EM data is obtained while navigating the tip of the endoscope forward or backward along the one or more airways.

5. The method of claim 1 , wherein the EM data-based path is generated by applying binning filtering and / or temporal filtering to the EM data.

6. The method of claim 5 , wherein the EM data-based path comprises a sequence of points evenly distributed in both the spatial and temporal domains and represents a drive trajectory of the tip of the endoscope.

7. The method of claim 1 , wherein the body part is a lung and the endoscope is a bronchoscope.

8. The method of claim 1 , wherein deformation parameters indicate deformation of the body part due to movement of the patient.

9. The method of claim 1 , wherein the biological model of the body part of the patient is generated based on a computed tomography (CT) scan.

10. The method of claim 1 , wherein the deformation parameter is indicative of deformation due to CT-body dissociation.

11. The method of claim 1 , wherein the one or more path hypotheses are within a predetermined position range from a most recent data point in the sequence of EM data, are within a measurement error or deformation threshold, or exceed a likelihood threshold.

12. The method of claim 1 , wherein each of the one or more path hypotheses includes at least a portion of a centerline of the airway of the biological model.

13. The method of claim 1 , further comprising dynamically adding or removing path hypotheses while the tip of the endoscope is being driven through the body part.

14. 2. The method of claim 1 , wherein mapping the one or more path hypotheses to the EM data-based path comprises applying the optimization algorithm to determine deformation and shape matches between the one or more path hypotheses and the EM data-based path.

15. The method of claim 1 , wherein the selected deformation path from the one or more deformation path paths is associated with the highest probability.

16. The method of claim 15 , wherein the highest probability is determined based at least in part on normalized probabilities associated with each of the one or more path hypotheses.

17. 1. A system for locating an endoscope in a body portion of a patient, the system comprising: a memory storing computer-executable instructions; and one or more processors in communication with the endoscope, the processor comprising: (a) acquiring a sequence of electromagnetic (EM) data; (b) generating an EM data-based path based at least in part on the sequence of EM data; and (c) identifying one or more path hypotheses based at least in part on data points from the sequence of EM data, the one or more path hypotheses having a shape based on a biological model of the body part of the patient; (d) generating one or more deformed paths by mapping the one or more path hypotheses to a path based on the EM data using an optimization algorithm; and (e) selecting a variant path from the one or more variant paths based at least in part on a probability associated with each of the one or more path hypotheses, and determining a position of the tip of the endoscope based on the selected variant path; and one or more processors configured to execute the computer-executable instructions to perform the steps of:

18. 20. The system of claim 17, wherein prior to (c), the EM data-based path is transformed into the coordinate frame of the biological model.

19. 20. The system of claim 17, wherein the biological model includes one or more airways.

20. 20. The system of claim 19, wherein the EM data is acquired while navigating the tip of the endoscope forward or backward along the one or more airways.

21. The system of claim 17 , wherein the EM data-based path is generated by applying binning filtering and / or temporal filtering to the EM data.

22. 22. The system of claim 21, wherein the EM data-based path comprises a sequence of points evenly distributed in both the spatial and temporal domains and represents a drive trajectory of the tip of the endoscope.

23. 18. The system of claim 17, wherein the body part is a lung and the endoscope is a bronchoscope.

24. The system of claim 17 , wherein deformation parameters indicate deformation of the body part due to movement of the patient.

25. 20. The system of claim 17, wherein the biological model of the body part of the patient is generated based on a computed tomography (CT) scan.

26. The system of claim 17, wherein the deformation parameter is indicative of deformation due to CT-body dissociation.

27. 20. The system of claim 17, wherein the one or more path hypotheses are within a predetermined position range from a most recent data point in the sequence of EM data, within a measurement error or deformation threshold, or above a likelihood threshold.

28. 20. The system of claim 17, wherein each of the one or more path hypotheses includes at least a portion of a centerline of the airway of the living model.

29. 20. The system of claim 17, wherein the computer-executable instructions further comprise dynamically adding or removing path hypotheses while the tip of the endoscope is being driven through the body part.

30. 20. The system of claim 17, wherein mapping the one or more path hypotheses to the EM data-based path comprises applying the optimization algorithm to determine deformation and shape matches between the one or more path hypotheses and the EM data-based path.

31. The system of claim 17 , wherein the variant path selected from the one or more variant path paths is associated with the highest probability.

32. 32. The system of claim 31, wherein the highest probability is determined based at least in part on normalized probabilities associated with each of the one or more route hypotheses.

Images