Systems and methods for robotic bronchoscopy navigation

Abstract

To provide a method for auto-registration for a robotic endoscopic apparatus.SOLUTION: The method comprises: (a) generating a first transformation between an orientation of the robotic endoscopic apparatus and an orientation of a location sensor based at least in part on a first set of sensor data collected using the location sensor; (b) generating a second transformation between a coordinate frame of the robotic endoscopic apparatus and a coordinate frame of a model representing an anatomical luminal network based at least in part on the first transformation and a second set of sensor data; and (c) updating, based at least in part on a third set of sensor data, the second transformation using an updating algorithm.SELECTED DRAWING: Figure 7

Description

Technical Field

[0001] Reference

[0001] This application claims the benefit of U.S. Provisional Patent Application No. 62 / 950,740, filed Dec. 19, 2019, and U.S. Provisional Patent Application No. 63 / 091,283, filed Oct. 13, 2020, each of which is hereby incorporated by reference in its entirety.

Background Art

[0002] Background of the Invention

[0002] Early diagnosis of lung cancer is extremely important. The 5-year survival rate of lung cancer is about 18%, which is significantly lower than the next three most prevalent cancers: breast cancer (90%), colorectal cancer (65%), and prostate cancer (99%). In 2018, a total of 142,000 people were recorded as having died from lung cancer.

[0003]

[0003] Generally, the diagnostic and surgical treatment processes for typical lung cancer can vary dramatically depending on the techniques, clinical protocols, and clinical settings used by healthcare providers. An inconsistent process may delay cancer diagnosis and impose high costs on patients and the healthcare system.

[0004]

[0004] These medical procedures, such as endoscopy (e.g., bronchoscopy), may involve accessing and visualizing the interior of a patient's lumen (e.g., airway) for diagnostic and / or therapeutic purposes. During the procedure, a flexible tubular instrument, such as an endoscope, may be inserted into the patient's body for diagnosis and / or treatment, and the device can be passed through the endoscope to reach the tissue site identified.

Summary of the Invention

[0005] Summary of the Invention

[0005] Endoscopes have extensive applications in the diagnosis and treatment of various conditions such as physical diseases (e.g., early lung cancer diagnosis and treatment). An endoscopic navigation system can use various detection modalities (e.g., camera imaging data, electromagnetic (EM) position data, robotic position data, etc.) modeled, for example, through adaptation adjustment probabilities. Navigation techniques may rely on an initial estimate of where the tip of the endoscope is relative to the airway to initiate tracking of the tip of the endoscope. Some endoscopic techniques may include three-dimensional (3D) modeling of the patient's biological structure and guided navigation using EM fields and position sensors. Prior to the procedure, the precise alignment (e.g., registration) between the virtual space of the 3D model, the physical space of the patient's biological structure represented by the 3D model, and the EM field may be unknown. Therefore, prior to the generation of the registration, the position of the endoscope within the patient's biological structure cannot be precisely mapped to the corresponding location within the 3D model.

[0006] [

[0006] ] Bronchoscopic navigation can be difficult due to inaccurate registration. In particular, current registration procedures are time-consuming and cumbersome, or may produce inconsistent results affected by human input. Current registration procedures may include generating an estimated initial registration and then refining or updating the initial registration during the procedure. For example, current registration methods may compensate for changes to the navigation system by sampling from real-time sensor data. However, as the device is driven during long-term operation and more input data is collected by the system, the computation time becomes dramatically longer. Furthermore, current registration procedures do not have a satisfactory online update ability, which may lead to a loss of registration accuracy. For example, current registration procedures may not be able to adapt to local changes, which may lead to an inaccurate registration algorithm. For example, a sampled dataset that does not accurately reflect local changes (e.g., due to local minor mechanical deformations) is typically used to perform a global update on the registration algorithm (e.g., transformation). In another example, current registration algorithms may have poor transformation accuracy if the endoscope does not move along a predetermined path (e.g., resection of the centerline of the airway, etc.).

Means for Solving the Problem

[0007]

[0007] It is recognized herein that there is a need for a minimally invasive system that can improve reliability and cost - effectiveness to perform surgical procedures or diagnostic operations. Another need recognized herein is for an improved registration algorithm that can increase registration accuracy while shortening registration time. The present disclosure provides systems and methods that enable low - cost, standardized diagnosis and treatment of early - stage lung cancer. The present disclosure provides affordable and more cost - effective methods and systems for early - stage cancer diagnosis and treatment. In some embodiments of the present invention, at least a portion of the robotic bronchoscopy is disposable. For example, the catheter portion can be designed to be low - cost and disposable while maintaining surgical performance and functionality. Further, the provided robotic bronchoscopy system is designed to have the ability to access tissues that are difficult to reach, such as bronchi, lungs, etc., without introducing additional costs.

[0008]

[0008] An adaptive navigation algorithm may be capable of identifying on - the - fly updated registration or mapping between the coordinate system of a 3D model (e.g., the coordinate system of the CT scanner used to generate the model) and the coordinate system of an EM field (e.g., an EM field generator).

[0009]

[0009] In one aspect, a method of navigating a robotic endoscope device is provided. The method includes (a) generating a first transformation between the orientation of the robotic endoscope device and the orientation of a location sensor, at least partially based on a first set of sensor data collected using the location sensor; (b) generating a second transformation between the coordinate system of the robotic endoscope device and the coordinate system of a model representing an anatomical lumen network, at least partially based on the first transformation and a second set of sensor data; and (c) updating the second transformation using an update algorithm, at least partially based on a third set of sensor data.

[0010]

[0010] In some embodiments, the update algorithm includes a high-speed interval recalculation operation and a low-speed interval recalculation operation. In some cases, the high-speed interval recalculation operation includes (i) calculating the relevance of the first set using a subset of the data sampled from the third set of sensor data, and (ii) combining the relevance of the first set with the relevance of the second set to generate a second transformation in (b). For example, the method further includes calculating a point cloud using the combined relevance of the first and second sets. In some cases, the low-speed interval recalculation operation includes a nearest neighbor algorithm. In some cases, the low-speed interval recalculation operation includes updating the second transformation using only the third set of sensor data.

[0011]

[0011] In some embodiments, the location sensor is an electromagnetic sensor. In some embodiments, the coordinate system of the model representing the anatomical lumen network is generated using a preoperative imaging system.

[0012]

[0012] In some embodiments, the robotic endoscope device includes a disposable catheter assembly. In some embodiments, the location sensor is disposed at the distal end of the robotic endoscope device.

[0013]

[0013] In another aspect, a system for navigating a robotic endoscope device is provided. The system includes a location sensor disposed at the distal end of the robotic endoscope device and one or more processors in communication with the location sensor and the robotic endoscope device, the one or more processors configured to execute a set of instructions to cause the system to (a) generate a first transformation between the orientation of the robotic endoscope device and the orientation of the location sensor based at least in part on a first set of sensor data collected using the location sensor; (b) generate a second transformation between the coordinate system of the robotic endoscope device and the coordinate system of a model representing an anatomical lumen network based at least in part on the first transformation and a second set of sensor data; and (c) update the second transformation using an update algorithm based at least in part on a third set of sensor data.

[0014]

[0014] In some embodiments, the update algorithm includes a high-speed interval recalculation operation and a low-speed interval recalculation operation. In some cases, the high-speed interval recalculation operation includes (i) calculating the relevance of the first set using a subset of the data sampled from the third set of sensor data, and (ii) combining the relevance of the first set with the relevance of the second set to generate a second transformation in (b). For example, the high-speed interval recalculation operation further includes calculating a point cloud using the combined relevance of the first and second sets. In some cases, the low-speed interval recalculation operation includes a nearest neighbor algorithm. In some cases, the low-speed interval recalculation operation includes updating the second transformation using only the third set of sensor data.

[0015]

[0015] In some embodiments, the location sensor is an electromagnetic sensor. In some embodiments, the coordinate system of the model representing the anatomical lumen network is generated using a preoperative imaging system. In some embodiments, the robotic endoscope device includes a disposable catheter assembly.

[0016]

[0016] According to some aspects of the present disclosure, a robotic endoscope device is provided. The device may include a disposable elongate member having a proximal end and a distal end, the proximal end being detachably attached to a robotic arm. The distal end includes a plurality of pull wires, and the pull wires are integrated with the wall of the elongate member. The elongate member may also be referred to as a bronchoscope, a catheter, etc., and these can be used synonymously throughout this specification.

[0017]

[0017] In another aspect of the present disclosure, an improved registration algorithm or navigation method is provided. The registration algorithm may enable automatic registration and online registration updates with minimal user interaction, which advantageously improves registration accuracy. Note that the provided robotic system and / or registration algorithm can be used in various minimally invasive surgical procedures involving various types of tissues, including heart, bladder, and lung tissues.

[0018]

[0018] Additional aspects and advantages of the present disclosure will become readily apparent to those skilled in the art from the following detailed description, which illustrates and describes only exemplary embodiments of the present disclosure. As will be recognized, the present disclosure is capable of other and different embodiments, and some of the details thereof are capable of modification in various obvious respects, all without departing from the present disclosure. Accordingly, the drawings and description are to be regarded as illustrative in nature and not as restrictive.

[0019] Incorporation by reference

[0019] All publications, patents, and patent applications mentioned in this specification are hereby incorporated by reference as if each individual publication, patent, or patent application was specifically and individually indicated to be incorporated by reference. To the extent that the incorporated publications and patents or patent applications conflict with the disclosure contained herein, this specification takes precedence over any such conflicting materials and / or is intended to supersede them.

[0020] Brief description of the drawings

[0020] The novel features of the present invention are particularly set forth in the appended claims. A better understanding of the features and advantages of the present invention will be obtained by reference to the following detailed description, which describes exemplary embodiments in which the principles of the present invention are utilized, and to the accompanying drawings (also referred to herein as "Figure" and "FIG.").

Brief description of the drawings

[0021]

Figure 1

[0021] An example of a standardized lung cancer diagnosis workflow enabled by the robotic bronchoscope system described herein is shown.

Figure 2A

[0022] An example of a robotic bronchoscope system according to some embodiments of the present invention is shown.

Figure 2B

[0023] Different views of an example of a robotic bronchoscope system according to some embodiments of the present invention are shown.

Figure 3

[0024] An example of a fluoroscopic (tomosynthesis) imaging system is shown.

Figure 4

[0025] A C-arm fluoroscopic (tomosynthesis) imaging system in different (rotated) poses while imaging a subject is shown.

Figure 5

[0026] An example of a user interface for visualizing a virtual airway in which an optimal path, the location of the tip of a catheter, and the location of a lesion are overlaid is shown.

Figure 6

[0027] An example of a portable robotic cone beam CT is shown.

Figure 7

[0028] An example of an automatic registration method according to some embodiments of the present invention is schematically shown.

Figure 8

[0029] An example of a customized iterative closest point (ICP) algorithm is shown.

Figure 9

[0030] An example of classification results is shown.

Figure 10

[0031] An example of a data structure (spatial data structure) is shown in which the dedicated data structure can be a k-d tree or an octree data structure.

Best Mode for Carrying Out the Invention

[0022] Detailed Description of the Invention

[0032] Various embodiments of the present invention have been illustrated and described in this specification, but it will be apparent to those skilled in the art that such embodiments are provided only as examples. Those skilled in the art can conceive of many variations, modifications, and substitutions without departing from the present invention. It should be understood that various changes to the embodiments of the present invention described in this specification can be adopted.

[0023]

[0033] The exemplary embodiments mainly relate to bronchoscopes, but this is not intended to be limiting, and those skilled in the art will understand that the devices described herein can be used in other treatment or diagnostic and non-limiting, digestive systems including the esophagus, liver, stomach, colon, urinary tract, or non-limiting, other biological structure regions of the patient's body such as the respiratory system including the bronchi, lungs, etc. The registration methods / algorithms herein can be used to align the coordinate system of a medical device with the coordinate system of a 3D model (e.g., an airway model or the patient's coordinate system generated by a CT scanner), regardless of the type of device or the operation being performed.

[0024]

[0034] The embodiments disclosed herein can be combined in one or more of many ways to provide improved diagnosis and treatment to patients. The disclosed embodiments can be combined with existing methods and devices, such as for example, lung diagnosis, surgery, and combinations with known methods for other tissues and organs, to provide improved treatment. Any one or more of the structures and steps described herein can be combined with one or more additional structures and steps of the methods and devices described herein, and it should be understood that the drawings and support text provide an explanation according to the embodiments.

[0025]

[0035] The treatment plans and definitions of the diagnosis or surgical procedures described herein are presented with respect to lung diagnosis or surgery, but the methods and devices described herein can be used for the treatment of any tissue of the body as well as any organ and vasculature of the body such as the brain, heart, lungs, intestines, eyes, skin, kidneys, liver, pancreas, stomach, uterus, ovaries, testicles, bladder, ears, nose, mouth, bone marrow, adipose tissue, muscle, glandular tissue, and mucosal tissue, soft tissues such as spinal tissue and nerve tissue, cartilage, hard biological tissues such as teeth and bones, and body cavities and body passages such as sinuses, ureters, colon, esophagus, lung passages, blood vessels, and larynx.

[0026]

[0036] When the terms "at least", "more than ~", or "or more" precede the first numerical value in a series of two or more numerical values, the terms "at least", "more than ~", or "or more" always apply to each numerical value in the series. For example, 1, 2, or 3 or more is equivalent to 1 or more, 2 or more, or 3 or more.

[0027]

[0037] When the terms "not exceeding ~", "less than", or "or less" precede the first numerical value in a series of two or more numerical values, the terms "not exceeding ~", "less than", or "or less" always apply to each numerical value in the series. For example, 3, 2, or 1 or less is equivalent to 3 or less, 2 or less, or 1 or less.

[0028]

[0038] As used herein, processor includes one or more processors, e.g., a single processor or multiple processors such as in a distributed processing system. The controller or processor described herein generally includes a tangible medium storing instructions for performing steps of a process, and the processor can include, for example, one or more of a central processing unit, programmable array logic, gate array logic, or field programmable gate array. In some cases, the one or more processors can be a programmable processor (e.g., a central processing unit (CPU) or a microcontroller), a digital signal processor (DSP), a field programmable gate array (FPGA), and / or one or more advanced RISC machines (ARM) processors. In some cases, the one or more processors can be operably coupled to a non-transitory computer-readable medium. The non-transitory computer-readable medium can store logic, code, and / or program instructions executable by the one or more processors to perform one or more steps. The non-transitory computer-readable medium can include one or more memory units (e.g., an external storage device such as a removable medium or an SD card or a random access memory (RAM)). The one or more methods or operations disclosed herein can be implemented by hardware components such as, for example, an ASIC, a dedicated computer, or a general-purpose computer, or a combination of hardware and software.

[0029]

[0039] As used herein, the terms proximal and distal generally refer to locations referenced from a device and can be the reverse of a biological structural reference. For example, the distal location of a bronchoscope or catheter can correspond to the proximal location of a patient's elongated limb, and the proximal location of a bronchoscope or catheter can correspond to the distal location of a patient's elongated limb.

[0030]

[0040] The system described in this specification includes an elongate portion or elongate member, such as a catheter. The terms "elongate member", "catheter", and "bronchoscope" are used synonymously throughout this specification, unless the context otherwise indicates. The elongate member can be disposed directly within a lumen or body cavity. In some embodiments, the system can further include a support device, such as a robotic manipulator (e.g., a robotic arm), to drive, support, position, or control the movement and / or operation of the elongate member. Alternatively or additionally, the support device can be a handheld device or other control device that may or may not include a robotic system. In some embodiments, the system can further include peripheral devices and subsystems, such as an imaging system, to assist and / or facilitate the navigation of the elongate member to a target site within the subject's body. Such navigation may require a registration process described later in this specification.

[0031]

[0041] In some embodiments of the present disclosure, a robotic bronchoscope system is provided for performing surgical procedures or diagnoses with improved performance and at low cost. For example, the robotic bronchoscope system can include an operable catheter that can be disposable overall. This advantageously can reduce the requirements for sterilization, which can be costly or difficult to perform and yet may not be effective in sterilizing or sanitizing. Further, one problem with bronchoscopes is reaching the upper lobes of the lungs while navigating through the airways. In some cases, the provided robotic bronchoscope system can be designed to have the ability to navigate autonomously or semi-autonomously through airways having small bends. Autonomous or semi-autonomous navigation may require a registration process described later in this specification. Alternatively, the robotic bronchoscope system can be navigated by an operator through a control system having visual guidance.

[0032]

[0042] The typical diagnosis and surgical treatment processes of lung cancer can vary dramatically depending on the techniques, clinical protocols, and clinical settings used by healthcare providers. Inconsistent processes can delay the diagnosis of early-stage lung cancer, increase the costs for the healthcare system and patients involved in the diagnosis and treatment of lung cancer, and increase the risk of clinical and procedural complexity. The provided robotic bronchoscope system can enable standardized early lung cancer diagnosis and treatment. FIG. 1 shows a workflow 100 of an example of standardized lung cancer diagnosis enabled by the robotic bronchoscope system described herein.

[0033]

[0043] As shown in FIG. 1, preoperative imaging can be performed to identify lesions. Any suitable imaging modality such as magnetic resonance (MR), positron emission tomography (PET), X-ray, computed tomography (CT), and ultrasound can be used to identify lesions or target areas. For example, a patient suspected of having lung cancer can be given a preoperative CT scan, and a suspected pulmonary nodule can be identified in the CT image. The preoperative imaging process can be performed prior to bronchoscopy.

[0034]

[0044] Next, the CT image can be analyzed to generate a map to guide the navigation of the robotic bronchoscope during bronchoscopy. For example, a lesion or target area (ROI) can be segmented in the image. When the lung is being imaged, the passage or path to the lesion can be highlighted in the reconstructed image to calculate the navigation path. The reconstructed image can guide the navigation of the robotic bronchoscope to the target tissue or target site. In some cases, the navigation path can be pre-planned using 3D image data. For example, a catheter can be advanced towards the target site under the robotic control of the robotic bronchoscope system. The catheter can be maneuvered or advanced towards the target site manually, autonomously, or semi-autonomously. In one example, the movement of the catheter can be image-guided so that the insertion direction and / or the maneuvering direction can be automatically controlled.

[0035]

[0045] In some cases, the location of the lesion in the preoperative imaging may not be accurate due to patient movement or body differences. In such cases, the location of the lesion can be confirmed prior to a surgical procedure (e.g., biopsy or treatment). The exact location of the lesion can be confirmed or updated using a robotic bronchoscope system. For example, the bronchoscope system can provide an interface to an imaging modality such as fluoroscopy that provides in vivo real-time imaging of the target site and surrounding area to localize the lesion. In one example, a C-arm or O-arm fluoroscopy imaging system can be used to generate tomosynthesis images to confirm or update the location of the lesion. Prior to a surgical procedure such as a biopsy, various surgical instruments such as a biopsy instrument, brush, or forceps can be inserted into the working channel of the catheter to perform the biopsy or other surgical procedure manually or automatically.

[0036]

[0046] Next, a sample of the lesion or any other target tissue can be obtained by an instrument inserted through the working channel of the catheter. The system is configured to maintain camera visualization throughout the procedure, including during insertion of the instrument through the working channel. In some cases, the tissue sample can be rapidly evaluated on-site by rapid on-site evaluation to determine whether repeated tissue sampling is necessary or to make a decision on further action. In some cases, the rapid on-site evaluation process can also provide rapid analysis of the tissue sample to determine subsequent surgical treatment. For example, if the tissue sample is determined to be malignant as a result of the rapid on-site evaluation process, a manual or robotic treatment device can be inserted through the working channel of the robotic bronchoscope to perform endobronchial treatment of lung cancer. Advantageously, this allows for the performance of diagnosis and treatment in one session, thereby providing a painless and rapid treatment targeted at early-stage lung cancer.

[0037]

[0047] Figures 2A and 2B show, by way of example, robotic bronchoscope systems 200, 230 according to some embodiments of the present invention. As shown in Figure 2A, the robotic bronchoscope system 200 may include a steerable catheter assembly 220 and a robotic support system 210 for supporting or carrying the steerable catheter assembly. The steerable catheter assembly can be a bronchoscope. In some embodiments, the steerable catheter assembly can be a single-use robotic bronchoscope. In some embodiments, the robotic bronchoscope system 200 may include an instrument drive mechanism 213 attached to the arm of the robotic support system. The instrument drive mechanism may be provided by any suitable controller device (e.g., a handheld controller) that may or may not include the robotic system. The instrument drive mechanism can provide a mechanical and electrical interface to the steerable catheter assembly 220. The mechanical interface can releasably couple the steerable catheter assembly 220 to the instrument drive mechanism. For example, the handle portion of the steerable catheter assembly can be attached to the instrument drive mechanism via high-speed installation / release means such as magnets, spring-loaded levels, etc. In some cases, the steerable catheter assembly can be manually coupled to or released from the instrument drive mechanism without using tools.

[0038]

[0048] The steerable catheter assembly 220 may include a handle portion 223, and the handle portion 223 may include components configured to process image data, provide power, or establish communication with other external devices. For example, the handle portion 223 may include circuitry and communication elements that enable electrical communication between the steerable catheter assembly 220 and the device drive mechanism 213 and any other external system or device. In another example, the handle portion 223 may include circuit elements such as a power source for powering the electronic circuitry (e.g., camera and LED light) of the endoscope. In some cases, the handle portion may communicate electrically with the device drive mechanism 213 via an electrical interface (e.g., a printed circuit board), whereby image / video data and / or sensor data can be received by the communication module of the device drive mechanism and transmitted to other external devices / systems. Alternatively or additionally, the device drive mechanism 213 may provide only a mechanical interface. The handle portion may communicate electrically with a modular wireless communication device or any other user device (e.g., a portable / handheld device or a controller) to transmit sensor data and / or receive control signals. Details regarding the handle portion are described later herein.

[0039]

[0049] The steerable catheter assembly 220 may include a flexible elongate member 211 coupled to the handle portion. In some embodiments, the flexible elongate member may include a shaft, a steerable distal end, and a steerable section. The steerable catheter assembly may be a single-use robotic bronchoscope. In some cases, only the elongate member may be disposable. In some cases, at least a portion of the elongate member (e.g., the shaft, the steerable distal end, etc.) may be disposable. In some cases, the entire steerable catheter assembly 220, including the handle portion and the elongate member, may be disposable. The flexible elongate member and the handle portion are designed such that the entire steerable catheter assembly is low-cost and disposable. Details regarding the flexible elongate member and the steerable catheter assembly are described later herein.

[0040]

[0050] In some embodiments, the provided bronchoscope system can also include a user interface. As shown in the exemplary system 230, the bronchoscope system can include a treatment interface module 231 (on the user console side) and / or a treatment control module 233 (on the patient and robot side). The treatment interface module can enable an operator or user to interact with the bronchoscope during a surgical procedure. In some embodiments, the treatment control module 233 can be a handheld controller. The treatment control module can, in some cases, include a proprietary user input device and one or more add-on elements detachably coupled to an existing user device to improve the user input experience. For example, a physical trackball or roller can replace or supplement at least one function of a virtual graphical element (e.g., a navigation arrow displayed on a touchpad) displayed in a graphical user interface (GUI) by providing the physical trackball or roller with a function similar to that of the replaced graphical element. Examples of user devices can include, without limitation, mobile devices, smartphones / cell phones, tablets, personal digital assistants (PDAs), laptop or notebook computers, desktop computers, media content players, and the like. Details regarding the user interface device and user console are described later in this specification.

[0041]

[0051] FIG. 2B shows a different view of the bronchoscope system. The user console 231 can be mounted on the robot support system 210. Alternatively or additionally, the user console or a portion of the user console (e.g., the treatment interface module) can be mounted on a separate mobile cart.

[0042] Robot Intraluminal Platform

[0052] In one aspect, a robotic endoluminal platform is provided. In some cases, the robotic endoluminal platform can be a bronchoscope platform. The platform can be configured to perform one or more operations consistent with the methods described in FIG. 1. FIGS. 3-7 show various examples of robotic endoluminal platforms and their components or subsystems according to some embodiments of the present invention. In some embodiments, the platform can include a robotic bronchoscope system and one or more subsystems that can be used in combination with the robotic bronchoscope system of the present disclosure.

[0043]

[0053] In some embodiments, one or more subsystems may include an imaging system such as a fluoroscopy (tomosynthesis) imaging system that provides real-time imaging of a target site (e.g., including a lesion). FIG. 3 shows an example of a fluoroscopy (tomosynthesis) imaging system 300. For example, the fluoroscopy (tomosynthesis) imaging system may perform accurate lesion location tracking or confirmation before or during the surgical procedure described in FIG. 1. In some cases, the lesion location can be tracked based on location data for the fluoroscopy (tomosynthesis) imaging system / station (e.g., a C-arm) and image data captured by the fluoroscopy (tomosynthesis) imaging system. The lesion location can be registered with the coordinate system of the robotic bronchoscope system. The location or movement of the fluoroscopy (tomosynthesis) imaging system can be measured using any suitable motion / location sensor 310 such as an inertial measurement unit (IMU), one or more gyroscopes, speed sensors, accelerometers, magnetometers, location sensors (e.g., a global positioning system (GPS) sensor), vision sensors (e.g., an imaging device capable of detecting visible light, infrared light, or ultraviolet light such as a camera), proximity or ranging sensors (e.g., ultrasonic sensors, lidar, time-of-flight or depth cameras), altitude sensors, attitude sensors (e.g., compasses), and / or field sensors (e.g., magnetometers, electromagnetic sensors, wireless sensors). One or more sensors for tracking the movement and location of the fluoroscopy (tomosynthesis) imaging station may be disposed on the imaging station or remotely disposed from the imaging station such as a wall-mounted camera 320. FIG. 4 shows a C-arm fluoroscopy (tomosynthesis) imaging system in different (rotated) postures during imaging of a subject. Various postures can be captured by one or more sensors as described above.

[0044]

[0054] In some embodiments, the location of the lesion can be segmented in the image data captured by a fluoroscopy (tomosynthesis) imaging system using the signal processing unit 330. One or more processors of the signal processing unit can be configured to further overlay the treatment location (e.g., the lesion) on the real-time fluoroscopy image / video. For example, the processing unit can be configured to generate an augmentation layer that includes augmentation information such as the location of the treatment location or the target site. In some cases, the augmentation layer can also include graphical markers indicating the path to this target site. The augmentation layer can be a substantially transparent image layer that includes one or more graphical elements (e.g., boxes, arrows, etc.). The augmentation layer can be overlaid on the optical view of the optical image or video stream captured by the fluoroscopy (tomosynthesis) imaging system and / or displayed on the display device. Due to the transparency of the augmentation layer, the optical image can be viewed by the user with the graphical elements overlaid thereon. In some cases, both the segmented lesion image and the optimal path for navigating the elongate member to reach the lesion can be overlaid on the real-time tomosynthesis image. Thereby, the operator or user can visualize the exact location of the lesion and the planned path of the bronchoscope movement. In some cases, the segmented and reconstructed image (e.g., the CT image described elsewhere) provided prior to the operation of the system described herein can be overlaid on the real-time image.

[0045]

[0055] In some embodiments, one or more subsystems of the platform can include a navigation and localization subsystem. The navigation and localization subsystem can be configured to construct a virtual airway model based on preoperative images (e.g., preoperative CT images or tomosynthesis). The navigation and localization subsystem can be configured to identify the segmented lesion location in the 3D-rendered airway model, and based on the location of the lesion, the navigation and localization subsystem can generate the optimal path from the main bronchus to the lesion along with the recommended approach angle to the lesion for performing a surgical procedure (e.g., a biopsy).

[0046]

[0056] In the registration step before driving the bronchoscope to the target site, the system can align the rendered virtual view of the airway to the patient's airway. Image registration can consist of a single registration step or a combination of a single registration step and real-time sensory updates to the registration information. The registration process can include finding a transformation that aligns an object (e.g., an airway model, a biological structure site) between different coordinate systems (e.g., EM sensor coordinates and the patient's 3D model coordinates based on preoperative CT imaging). Details about registration are described later in this specification.

[0047]

[0057] Once registered, all airways can be aligned to the preoperative rendered airways. While the robotic bronchoscope is being driven towards the target site, the location of the bronchoscope inside the airway can be tracked and displayed. In some cases, the location of the bronchoscope relative to the airway can be tracked using a position sensor. Sensor fusion techniques can be used to use other types of sensors (e.g., a camera) instead of or in combination with the position sensor. A position sensor such as an electromagnetic (EM) sensor can be incorporated at the tip of the catheter, and the EM field generator can be positioned next to the patient's torso during the procedure. The EM field generator can identify the EM sensor position in 3D space or the EM sensor position and orientation in 5D or 6D space. This can provide the operator with a visual guide as to when to drive the bronchoscope towards the target site.

[0048]

[0058] In real-time EM tracking, an EM sensor composed of one or more sensor coils incorporated at one or more locations and orientations (e.g., the tip of an endoscopic instrument) of a medical device measures fluctuations in an EM field generated by one or more static EM field generators positioned close to the patient. The location information detected by the EM sensor is stored as EM data. The EM field generator (or transmitter) can be placed near the patient to generate a low-intensity magnetic field that can be detected by the incorporated sensors. The magnetic field induces a small current in the sensor coils of the EM sensor, which can be analyzed to identify the distance and angle between the EM sensor and the EM field generator. These distances and orientations can be registered during the procedure to align a single location in the coordinate system with the position in the preoperative model of the patient's anatomy (e.g., a 3D model) to identify a registration transformation.

[0049]

[0059] Figure 5 shows an example of a user interface for visualizing the optimal path 503, the location of the tip of the catheter 501, and the virtual airway 509 overlaid with the lesion location 505. In this example, the location of the tip of the catheter is displayed in real-time relative to the virtual airway model 509, thereby providing visual guidance. As shown in the example of Figure 5, during robotic bronchoscope drive, the optimal path 503 can be displayed and overlaid on the virtual airway model. As described above, the virtual airway model can be constructed based on real-time fluoroscopic images / videos (and location data of the imaging system). In some cases, views of the real-time fluoroscopic image / video 507 can also be displayed in the graphical user interface. In some cases, the user may also be permitted access to the camera view or image / video 511 captured in real-time by the bronchoscope.

[0050]

[0060] In some embodiments, one or more subsystems of the platform can include one or more treatment subsystems such as manual or robotic devices (e.g., biopsy needles, biopsy forceps, biopsy brushes) and / or manual or robotic treatment devices (e.g., RF ablation devices, cryogenic devices, microwave devices, etc.).

[0051]

[0061] Conventional cone beam CT machines can have the emitter and receiver panels on the same mechanical structure having a C-shape or an O-shape. The connection between the emitter and the receiver panel can increase the size of the cone beam CT. This overly large design imposes limitations on the use cases and takes up a large space in a rather narrow operating room.

[0052]

[0062] Described herein is a design that disconnects the mechanical connection between the emitter and the receiver panel. FIG. 6 shows an example of a portable robotic cone beam CT. The emitter and the receiver panel can be separately mounted on two separate robotic arms as shown in the example of FIG. 6. During use, the two robots can move in the same coordinate system. A control algorithm can ensure that the two robots move in a synchronized manner.

[0053]

[0063] In addition, for patient gating motion, i.e., breathing, an additional external sensor - i.e., an IMU, EM, or image sensor - can be added to track the patient's motion. The change in the patient's position can be tracked using sensors such as an IMU, EM, or image sensor. Sensory signals can be used to command the two robotic arms. In some cases, one or both of the robotic arms can move to track the patient's motion, which basically keeps the emitter and the receiver stationary with respect to the patient's motion with respect to the region of interest (ROI) during tracking. The ROI can include a target site or a target location that can be determined automatically by the system or manually by a physician. Tracking can also be performed using other mechanisms such as an external camera and one or more trackers on the patient's body, without limitation.

[0054]

[0064] It should be understood by those skilled in the art that cone beam CT is a non-limiting example. The design described herein can also be used in other imaging modalities such as fluoroscopes, classical CT machines, and MRI machines.

[0055] Automatic Registration Algorithm

[0065] The present disclosure provides an improved automatic registration algorithm that can increase registration accuracy while shortening the registration time. The provided automatic registration algorithm can advantageously allow for minimal user interaction and establish improved transformation or registration from existing methods. Further, the provided automatic registration algorithm can allow for on-the-fly updating of the algorithm (e.g., automatic updating of registration based on registration errors detected in real time) to adapt to real-time situations and increase registration accuracy when the device is being driven inside the subject's body.

[0056]

[0066] In some embodiments, the automatic registration algorithm can include, or can be implemented in, three stages. FIG. 7 schematically shows an example of an automatic registration method 700 according to some embodiments of the present invention. The automatic registration method can include one or more algorithms that update registration information in real time without user input. The automatic registration algorithm can also include one or more algorithms that identify an optimal registration / transformation matrix and utilize time information to increase registration accuracy.

[0057]

[0067] As described above, to track a location within an EM field, an EM sensor can be coupled to the distal end of an endoscope. The EM field is stationary relative to the EM field generator, and the coordinate system of the 3D model of the luminal network (e.g., CT space) can be mapped to the coordinate system of the EM field. As shown in FIG. 7, the automatic registration method 700 can include a first stage that can be an initial data collection phase 701. The initial data collection phase can include reconstructing the distal end of the catheter and performing an initial translational transformation.

[0058]

[0068] The initial data collection phase 701 can register the orientation of the EM sensor to the orientation of the tip. This can establish an association between the robot endoscope device space and the location sensor space. For example, an electromagnetic coil disposed at the tip can be used in combination with an electromagnetic tracking system to detect the position and orientation of the tip of the catheter while it is disposed within a biological structure system (e.g., an anatomical lumen network). In some embodiments, the coil can be tilted to provide sensitivity to electromagnetic fields along different axes, giving the disclosed navigation system the ability to measure up to six degrees of freedom: three degrees of freedom of position and three degrees of freedom of angle. The orientation / location of the EM field generator can be unknown, and the initial data collection phase can register the EM sensor orientation (e.g., the Z-axis of the EM coordinate system) to the tip orientation (e.g., the catheter tip Z-axis) using the collected sensor data.

[0059]

[0069] During the first phase, sensor data (e.g., EM sensor data) can be collected to register the z-axis of the magnetic field generator with the z-axis of the catheter tip. This can enable flexibility in placing the magnetic field generator without prior knowledge of location and orientation. The sensor data can include stream data collected when the catheter is driven in a known direction. The stream data can include various types of data, including but not limited to time-series data such as spatio-temporal point measurements generated by an EM sensor. In some cases, the time-series data can be collected when the catheter is driven in a known direction, which may or may not be inside the patient's body. For example, EM sensor data points can be collected when the tip is placed inside a subject, such as in a bronchus or trachea, and moved along a known direction. In some cases, the z-axis of the catheter tip and the EM orientation can be calibrated during assembly or manufacturing. In some cases, the z-axis of the catheter tip can be obtained from robot data when driving the catheter tip in a known direction, or can be obtained using CT imaging or other real-time imaging techniques. An initial translational transformation between the catheter tip and the EM field can be obtained. Such translational and rotational transformations can be continuously improved and changed during the first phase as more input sensor data is collected. In some cases, the translational simultaneous transformation in the first phase can be updated and improved until a threshold is met. The threshold can be an accuracy threshold indicating that the accuracy has been met. Alternatively or additionally, the update can be performed continuously (e.g., the error can be calculated at each time interval), and the update can be stopped when tomosynthesis targeting progresses.

[0060]

[0070] In some cases, the collected sensor data can also provide registration regarding the translational transformation between the CT space (e.g., tissue) and the EM space (e.g., catheter). For example, a registration transformation between the EM field and a patient model (e.g., the coordinate system of a lumen network model) can be identified. This translational transformation in the second phase can include an initial translational transformation matrix, which can be later improved in the final phase described later herein.

[0061]

[0071] The second phase 703 of the registration method may include creating an association between the EM space and the model (e.g., CT) space. During the second phase, the initial transformation matrix between the EM space and the CT space may be generated based at least in part on the translational information and real-time sensor data generated in the first phase. In some cases, the data points (e.g., EM data) utilized in the second phase may be collected as the catheter navigates inside the trachea and / or its sub-branches along a predefined navigation path. The predefined navigation path may be defined in the CT space. As described elsewhere herein, the EM data may include information about orientation, position, and error data. The coordinate system of the CT space or the biological structure model may be generated during a preoperative procedure or a surgical operation. Using such sensor data points, a plane in the space is established, and a set of associations between the EM space and the CT space may be established based on a set of direction distances (e.g., both orientation and length) moved in the EM space and the CT space respectively. The set of associations may be generated based at least in part on time-series data such as spatio-temporal data points generated by the EM sensors.

[0062]

[0072] In some cases, a set of associations may be processed to identify an optimal transformation matrix. For example, the best fit of an EM curve (e.g., a point cloud in the EM space) to an established CT curve (e.g., a point cloud in the CT space) in 3D space may be determined to identify the optimal transformation matrix. For example, the provided automatic registration method may employ a modified version of the iterative closest point (ICP) algorithm along with nearest neighbor and singular value decomposition to iterate the calculation towards an optimal solution. It should be noted that other algorithms or variations suitable for identifying the optimal fit may also be employed.

[0063]

[0073] Figure 8 shows an example of a customized Iterative Closest Point (ICP) algorithm for generating an initial transformation between two data point clouds (e.g., an EM domain point cloud and a CT domain point cloud). ICP is an iterative algorithm that alternately determines the nearest neighbor (NN) of every point of one shape based on the current transformation estimate and updates the estimate based on the NN. The correlation solver receives the calculated set of relationships between two sets of data point clouds (e.g., Cn, Xn) in the CT space and the EM space and can be executed to obtain a transformation that best matches a set of relationships. As an example, a set of relationships can be calculated using the k-nearest neighbor (k-NN) algorithm. The best-fit transformation can be calculated using singular value decomposition during the error minimization step. Assumptions (e.g., proximity assumptions) can be modified to soft-lock the transformation translation and rotation to good initial values.

[0064]

[0074] An example of the classification result is shown in Figure 9. The provided method can further improve the k-NN algorithm performance and reduce the calculation by utilizing a dedicated data structure. Figure 10 shows an example of a data structure (spatial data structure). The dedicated data structure can be a k-d tree or an octree data structure. The k-d tree or octree data structure can enable an efficient implementation of NNS due to a regular partitioning of the search space and a high branching factor, and the coordinate query is fast. Other data structures suitable for shape registration and memory efficiency can also be utilized.

[0065]

[0075] Referring back to Figure 8, the error minimization operation can include calculating the best-fit transformation using singular value decomposition. For example, singular value decomposition can be applied to a set of relationships to calculate the best transformation. The singular value decomposition method can be used to calculate the solution to the total least squares minimization problem.

[0066]

[0076] Note that the above algorithm can be changed or replaced by other algorithms. For example, the coherent point drift algorithm can be adopted to identify the best fit transformation based on probability rather than a fixed point location. The algorithm can scale up the point cloud (and / or deform the point cloud using probabilistic convergence) while rotating the calculated features, usually the centroids, to traverse the probabilistic optimization surface until it aligns with other point clouds. The coherent point drift algorithm or other probability-based algorithms can reduce the influence of noise on the dataset.

[0067]

[0077] Referring back to FIG. 7, the third phase 705 of the automatic registration algorithm may include updating the initial registration / transformation generated in the second phase. In some embodiments, the update method may include weak real-time updates and strong real-time updates to the initial transformation. Weak real-time updates can also be called high-speed interval updates, and strong real-time updates can also be called low-speed interval updates, which are used synonymously throughout this specification. The update can be performed using real-time data without user intervention. In some cases, the update can utilize real-time EM sensor data collected from a large point cloud. In some cases, a subset of data points from the large point cloud can be selected / sampled for error calculation and determination of whether an update is necessary based on the error.

[0068]

[0078] In some cases, the update method may include weak and strong real-time updates to the initial transformation. For example, the update method may include high-speed interval improvement steps and low-speed interval recalculation steps. In some cases, after each update, the error can be recalculated, and if the error is reduced, the transformation matrix can be updated.

[0069]

[0079] The high-speed interval improvement operation (e.g., high-speed interval update) may include randomly sampling a subset of data points from a large point cloud (e.g., data points collected in the third phase). Next, the current transformation and the nearest neighbor algorithm can be applied to the sampled set to create a set of associations. The set of associations can be combined with the set of associations calculated in the second phase to create a final point cloud. The final point cloud data is supplied to the modified ICP algorithm described in the second phase to generate an updated transformation, thereby improving the current transformation.

[0070]

[0080] The low-speed interval recalculation operation may include an update operation similar to that described above while using only the nearest neighbor algorithm without using the initial transformation information or the set of associations generated in the second phase. For example, the set of associations can be calculated by applying the nearest neighbor algorithm to sensor data collected in the third phase, and the transformation matrix can be updated using only the set of associations without information from the second phase. In some cases, the subset of data can be randomly sampled from the large dataset. Alternatively or additionally, the subset of data can be selected based on distance. For example, a distance-based time filter can be used to sample points evenly distributed along the time path and select such points for the update calculation. The distance used for filtering can be based on the CT scan and / or the patient's anatomy.

[0071]

[0081] Such low-speed interval recalculation operations (i.e., strong updates) can be performed at larger time / point intervals because they can result in the following very small "jumps". In some cases, the low-speed recalculation can be performed at regular intervals (e.g., time intervals, a predetermined number of data points, etc.). In some cases, the low-speed recalculation can be triggered when the update error value indicating that the minimum or minimum value has been reached begins to converge. Since the initial transformation is always improved by weak updates (i.e., high-speed interval recalculations) and iterations of the ICP algorithm, the "jump" can be guaranteed to be in the desired step or the correct direction. By combining weak updates (e.g., high-speed interval improvements) with strong updates and performing weak updates more frequently than strong updates, advantageously, the "jump" can be guaranteed to be in the correct direction because the improvement step increases the distance to an inaccurate minimum while reducing the distance to the minimum value.

[0072]

[0082] Random sampling of data points in the update phase (e.g., high-speed interval updates) can also advantageously prevent overfitting. Overfitting can occur due to fitting noisy data and cylinder data to a single line. Random sampling can provide the ability to find and improve the general solution instead of jumping from one minimum to another, thereby enabling a smooth traversal of the error surface and improving registration accuracy and stability. Random sampling of data points can also indirectly weight the initial relevance to the estimated relevance (calculated in high-speed interval recalculations). Since high-precision data is calculated in the second phase, it is desirable to weight this calculation more than the nearest neighbor calculation due to the complex structure and small gaps between adjacent airways. Random data sampling in the update step can further prevent deadlocks at the minimum, which may interrupt the entire registration process. Mainly, even if a global solution is found in iteration x, iteration x + 1 can add data and minimize the previous transformation, so reaching the minimum may be inevitable. Random sampling can advantageously smooth the iterations and error calculations.

[0073]

[0083] In some embodiments, the automatic registration algorithm can also adopt a preprocessing algorithm to preprocess the collected sensor data to improve performance. In some cases, the Ramada Douglas Peucker algorithm can be used to preprocess a large dataset of time-series data between two location points. The algorithm recursively finds points that maintain the overall dataset shape while simplifying the path. In some cases, an algorithm such as exponential smoothing can be used to remove noise from the dataset while keeping the fluctuations of the curve small. In another example, a filter such as alpha-beta filtering can be utilized to remove noise from the dataset. In some cases, a distance-based time filter can be used to select a subset of the data as described above.

[0074]

[0084] Preferred embodiments of the present invention have been illustrated and described herein, but it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Here, without departing from the present invention, those skilled in the art will come up with many variations, modifications, and substitutions. It should be understood that various changes to the embodiments of the present invention described herein can be adopted in practicing the present invention. The following claims define the scope of the present invention, and it is intended that methods and structures within the scope of these claims and their equivalents be encompassed thereby.

Claims

Claim 1 A method for navigating a robotic endoscope device, comprising: (a) generating a first transformation between the orientation of the robotic endoscope device and the orientation of the location sensor, based at least in part on a first set of sensor data collected using the location sensor; (b) generating a second transformation between the coordinate system of the robotic endoscope device and the coordinate system of a model representing an anatomical lumen network, based at least in part on the first transformation and a second set of sensor data; (c) updating the second transformation using an update algorithm, based at least in part on a third set of sensor data; The method comprising the steps above. Claim 2 The method according to claim 1, wherein the update algorithm includes a high-rate interval recalculation operation and a low-rate interval recalculation operation. Claim 3 The method according to claim 2, wherein the high-rate interval recalculation operation includes: (i) calculating a first set of relevance using a subset of data sampled from the third set of sensor data; and (ii) combining the first set of relevance with a second set of relevance to generate the second transformation in (b). Claim 4 The method according to claim 3, further comprising calculating a point cloud using the combined first and second sets of relevance. Claim 5 The method according to claim 2, wherein the low-rate interval recalculation operation includes a nearest neighbor algorithm. Claim 6 The method according to claim 2, wherein the low-rate interval recalculation operation includes updating the second transformation using only the third set of sensor data. Claim 7 The method according to claim 1, wherein the location sensor is an electromagnetic sensor. Claim 8 The method according to claim 1, wherein the coordinate system of the model representing the anatomical lumen network is generated using a preoperative imaging system. Claim 9 The method according to claim 1, wherein the robotic endoscope device comprises a disposable catheter assembly. Claim 10 The method according to claim 1, wherein the location sensor is disposed at a distal end of the robotic endoscope device. Claim 11 A system for navigating a robotic endoscope device, comprising: a location sensor disposed at a distal end of the robotic endoscope device; one or more processors in communication with the location sensor and the robotic endoscope device, wherein the one or more processors execute a set of instructions to cause the system to (a) generating a first transformation between the orientation of the robotic endoscope device and the orientation of the location sensor, based at least in part on a first set of sensor data collected using the location sensor; (b) generating a second transformation between the coordinate system of the robotic endoscope device and the coordinate system of a model representing the anatomical lumen network, based at least in part on the first transformation and a second set of sensor data; (c) updating the second transformation using an update algorithm, based at least in part on a third set of sensor data; A system configured to cause the above to be performed.

12. The system according to claim 11, wherein the update algorithm includes a high-speed interval recalculation operation and a low-speed interval recalculation operation.

13. The system according to claim 12, wherein the high-speed interval recalculation operation includes: (i) calculating a first set of relevance using a subset of data sampled from the third set of sensor data; and (ii) combining the first set of relevance with a second set of relevance to generate the second transformation in (b).

14. The system according to claim 13, wherein the high-speed interval recalculation operation further includes calculating a point cloud using the combined first and second sets of relevance.

15. The system according to claim 12, wherein the low-speed interval recalculation operation includes a nearest neighbor algorithm.

16. The system according to claim 12, wherein the low-speed interval recalculation operation includes updating the second transformation using only the third set of sensor data.

17. The system according to claim 11, wherein the location sensor is an electromagnetic sensor.

18. The system according to claim 11, wherein the coordinate system of the model representing the anatomical lumen network is generated using a preoperative imaging system.

19. The system according to claim 11, wherein the robotic endoscope device comprises a disposable catheter assembly.

Images