System and method for robotic bronchoscope navigation

Abstract

A method for automatic registration of a robotic endoscopic instrument is provided. The method includes: (a) generating a first transformation between an orientation of the robotic endoscopic instrument and an orientation of the position sensor based at least in part on a first sensor data set acquired using the position sensor; (b) generating a second transformation between a coordinate system of the robotic endoscopic instrument and a coordinate system of the model representative of the anatomical lumen network based at least in part on the first transformation and the second set of sensor data; and (c) updating the second transformation using an update algorithm based at least in part on the third sensor data set.

Description

[0001] This application is a divisional application of the Chinese patent application with the application date of December 18, 2020, application number 202080097097.6, and invention name “System and method for robotic bronchoscope navigation” (the corresponding PCT application with the application date of December 18, 2020 and application number PCT / US2020 / 066067).

[0002] References

[0003] This application claims priority to U.S. Provisional Patent Application No. 62 / 950,740 filed on December 19, 2019 and U.S. Provisional Patent Application No. 63 / 091,283 filed on October 13, 2020, the entire contents of each of which are hereby incorporated by reference herein. Background Art

[0004] Early diagnosis of lung cancer is critical. The five-year survival rate for lung cancer is approximately 18%, significantly lower than the next three most common cancers: breast cancer (90%), colorectal cancer (65%), and prostate cancer (99%). In 2018, a total of 142,000 people died from lung cancer.

[0005] In general, the typical lung cancer diagnosis and surgical treatment process can vary widely, depending on the technology used by healthcare providers, clinical protocols, and clinical locations. Inconsistent processes not only delay cancer diagnosis, but also result in high costs for patients and healthcare systems.

[0006] These medical procedures, such as endoscopy (e.g., bronchoscopy) may involve accessing and visualizing the interior of a patient's lumen (e.g., airway) for diagnostic and / or therapeutic purposes. During the procedure, for example, a flexible tubular tool such as an endoscope may be inserted into the patient's body, and instruments may be passed through the endoscope to a tissue site identified for diagnosis and / or treatment. Summary of the invention

[0007] Endoscopes have a wide range of applications in the diagnosis and treatment of various conditions, such as medical conditions (e.g., early lung cancer diagnosis and treatment). Endoscopic navigation systems can use different sensing modalities (e.g., camera imaging data, electromagnetic (EM) position data, robotic position data, etc.), for example, through adaptively adjusted probabilistic modeling. The navigation method may depend on an initial estimate of the position of the endoscope tip relative to the airway to begin tracking the endoscope tip. Some endoscopic examination techniques may involve a three-dimensional (3D) model of the patient's anatomy and use EM fields and position sensors to guide navigation. Prior to the procedure, the precise alignment (e.g., registration) between the virtual space of the 3D model, the physical space of the patient's anatomy represented by the 3D model, and the EM field may be unknown. Therefore, until the registration is generated, it is not possible to accurately map the endoscope position within the patient's anatomy to a corresponding position within the 3D model.

[0008] Navigation of bronchoscopes can be challenging due to inaccurate registration. In particular, current registration procedures can be time-consuming and cumbersome, or produce inconsistent results that are affected by manual input. Current registration procedures may involve generating an estimated initial registration, and then refining or updating the initial registration during the procedure. For example, current registration methods can compensate for changes in the navigation system by sampling from real-time sensor data. However, as the device is driven during long-term operation and as the system collects more input data, the computation time increases dramatically. In addition, current registration procedures may not have satisfactory online update capabilities, which causes the registration accuracy to be impaired. For example, current registration procedures may not be able to adapt to local changes that may cause inaccuracies in the registration algorithm. For example, the sampled data set is often used to perform a global update of the registration algorithm (e.g., a transformation), which does not accurately reflect local changes (e.g., due to local, small mechanical deflections). In another example, when the endoscope does not travel along a predetermined path (e.g., the centerline of the cut airway, etc.), the current registration algorithm may have poor transformation accuracy.

[0009] It is recognized herein that there is a need for a minimally invasive system that allows surgical or diagnostic procedures to be performed with increased reliability and cost efficiency. It is recognized herein that another need is an improved registration algorithm that can improve registration accuracy while reducing registration time. The present disclosure provides systems and methods that allow standardized early lung cancer diagnosis and treatment at a reduced cost. The present disclosure provides accessible, more cost-effective methods and systems for early diagnosis and treatment of cancer. In some embodiments of the present invention, at least a portion of the robotic bronchoscopy system is disposable. For example, the catheter portion can be designed to be low-cost disposable while maintaining surgical performance and functionality. Moreover, the provided robotic bronchoscopy system is designed to be able to access difficult-to-reach tissues, such as bronchi and lungs, without adding additional costs.

[0010] The adaptive navigation algorithm may be able to identify a registration or mapping between the coordinate system of the 3D model (e.g., the coordinate system of the CT scanner used to generate the model) and the coordinate system of the EM field (e.g., the coordinate system of the EM field generator) with instant updating capabilities.

[0011] In one aspect, a method for navigating a robotic endoscopic instrument is provided. The method includes: (a) generating a first transformation between an orientation of the robotic endoscopic instrument and an orientation of the position sensor based at least in part on a first sensor data set acquired using a position sensor; (b) generating a second transformation between a coordinate system of the robotic endoscopic instrument and a coordinate system of a model representing an anatomical luminal network based at least in part on the first transformation and a second sensor data set; and (c) updating the second transformation using an update algorithm based at least in part on a third sensor data set.

[0012] In some embodiments, the update algorithm includes a fast interval recalculation operation and a slow interval recalculation operation. In some cases, the fast interval recalculation operation includes (i) using a subset of data sampled from the third sensor data set to calculate a first set of associations and (ii) combining the first set of associations with a second set of associations calculated to generate the second transformation in (b). For example, the method also includes calculating a point cloud using the combined first and second sets of associations. In some cases, the slow interval recalculation operation includes a nearest neighbor algorithm. In some cases, the slow interval recalculation operation includes updating the second transformation using only the third sensor data set.

[0013] In some embodiments, the position sensor is an electromagnetic sensor.In some embodiments, the coordinate system of the model representing the anatomical luminal network is generated using a preoperative imaging system.

[0014] In some embodiments, the robotic endoscopic instrument includes a disposable catheter assembly. In some embodiments, the position sensor is located at a distal tip of the robotic endoscopic instrument.

[0015] In another aspect, a system for navigating a robotic endoscopic instrument is provided. The system includes: a position sensor located at a distal end of the robotic endoscopic instrument; and one or more processors in communication with the position sensor and the robotic endoscopic instrument and configured to execute a set of instructions to cause the system to: (a) generate a first transformation between an orientation of the robotic endoscopic instrument and an orientation of the position sensor based at least in part on a first sensor data set acquired using the position sensor; (b) generate a second transformation between a coordinate system of the robotic endoscopic instrument and a coordinate system of a model representing an anatomical luminal network based at least in part on the first transformation and a second sensor data set; and (c) update the second transformation using an update algorithm based at least in part on a third sensor data set.

[0016] In some embodiments, the update algorithm includes a fast interval recalculation operation and a slow interval recalculation operation. In some cases, the fast interval recalculation operation includes (i) using a subset of data sampled from the third sensor data set to calculate a first association set and (ii) combining the first association set with a second association set calculated to generate the second transformation in (b). For example, the fast interval recalculation operation also includes calculating a point cloud using the combined first association set and the second association set. In some cases, the slow interval recalculation operation includes a nearest neighbor algorithm. In some cases, the slow interval recalculation operation includes updating the second transformation using only the third sensor data set.

[0017] In some embodiments, the position sensor is an electromagnetic sensor. In some embodiments, the coordinate system of the model representing the anatomical luminal network is generated using a preoperative imaging system. In some embodiments, the robotic endoscopic instrument includes a disposable catheter assembly.

[0018] According to some aspects of the present disclosure, a robotic endoscopic instrument is provided. The instrument may include a disposable elongated member including a proximal end and a distal end, and the proximal end is removably attached to a robotic arm. The distal end includes a plurality of pull wires and the pull wires are integral with the wall of the elongated member. The elongated member may also be referred to as a bronchoscope, a catheter, which are used interchangeably throughout the specification.

[0019] In another aspect of the present disclosure, an improved registration algorithm or navigation method is provided. The registration algorithm can allow automatic registration with minimal user interaction, as well as online registration updates that beneficially improve registration accuracy. It should be noted that the provided robotic system and / or registration algorithm can be used for various minimally invasive surgical procedures involving various types of tissues, including heart, bladder, and lung tissue, among others.

[0020] Other aspects and advantages of the present disclosure will become readily understood by those skilled in the art through the following detailed description, wherein only illustrative embodiments of the present disclosure are shown and described. As will be appreciated, the present disclosure can have other and different embodiments, and its several details can be modified in various obvious aspects, all without departing from the present disclosure. Therefore, the drawings and descriptions should be considered illustrative in nature, rather than restrictive.

[0021] Incorporation by reference

[0022] All publications, patents, and patent applications mentioned in this specification are incorporated herein by reference to the same extent as if each individual publication, patent, or patent application was specifically and individually indicated to be incorporated by reference. To the extent that publications and patents or patent applications incorporated by reference conflict with the disclosure contained in the specification, the specification is intended to supersede and / or take precedence over any such conflicting material. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The novel features of the present invention are particularly set forth in the appended claims. A better understanding of the features and advantages of the present invention will be obtained by reference to the following detailed description of illustrative embodiments in which the principles of the present invention are utilized and the accompanying drawings (also referred to herein as "drawings" and "figures"), in which:

[0024] Figure 1 An example workflow for standardized lung cancer diagnosis enabled by the robotic bronchoscopy system described herein is shown.

[0025] Figure 2A An example of a robotic bronchoscopy system according to some embodiments of the invention is shown.

[0026] Figure 2B Different views of an example robotic bronchoscopy system are shown according to some embodiments of the present invention.

[0027] Figure 3 An example of a fluoroscopic (tomosynthesis) imaging system is shown.

[0028] Figure 4 A C-arm fluoroscopic (tomosynthesis) imaging system is shown in different (rotated) postures when capturing images of a subject.

[0029] Figure 5 An example of a user interface for visualizing a virtual airway overlaid with optimal path, catheter tip location, and lesion location is shown.

[0030] Figure 6 An example portable robotic cone beam CT is shown.

[0031] Figure 7 An example of an automatic registration method according to some embodiments of the present invention is schematically illustrated.

[0032] Figure 8 An example of a customized iterative closest point (ICP) algorithm is shown.

[0033] Fig. 9 An example of the classification result is shown.

[0034] Fig.10 An example of a data structure (spatial data structure) is shown. A specialized data structure may be a kd-tree or an octree data structure. DETAILED DESCRIPTION

[0035] Although various embodiments of the present invention have been shown and described herein, it is readily understood by those skilled in the art that these embodiments are provided as examples only. Without departing from the present invention, those skilled in the art may conceive of many variations, changes and substitutions. It should be understood that various alternatives to the embodiments of the present invention described herein may be employed.

[0036] While the exemplary embodiments will be primarily directed to bronchoscopes, those skilled in the art will appreciate that this is not intended to be limiting and that the devices described herein may be used for other therapeutic or diagnostic procedures and for other anatomical regions of a patient's body, such as the digestive system, including but not limited to the esophagus, liver, stomach, colon, urinary tract, or the respiratory system, including but not limited to the bronchi, lungs, and various other systems. The registration methods / algorithms herein may be used to align the coordinate system of a medical device with the coordinate system of a 3D model (e.g., an airway model or a patient coordinate system generated by a CT scanner), regardless of the type of device or procedure to be performed.

[0037] The embodiments disclosed herein can be combined in one or more of a variety of ways to provide improved diagnosis and treatment to patients. The disclosed embodiments can be combined with existing methods and instruments to provide improved treatment, such as in combination with known lung diagnostic methods, surgical procedures, and surgical procedures of other tissues and organs. It should be understood that any one or more of the structures and steps described herein can be combined with any one or more additional structures and steps of the methods and instruments described herein, and the accompanying drawings and supporting text provide descriptions according to the embodiments.

[0038] Although the treatment plans and definitions of diagnostic or surgical procedures described herein are presented in the context of pulmonary diagnosis or surgery, the methods and apparatus described herein can be used to treat any tissue of the body and any organ and blood vessels of the body, such as the brain, heart, lungs, intestines, eyes, skin, kidneys, liver, pancreas, stomach, uterus, ovaries, testicles, bladder, ears, nose, mouth, soft tissue (e.g., bone marrow), adipose tissue, muscle, glandular and mucosal tissue, spinal cord and nerve tissue, cartilage, hard biological tissue (e.g., teeth, bones, etc.), and body cavities and passages (e.g., sinuses, ureters, colon, esophagus, lung tracts, blood vessels, and throat).

[0039] Whenever the term "at least", "greater than", or "greater than or equal to" precedes the first value in a series of two or more values, the term "at least", "greater than", or "greater than or equal to" applies to each value in the series. For example, greater than or equal to 1, 2, or 3 is equivalent to greater than or equal to 1, greater than or equal to 2, or greater than or equal to 3.

[0040] Whenever the term "not greater than," "less than," or "less than or equal to" precedes the first value in a series of two or more values, the term "not greater than," "less than," or "less than or equal to" applies to every value in the series. For example, less than or equal to 3, 2, or 1 is equivalent to less than or equal to 3, less than or equal to 2, or less than or equal to 1.

[0041] As used herein, a processor includes one or more processors, such as a single processor, or multiple processors of a distributed processing system, for example. For example, a controller or processor as described herein generally includes a tangible medium to store instructions to implement the steps of a process, and the processor may include one or more of a central processing unit, a programmable array logic, a gate array logic, or a field programmable gate array. In some cases, one or more processors may be a programmable processor (e.g., a central processing unit (CPU) or a microcontroller), a digital signal processor (DSP), a field programmable gate array (FPGA), and / or one or more advanced RISC machine (ARM) processors. In some cases, one or more processors are operably coupled to a non-transitory computer-readable medium. A non-transitory computer-readable medium may store logic, code, and / or program instructions that may be executed by one or more processor units to perform one or more steps. A non-transitory computer-readable medium may include one or more memory units (e.g., removable media or external storage, such as an SD card or a random access memory (RAM)). One or more methods or operations disclosed herein may be implemented in a hardware component or a combination of hardware and software, such as an ASIC, a special-purpose computer, or a general-purpose computer.

[0042] As used herein, the terms distal and proximal may generally refer to a position referenced from an instrument and may be in contrast to an anatomical reference. For example, a distal position of a bronchoscope or catheter may correspond to a proximal position of an elongated member of a patient, and a proximal position of a bronchoscope or catheter may correspond to a distal position of an elongated member of a patient.

[0043] A system as described herein includes an elongated portion or elongated member, such as a catheter. Unless the context indicates otherwise, the terms "elongated member", "catheter", "bronchoscope" are used interchangeably throughout the specification. The elongated member can be placed directly in a body cavity or body cavity. In some embodiments, the system may also include a supporting instrument, such as a robotic manipulator (e.g., a robotic arm) to drive, support, position or control the movement and / or operation of the elongated member. Alternatively or in addition, the supporting instrument may be a handheld instrument or may or may not include other control devices of the robotic system. In some embodiments, the system may further include peripheral devices and subsystems, such as an imaging system that assists and / or facilitates the navigation of the elongated member to a target site in the subject's body. Such navigation may require a registration process, which will be described later in this article.

[0044] In some embodiments of the present disclosure, a robotic bronchoscopy system is provided that performs surgical procedures or diagnostics with improved performance at low cost. For example, a robotic bronchoscopy system may include a completely disposable, steerable catheter. This can beneficially reduce sterilization requirements that are costly or difficult to operate, but sterilization or disinfection may be ineffective. In addition, one challenge of bronchoscopy is reaching the upper lobes of the lungs while passing through the airways. In some cases, the provided robotic bronchoscopy system may be designed to have the ability to navigate through airways with small bends in an autonomous or semi-autonomous manner. Autonomous or semi-autonomous navigation may require a registration process as described later in this article. Alternatively, the robotic bronchoscopy system may be navigated by an operator via a control system with visual guidance.

[0045] The typical lung cancer diagnosis and surgical treatment process can vary widely, depending on the technology used by healthcare providers, clinical protocols, and clinical locations. Inconsistent processes can lead to delays in early lung cancer diagnosis, high costs to healthcare systems and patients diagnosing and treating lung cancer, and a high risk of clinical and surgical complications. The provided robotic bronchoscopy system can enable standardized early lung cancer diagnosis and treatment. Figure 1 An example workflow 100 for standardized lung cancer diagnosis enabled by the robotic bronchoscopy system described herein is shown.

[0046] like Figure 1As shown, preoperative imaging can be performed to identify lesions. Any suitable imaging modality, such as magnetic resonance (MR), positron emission tomography (PET), X-ray, computed tomography (CT), and ultrasound, can be used to identify lesions or regions of interest. For example, a preoperative CT scan can be performed on a patient suspected of lung cancer, and suspected lung nodules can be identified in the CT images. The preoperative imaging process can be performed before the bronchoscopy.

[0047] Next, the CT images can be analyzed to generate a map to guide the navigation of the robotic bronchoscope during bronchoscopy. For example, a lesion or region of interest (ROI) can be segmented on the image. When the lungs are being imaged, the passage or pathway to the lesion can be highlighted on the reconstructed image to plan the navigation path. The reconstructed image can guide the robotic bronchoscope to navigate to the target tissue or target site. In some cases, the navigation path can be pre-planned using 3D image data. For example, under the robotic control of the robotic bronchoscope system, the catheter can be advanced to the target site. The catheter can be manipulated or advanced to the target site manually, autonomously, or semi-autonomously. In an example, the movement of the catheter can be guided by an image so that the insertion and / or manipulation direction can be automatically controlled.

[0048] In some cases, the location of the lesion in preoperative imaging may be inaccurate due to patient movement or body divergence. In this case, the location of the lesion can be verified before a surgical procedure (e.g., a biopsy or treatment). The exact location of the lesion can be verified or updated with the aid of a robotic bronchoscopy system. For example, a bronchoscopy system can provide an interface to an imaging mode such as fluoroscopy to provide real-time in vivo imaging of the target site and surrounding area to locate the lesion. In an example, a C-arm or O-arm fluoroscopic imaging system can be used to generate a tomosynthesis image to verify or update the location of the lesion. When performing a surgical procedure such as a biopsy, various surgical tools such as a biopsy tool, a brush, or forceps can be inserted into the working channel of the catheter to perform a biopsy or other surgical procedure manually or automatically.

[0049] Next, a sample of the lesion or any other target tissue can be obtained by a tool inserted through the working channel of the catheter. The system allows camera visualization to be maintained throughout the entire process, including during the insertion of the tool through the working channel. In some cases, the tissue sample can be quickly evaluated on-site through a rapid field assessment process to determine whether repeated tissue sampling is needed, or to decide on further actions. In some cases, the rapid field assessment process can also provide a rapid analysis of the tissue sample to determine the next surgical treatment. For example, if the tissue sample is determined to be malignant due to the rapid field assessment process, a manual or robotic treatment instrument can be inserted through the working channel of the robotic bronchoscope and endobronchial treatment of lung cancer can be performed. This beneficially allows diagnosis and treatment to be performed in one treatment session, thereby providing targeted, painless and rapid treatment of early-stage lung cancer.

[0050] Figure 2A and Figure 2B An example of a robotic bronchoscopy system 200, 230 according to some embodiments of the present invention is shown. Figure 2A As shown, the robotic bronchoscopy system 200 may include a steerable catheter assembly 220 and a robotic support system 210 for supporting or carrying the steerable catheter assembly. The steerable catheter assembly may be a bronchoscope. In some embodiments, the steerable catheter assembly may be a disposable robotic bronchoscope. In some embodiments, the robotic bronchoscopy system 200 may include an instrument drive mechanism 213 attached to the arm of the robotic support system. The instrument drive mechanism may be provided by any suitable controller device (e.g., a handheld controller) that may or may not include a robotic system. The instrument drive mechanism may provide a mechanical and electrical interface for the steerable catheter assembly 220. The mechanical interface may allow the steerable catheter assembly 220 to be releasably coupled to the instrument drive mechanism. For example, the handle portion of the steerable catheter assembly may be attached to the instrument drive mechanism via a quick installation / release tool (e.g., a magnet, a spring-loaded level, etc.). In some cases, the steerable catheter assembly may be manually coupled to the instrument drive mechanism or released from the instrument drive mechanism without using a tool.

[0051] The manipulable catheter assembly 220 may include a handle portion 223, which may include a component configured to process image data, provide power, or establish communication with other external devices. For example, the handle portion 223 may include circuits and communication elements that can communicate electrically between the manipulable catheter assembly 220 and the instrument drive mechanism 213 and any other external system or device. In another example, the handle portion 223 may include circuit elements, such as a power supply for powering the electronic devices (e.g., cameras and LED lights) of the endoscope. In some cases, the handle portion may communicate electrically with the instrument drive mechanism 213 via an electrical interface (e.g., a printed circuit board), so that the communication module of the instrument drive mechanism may receive image / video data and / or sensor data, and may be transmitted to other external devices / systems. Alternatively or in addition, the instrument drive mechanism 213 may only provide a mechanical interface. The handle portion may communicate electrically with a modular wireless communication device or any other user device (e.g., a portable / handheld device or a controller) for transmitting sensor data and / or receiving control signals. The details of the handle portion herein will be described later.

[0052] Manipulable catheter assembly 220 can include a flexible elongated member 211 coupled to a handle portion. In some embodiments, the flexible elongated member can include a shaft, a manipulable tip, and an operable segment. The manipulable catheter assembly can be a disposable robotic bronchoscope. In some cases, only the elongated member can be disposable. In some cases, at least a portion of the elongated member (e.g., a shaft, a manipulable tip, etc.) can be disposable. In some cases, the entire manipulable catheter assembly 220 including the handle portion and the elongated member can be disposable. The flexible elongated member and the handle portion are designed so that the entire manipulable catheter assembly can be disposed of at low cost. The details of this article about the flexible elongated member and the manipulable catheter assembly will be described later.

[0053] In some embodiments, the provided bronchoscope system may further include a user interface. As shown in the example system 230, the bronchoscope system may include a treatment interface module 231 (user console side) and / or a treatment control module 233 (patient and robot side). The treatment interface module may allow an operator or user to interact with the bronchoscope during a surgical procedure. In some embodiments, the treatment control module 233 may be a handheld controller. In some cases, the treatment control module may include a proprietary user input device and one or more additional elements that are removably coupled to an existing user device to improve the user input experience. For example, a physical trackball or scroll wheel may replace or supplement the functionality of at least one virtual graphical element (e.g., a navigation arrow displayed on a touchpad) displayed on a graphical user interface (GUI) by giving it functionality similar to the graphical element it replaces. Examples of user devices may include, but are not limited to, mobile devices, smart phones / cellular phones, tablet computers, personal digital assistants (PDAs), laptop or notebook computers, desktop computers, media content players, and the like. Details of the user interface device and the user console will be described later herein.

[0054] Figure 2B Different views of a bronchoscope system are shown.A user console 231 may be mounted to the robotic support system 210. Alternatively or in addition, the user console or a portion of the user console (eg, a therapy interface module) may be mounted to a separate mobile cart.

[0055] Robotic intracavitary platform

[0056] In one aspect, a robotic intraluminal platform is provided. In some cases, the robotic intraluminal platform can be a bronchoscopy platform. The platform can be configured to perform Figure 1 One or more operations consistent with the method described in. Figures 3 to 7 Various examples of robotic endoluminal platforms and components or subsystems thereof according to some embodiments of the present invention are shown. In some embodiments, the platform can include a robotic bronchoscopy system and one or more subsystems that can be used in conjunction with the robotic bronchoscopy system of the present disclosure.

[0057] In some embodiments, one or more subsystems may include an imaging system, such as a fluoroscopic (tomosynthesis) imaging system for providing real-time imaging of a target site (eg, including a lesion). Figure 3 An example of a fluoroscopic (tomosynthesis) imaging system 300 is shown. For example, a fluoroscopic (tomosynthesis) imaging system can be used in a Figure 1Perform accurate lesion position tracking or verification before or during the surgical procedure described in . In some cases, the lesion position can be tracked based on the position data of the perspective (tomosynthesis) imaging system / station (e.g., C-arm) and the image data captured by the perspective (tomosynthesis) imaging system. The lesion position can be registered with the coordinate system of the robotic bronchoscopy system. The position or motion of the perspective (tomosynthesis) imaging system can be measured using any suitable motion / position sensor 310 (e.g., inertial measurement unit (IMU)), one or more gyroscopes, velocity sensors, accelerometers, magnetometers, position sensors (e.g., global positioning system (GPS) sensors), visual sensors (e.g., imaging devices capable of detecting visible light, infrared or ultraviolet light, such as cameras), proximity or distance sensors (e.g., ultrasonic sensors, lidar, time of flight or depth cameras), altitude sensors, attitude sensors (e.g., compasses) and / or field sensors (e.g., magnetometers, electromagnetic sensors, radio sensors). One or more sensors for tracking the motion and position of the fluoroscopic (tomosynthesis) imaging station may be disposed on the imaging station or located remotely from the imaging station, such as a wall mounted camera 320 . Figure 4 A C-arm fluoroscopic (tomosynthesis) imaging system is shown in different (rotated) poses when taking images of a subject. As described above, various poses may be captured by one or more sensors.

[0058] In some embodiments, the location of the lesion can be segmented in the image data captured by the perspective (tomography) imaging system with the help of the signal processing unit 330. One or more processors of the signal processing unit can be configured to further cover the treatment position (e.g., lesion) on the real-time perspective image / video. For example, the processing unit can be configured to generate an enhancement layer including enhancement information, such as the location of the treatment position or the target site. In some cases, the enhancement layer can also include a graphic mark representing the path to the target site. The enhancement layer can be a substantially transparent image layer including one or more graphic elements (e.g., boxes, arrows, etc.). The enhancement layer can be overlaid on the optical view of the optical image or video stream captured by the perspective (tomography) imaging system, and / or displayed on a display device. The transparency of the enhancement layer allows the user to view the optical image, which is overlaid with graphic elements. In some cases, the segmented lesion image and the best path for navigating the elongated member to reach the lesion can be overlaid on the real-time tomosynthesis image. This can allow the operator or user to visualize the exact location of the lesion and the planned path for the movement of the bronchoscope. In some cases, segmented and reconstructed images (eg, CT images described elsewhere) provided prior to operation of the systems described herein may be overlaid on the real-time images.

[0059] In some embodiments, one or more subsystems of the platform may include a navigation and positioning subsystem. The navigation and positioning subsystem may be configured to build a virtual airway model based on preoperative images (e.g., preoperative CT images or tomosynthesis). The navigation and positioning subsystem may be configured to identify the segmented lesion location in the 3D rendered airway model, and based on the lesion location, the navigation and positioning subsystem may generate an optimal path from the main bronchus to the lesion and recommend a surgical procedure (e.g., a biopsy) at an angle close to the lesion.

[0060] In a registration step before driving the bronchoscope to the target site, the system can align the rendered virtual view of the airway with the patient's airway. Image registration can include a single registration step or a combination of a single registration step and real-time sensory updating of the registration information. The registration process can include finding a transformation of the aligned objects (e.g., airway model, anatomical site) between different coordinate systems (e.g., EM sensor coordinates and patient 3D model coordinates based on preoperative CT imaging). Details of the registration will be described later in this article.

[0061] Once registered, all airways can be aligned with the airways presented preoperatively. During the driving of the robotic bronchoscope toward the target site, the position of the bronchoscope within the airway can be tracked and displayed. In some cases, a position sensor can be used to track the position of the bronchoscope relative to the airway. Other types of sensors (e.g., cameras) can also be used instead of or in combination with position sensors using sensor fusion techniques. Position sensors such as electromagnetic (EM) sensors can be embedded in the distal tip of the catheter, and an EM field generator can be placed next to the patient's torso during surgery. The EM field generator can place the EM sensor position in 3D space, or can place the EM sensor position and orientation in 5D or 6D space. This can provide visual guidance for the operator when driving the bronchoscope toward the target site.

[0062] In real-time EM tracking, an EM sensor comprising one or more sensor coils is embedded in a medical instrument (e.g., the tip of an endoscopic tool) at one or more locations and orientations, measuring changes in the EM field generated by one or more static EM field generators located adjacent to the patient. The position information detected by the EM sensor is stored as EM data. An EM field generator (or transmitter) can be placed adjacent to the patient to generate a low-intensity magnetic field that can be detected by the embedded sensor. The magnetic field induces a small current in the sensor coil of the EM sensor, which can be analyzed to determine the distance and angle between the EM sensor and the EM field generator. These distances and orientations can be registered to the patient's anatomy (e.g., a 3D model) during surgery to determine a registration transformation that aligns a single location in a coordinate system with a location in a preoperative model of the patient's anatomy.

[0063] Figure 5 An example of a user interface for visualizing a virtual airway 509 overlaid with an optimal path 503, a catheter tip position 501, and a lesion location 505 is shown. In this example, the catheter tip position is displayed in real time relative to the virtual airway model 509, providing visual guidance. Figure 5 As shown in the example of, during the driving of the robotic bronchoscope, the optimal path 503 can be displayed and overlaid on the virtual airway model. As described above, the virtual airway model can be constructed based on the real-time perspective image / video (and the position data of the imaging system). In some cases, a view of the real-time perspective image / video 507 can also be displayed on the graphical user interface. In some cases, the user can also be allowed to access the camera view or image / video 511 captured in real time by the bronchoscope.

[0064] In some embodiments, one or more subsystems of the platform may include one or more therapeutic subsystems, such as manual or robotic instruments (e.g., biopsy needles, biopsy forceps, biopsy brushes) and / or manual or robotic therapeutic instruments (e.g., RF ablation instruments, cryogenic instruments, microwave instruments, etc.).

[0065] Conventional cone beam CT machines can have C-shaped or O-shaped transmitter and receiver panels on the same mechanical structure. The connection between the transmitter and receiver panels causes the cone beam CT to be large in size. This oversized design limits the use cases and takes up a lot of space in the rather cramped operating room.

[0066] Described herein is a design for decoupling the mechanical connection between the transmitter and receiver panels. Figure 6 An example of a portable robotic cone beam CT is shown. The transmitter and receiver panels can be mounted on two separate robotic arms, as shown in Figure 6 When used, the two robots can move in the same coordinate system. The control algorithm can ensure that the two robots move in synchronized motion.

[0067] In addition, for patient gated motion, i.e., breathing, additional external sensors - i.e., IMU, EM, or image sensors - can be added to track the patient's motion. The patient's position changes can be tracked using sensors such as IMU, EM, or image sensors. Sensory signals can be used to command both robotic arms. In some cases, one or both of the robotic arms can move to track the patient's motion, which essentially keeps the transmitter and receiver stationary while tracking the patient's motion in the region of interest (ROI). The ROI can include a target site or target position that can be automatically determined by the system or manually determined by the physician. Tracking can also use other mechanisms, such as, but not limited to, external cameras and one or more trackers on the patient's body.

[0068] It will be appreciated by those skilled in the art that cone beam CT is a non-limiting example. The designs described herein can be used in other imaging modalities, such as fluoroscopy machines, classical CT machines, and MRI machines.

[0069] Automatic registration algorithm

[0070] The present disclosure provides an improved automatic registration algorithm that can improve registration accuracy while reducing registration time. The provided automatic registration algorithm can advantageously allow for minimal user interaction and can establish a transformation or registration that is improved over existing methods. In addition, the provided automatic registration algorithm can allow for dynamic updating of the algorithm to adapt to real-time conditions (e.g., automatically updating the registration based on real-time detected registration errors), thereby improving registration accuracy as the device is driven within a subject.

[0071] In some implementations, the automatic registration algorithm may include or may be implemented in three stages. Figure 7 An example of an automatic registration method 700 according to some embodiments of the present invention is schematically shown. The automatic registration method may include one or more algorithms to update the registration information in real time without user input. The automatic registration algorithm may also include one or more algorithms for improving the registration accuracy by identifying the best registration / transformation matrix and utilizing time information.

[0072] As described above, an EM sensor can be coupled to the distal end of the endoscope to track its position within the EM field. The EM field is stationary relative to the EM field generator, and the coordinate system of the 3D model of the luminal network (e.g., CT space) can be mapped to the coordinate system of the EM field. Figure 7 As shown, the automatic registration method 700 may include a first stage which may be an initial data acquisition stage 701. The initial data acquisition stage may include reconstructing the distal tip of the catheter and performing an initial translation transformation.

[0073] The initial data acquisition phase 701 can align the EM sensor orientation to the distal tip orientation. This can establish an association between the robotic endoscopic instrument space and the position sensor space. For example, an electromagnetic coil located at the distal end can be used with an electromagnetic tracking system to detect the position and orientation of the distal tip of the catheter while it is disposed within an anatomical system (e.g., an anatomical luminal network). In some embodiments, the coil can be tilted to provide sensitivity to electromagnetic fields along different axes, thereby enabling the disclosed navigation system to measure a full 6 degrees of freedom: three positions and three angles. The orientation / position of the EM field generator may be unknown, and the initial data acquisition phase can use the acquired sensor data to align the EM sensor orientation (e.g., the Z axis of the EM coordinate system) to the distal tip orientation (e.g., the catheter tip Z axis).

[0074] During the first stage, sensor data (e.g., EM sensor data) may be collected to register the z-axis of the magnetic field generator to the z-axis of the catheter tip. This may allow flexibility in placing the magnetic field generator without a pre-known position and orientation. The sensor data may include flow data collected when the catheter is driven in a known orientation. The flow data may include various types of data, including but not limited to: time series data, such as space-time point measurements generated by EM sensors. In some cases, time series data may be collected when the catheter is driven in a known orientation, which may or may not be in the patient's body. For example, when the distal tip is placed inside an object such as a bronchus or trachea and travels in a known orientation, EM sensor data points may be collected. In some cases, the z-axis and EM orientation of the catheter tip may be calibrated during assembly or manufacturing. In some cases, the z-axis of the catheter tip may be obtained from robot data when the catheter tip is driven in a known direction or with the aid of CT imaging or other real-time imaging techniques. An initial translational transformation between the catheter tip and the EM field may be obtained. As more and more input sensor data is collected, this translational and rotational transformation may be continuously refined and modified during the first stage. In some cases, the first stage translation homogeneous transform may be updated and refined until a threshold is reached. The threshold may be an accuracy threshold indicating that accuracy has been met. Alternatively or in addition, updates may be performed continuously (e.g., errors may be calculated at each time interval) and updates may be stopped when tomosynthesis targeting is performed.

[0075] In some cases, the acquired sensor data can also provide registration in terms of a translational transformation between CT space (e.g., tissue) and EM space (e.g., catheter). For example, a registration transformation between the EM field and the patient model (e.g., the coordinate system of the luminal network model) can be identified. This translational transformation in the second stage can include an initial translational transformation matrix that can be later refined in the final stage, as described later in this article.

[0076] The second stage 703 of the registration method can include creating an association between EM space and model (e.g., CT) space. During the second stage, based on the translation information and real-time sensor data generated in the first stage, an initial transformation matrix between EM space and CT space can be generated at least in part. In some cases, when the catheter navigates in the trachea and / or along the sub-branch of the predefined navigation path, the data points (e.g., EM data) used in the second stage can be collected. The predefined navigation path can be defined in the CT space. As described elsewhere herein, the EM data can include information about orientation, position, and error data. The coordinate system of the CT space or anatomical model can be generated in the preoperative process or during the surgical operation. Such sensor data points can be used to establish a plane in space, and based on the direction distance set (e.g., orientation and length) respectively traveled in the EM space and CT space, an association set is established between the EM space and the CT space. The association set can be generated at least in part based on time series data of spatiotemporal data points such as those generated by the EM sensor.

[0077] In some cases, the association set can be processed to identify an optimal transformation matrix. For example, the best fit of an EM curve (e.g., a point cloud in EM space) and an established CT curve (e.g., a point cloud in CT space) can be determined in 3D space to identify the optimal transformation matrix. For example, the provided automatic registration method can use a modified version of an iterative closest point (ICP) algorithm with nearest neighbor and singular value decomposition to iteratively calculate to the optimal solution. It is worth noting that other algorithms or variants suitable for identifying the best fit can be used.

[0078] Figure 8 An example of a customized iterative closest point (ICP) algorithm for generating an initial transformation between two data point clouds (e.g., an EM domain point cloud and a CT domain point cloud) is shown. ICP is an iterative algorithm that alternates between determining the nearest neighbor (NN) for each point of a shape based on a current transformation estimate and updating the estimate based on the NN. An association solver can receive a calculated association set (e.g., Cn, Xn) between two sets of data point clouds in CT space and EM space and be executed to obtain a transformation that best matches the association set. As an example, the association set can be calculated using a k-nearest neighbor (k-NN) algorithm. The best fit transformation can be calculated using singular value decomposition during the error minimization step. Assumptions (e.g., neighbor assumptions) can be modified to soft-lock the transformation translations and rotations to good initial values.

[0079] Examples of classification results are given in Fig. 9 The method provided can further improve the performance of the k-NN algorithm by utilizing a specialized data structure to reduce the computation time. Fig.10An example of a data structure (spatial data structure) is shown. The specialized data structure may be a kd-tree or octree data structure. Due to their regular partitioning of the search space and high branching factor, kd-tree or octree data structures may allow efficient implementation of NNS, and coordinate queries are fast. Other data structures suitable for shape registration and storage efficiency may also be used.

[0080] Return to reference Figure 8 , the error minimization operation may include using singular value decomposition to compute the best fit transformation. For example, singular value decomposition may be applied to the association set to compute the best transformation. The singular value decomposition method may be used to solve the total least squares minimization problem.

[0081] It is worth noting that the above algorithms can be modified or replaced by other algorithms. For example, the Coherent Point Drift algorithm can be used to identify the best fit transformation based on probability rather than rigid point positions. The algorithm can scale the point cloud while rotating it (and / or deform the point cloud using probabilistic convergence) so that the calculated features (usually the centroid) traverse the probabilistic optimization surface until it aligns with another point cloud. The Coherent Point Drift algorithm or other probability-based algorithms can reduce the impact of noise on the dataset.

[0082] Return to reference Figure 7 , the third stage 705 of the automatic registration algorithm may include updating the initial registration / transformation produced in the second stage. In some embodiments, the update method may include weak real-time updates and strong real-time updates to the initial transformation. The weak real-time updates may also be referred to as fast interval updates, and the strong real-time updates may also be referred to as slow interval updates, which are used interchangeably throughout the specification. The updates may be performed using real-time data without user intervention. In some cases, the updates may utilize real-time EM sensor data collected from a large point cloud. In some cases, a subset of data points may be selected / sampled from the large point cloud for calculating errors and determining whether an update is needed based on the errors.

[0083] In some cases, the update method may include weak real-time updates and strong real-time updates to the initial transformation. For example, the update method may include a fast interval refinement step and a slow interval recalculation step. In some cases, after each update, the error may be recalculated and the transformation matrix may be updated when the error decreases.

[0084] The fast interval refinement operation (e.g., fast interval update) can involve randomly sampling a subset of data points from a large point cloud (e.g., the data points collected in the third stage). Next, the current transformation and the nearest neighbor algorithm can be applied to the sampled set to create a correlation set. The correlation set can be combined with the correlation set calculated in the second stage to create the final point cloud. The final point cloud data can be input into the modified ICP algorithm as described in the second stage to generate an updated transformation, thereby refining the current transformation.

[0085] The slow interval recalculation operation can include an update operation similar to that described above, using only a nearest neighbor algorithm without utilizing the initial transformation information or association set generated in the second stage. For example, the association set can be calculated by applying a nearest neighbor algorithm to the sensor data collected in the third stage, and the transformation matrix can be updated using only the association set for which there is no information in the second stage. In some cases, the data subset may be randomly drawn from a large data set. Alternatively or in addition, the data subset can be selected based on distance. For example, a distance-based time filter can be used to sample uniformly distributed points along a time path and select these points for update calculations. The distance used for filtering can be based on a CT scan and / or patient anatomy.

[0086] Such a slow interval recalculation operation (i.e., strong update) can be performed at larger time / point intervals because it may cause a "jump" to the next local minimum. In some cases, the slow recalculation can be performed at fixed intervals (e.g., time intervals, a predetermined number of data points, etc.). In some cases, the slow recalculation may be triggered when the update error value begins to converge, indicating that a local or global minimum has been reached. Since the initial transformation is continuously refined by weak updates (i.e., fast interval recalculation), and in the iterations of the ICP algorithm, the "jump" can be ensured to be the required step or to proceed in the right direction. Combining weak updates (e.g., fast interval refinement) with strong updates, as well as performing weak updates more frequently than strong updates, can beneficially ensure that the "jump" is performed in the right direction, as the refinement step increases the distance to the incorrect local minimum and reduces the distance to the global minimum.

[0087] Random sampling of data points during the update phase (e.g., fast interval updates) can also be beneficial to prevent overfitting. Overfitting can occur due to noisy data, fitting cylindrical data to a single line. Random sampling can provide the ability to find and refine a general solution, rather than jumping from one local minimum to another, allowing for smooth traversal of the error surface and improving registration accuracy and stability. Randomly sampled data points can also indirectly weight the initial associations into the estimated associations (computed in the fast interval recalculation). High-precision data is computed in the second phase, and due to the complex structure and small gaps between peripheral airways, it is desirable to weight this calculation more than the nearest neighbor calculation. Random data sampling during the update step can further avoid getting stuck in a local minimum that can stall the entire registration process. Reaching a local minimum can be unavoidable, primarily because even if a global solution is found at iteration x, iteration x+1 may add data that causes the previous transform to become a local minimum. Random sampling can be beneficial to smooth iterations and error calculations.

[0088] In some embodiments, the automatic registration algorithm can also use a preprocessing algorithm to preprocess the collected sensor data to improve performance. In some cases, the Ramer-Douglas-Peucker algorithm can be used to preprocess large data sets of time series data between two location points. The algorithm recursively finds points that retain the shape of the overall data set while simplifying the path. In some cases, algorithms such as exponential smoothing can be used to remove noise in the data set while retaining small fluctuations in the curve. In another example, filters such as alpha-beta filtering can be used to remove noise in the data set. In some cases, a distance-based time filter can be used to select a subset of data, as described above.

[0089] The present invention provides embodiments including but not limited to the following:

[0090] 1. A method for navigating a robotic endoscopic instrument, comprising:

[0091] (a) generating a first transformation between an orientation of the robotic endoscopic instrument and an orientation of the position sensor based at least in part on a first sensor data set acquired using a position sensor;

[0092] (b) generating a second transformation between a coordinate system of the robotic endoscopic instrument and a coordinate system of a model representing an anatomical luminal network based at least in part on the first transformation and a second sensor data set; and

[0093] (c) updating the second transform using an update algorithm based at least in part on a third sensor data set.

[0094] 2. The method of embodiment 1, wherein the update algorithm comprises a fast interval recalculation operation and a slow interval recalculation operation.

[0095] 3. A method according to embodiment 2, wherein the fast interval recalculation operation includes (i) calculating a first association set using a subset of data sampled from the third sensor data set and (ii) combining the first association set with a second association set calculated to generate the second transformation in (b).

[0096] 4. The method according to embodiment 3 also includes calculating a point cloud using the combined first association set and second association set.

[0097] 5. A method according to embodiment 2, wherein the slow interval recalculation operation includes a nearest neighbor algorithm.

[0098] 6. The method of embodiment 2, wherein the slow interval recalculation operation comprises updating the second transform using only the third sensor data set.

[0099] 7. The method of embodiment 1, wherein the position sensor is an electromagnetic sensor.

[0100] 8. A method according to embodiment 1, wherein the coordinate system of the model representing the anatomical luminal network is generated using a preoperative imaging system.

[0101] 9. The method of embodiment 1, wherein the robotic endoscopic instrument comprises a disposable catheter assembly.

[0102] 10. The method of embodiment 1, wherein the position sensor is located at a distal tip of the robotic endoscopic instrument.

[0103] 11. A system for navigating a robotic endoscopic instrument, comprising:

[0104] a position sensor located at a distal end of the robotic endoscopic instrument; and

[0105] one or more processors in communication with the position sensor and the robotic endoscopic instrument and configured to execute a set of instructions to cause the system to:

[0106] (a) generating a first transformation between an orientation of the robotic endoscopic instrument and an orientation of the position sensor based at least in part on a first sensor data set acquired using the position sensor;

[0107] (b) generating a second transformation between a coordinate system of the robotic endoscopic instrument and a coordinate system of a model representing an anatomical luminal network based at least in part on the first transformation and a second sensor data set; and

[0108] (c) updating the second transform using an update algorithm based at least in part on a third sensor data set.

[0109] 12. The system of embodiment 11, wherein the update algorithm comprises a fast interval recalculation operation and a slow interval recalculation operation.

[0110] 13. A system according to embodiment 12, wherein the fast interval recalculation operation includes (i) calculating a first association set using a subset of data sampled from the third sensor data set and (ii) combining the first association set with a second association set calculated to generate the second transformation in (b).

[0111] 14. The system of embodiment 13, wherein the fast interval recalculation operation further comprises calculating a point cloud using the combined first association set and second association set.

[0112] 15. A system according to embodiment 12, wherein the slow interval recalculation operation includes a nearest neighbor algorithm.

[0113] 16. The system of embodiment 12, wherein the slow interval recalculation operation comprises updating the second transform using only the third sensor data set.

[0114] 17. The system of embodiment 11, wherein the position sensor is an electromagnetic sensor.

[0115] 18. The system of embodiment 11, wherein the coordinate system of the model representing the anatomical luminal network is generated using a preoperative imaging system.

[0116] 19. The system of embodiment 11, wherein the robotic endoscopic instrument comprises a disposable catheter assembly.

[0117] Although preferred embodiments of the present invention have been shown and described herein, it is readily understood by those skilled in the art that these embodiments are provided as examples only. Without departing from the present invention, those skilled in the art will now appreciate that many variations, changes and substitutions may be employed in implementing the present invention. It is understood that various alternatives to the embodiments of the present invention described herein may be employed. The appended claims are intended to define the scope of the present invention, and methods and structures within the scope of these claims and their equivalents are thus covered.

Claims

1. A method for navigating a robotic endoscopic instrument, comprising: (a) generating a first transformation between an orientation of the robotic endoscopic instrument and an orientation of the position sensor based at least in part on a first sensor data set acquired using a position sensor; (b) generating a second transformation between a coordinate system of the robotic endoscopic instrument and a coordinate system of a model representing an anatomical luminal network based at least in part on the first transformation and a second sensor data set; and (c) updating the second transform using an update algorithm based at least in part on a third sensor data set. 2 . The method of claim 1 , wherein the update algorithm comprises a fast interval recalculation operation and a slow interval recalculation operation.

3. The method of claim 2, wherein the fast interval recalculation operation comprises (i) computing a first association set using a subset of data sampled from the third sensor data set and (ii) combining the first association set with a second association set computed to generate the second transform in (b). The method of claim 3 , further comprising computing a point cloud using the combined first and second association sets. The method of claim 2 , wherein the slow interval recalculation operation comprises a nearest neighbor algorithm. The method of claim 2 , wherein the slow interval recalculation operation comprises updating the second transform using only the third sensor data set. The method of claim 1 , wherein the position sensor is an electromagnetic sensor.

8. The method of claim 1, wherein the coordinate system of the model representing the anatomical luminal network is generated using a preoperative imaging system.

9. The method of claim 1, wherein the robotic endoscopic instrument comprises a disposable catheter assembly.

10. A system for navigating a robotic endoscopic instrument, comprising: a position sensor located at a distal end of the robotic endoscopic instrument; and one or more processors in communication with the position sensor and the robotic endoscopic instrument and configured to execute a set of instructions to cause the system to: (a) generating a first transformation between an orientation of the robotic endoscopic instrument and an orientation of the position sensor based at least in part on a first sensor data set acquired using the position sensor; (b) generating a second transformation between a coordinate system of the robotic endoscopic instrument and a coordinate system of a model representing an anatomical luminal network based at least in part on the first transformation and a second sensor data set; and (c) updating the second transform using an update algorithm based at least in part on a third sensor data set.

Images