System and method for robotic bronchoscopy exploration

Abstract

A method for automatic registration for a robotic endoscope device is provided. The method comprises: (a) generating a first transformation between the orientation of the robotic endoscope device and the orientation of the position sensor based at least partially on a first set of sensor data collected using a position sensor; (b) generating a second transformation between the coordinate frame of the robotic endoscope device and the coordinate frame of a model representing an anatomical lumen network based at least partially on the first transformation and a second set of sensor data; and (c) updating the second transformation using an update algorithm based at least partially on a third set of sensor data.

Description

Technology Field

[0001] reference

[0002] This application claims priority to U.S. Provisional Application No. 62 / 950,740, filed December 19, 2019 and U.S. Provisional Application No. 63 / 091,283, filed October 13, 2020, the full contents of each of these provisional applications are incorporated herein by reference. Background Technology

[0003] Early diagnosis is crucial for lung cancer. The 5-year survival rate for lung cancer is approximately 18%, which is significantly lower than that of the three most prevalent cancers: breast (90%), colorectal (65%), and prostate (99%). In 2018, a total of 142,000 people were recorded as having died from lung cancer.

[0004] In general, the typical process for lung cancer diagnosis and surgical treatment can vary significantly depending on the technology used by healthcare providers, clinical protocols, and the clinical site. Inconsistent processes can not only delay cancer diagnosis but also increase costs imposed on patients and the healthcare system.

[0005] These medical procedures, such as endoscopic examinations (e.g., bronchoscopy), may include a step of accessing the inside of a patient's lumen (e.g., airway) to visualize the interior for diagnostic and / or therapeutic purposes. During the procedure, a flexible tubular instrument, such as an endoscope, may be inserted into the patient's body, and the device may pass the endoscope through a tissue site identified for diagnosis and / or treatment. means of solving the problem

[0006] Endoscopy encompasses a vast range of use cases for the diagnosis and treatment of various medical conditions, such as the early diagnosis and treatment of lung cancer. Endoscopic navigation systems may utilize different sensing modalities modeled through adaptively adjusted probabilities, for example (e.g., camera imaging data, electromagnetic (EM) position data, robotic position data, etc.). The navigation approach may depend on an initial estimate of where the endoscope tip is relative to the airway to initiate tip tracking. Some endoscopic techniques may involve guided navigation using a three-dimensional (3D) model of the patient's anatomical structure, EM fields, and position sensors. Prior to the procedure, the precise alignment (e.g., registration) between the virtual space of the 3D model and the physical space of the patient's anatomical structure represented by the 3D model and EM fields may not be known. Therefore, until registration information is generated, the position of the endoscope within the patient's anatomical structure cannot be accurately mapped to the corresponding position within the 3D model.

[0007] Inaccurate registration information can make bronchoscopy navigation difficult. In particular, current registration procedures are time-consuming and complex, or can generate inconsistent results influenced by human input. Current registration procedures may involve generating initial estimated registration information and then modifying or updating this initial information during the process. For example, current registration methods may compensate for changes in the navigation system by sampling from real-time sensor data. However, as the device operates over long durations and more incoming data is collected by the system, computation time increases rapidly. Furthermore, current registration procedures lack satisfactory online update capabilities, which may compromise registration accuracy. For instance, current registration procedures may be unable to adapt to local changes, potentially leading to inaccurate registration algorithms. For example, sampled datasets are typically used to update registration algorithms (e.g., transformations) overall, which do not accurately reflect local changes (e.g., due to small local mechanical deviations). In other examples, current registration algorithms may experience reduced transformation accuracy when the endoscope does not travel along a specific path (e.g., taken by cutting the centerline of the airway).

[0008] The present invention recognizes the need for minimally invasive systems that allow surgical procedures or diagnostic surgeries to be performed with improved reliability and cost-effectiveness. The present invention also recognizes the need for improved registration algorithms that can increase registration accuracy while reducing registration time. The present disclosure provides systems and methods that allow for standardized early lung cancer diagnosis and treatment at reduced cost. The present disclosure provides accessible and more cost-effective methods and systems for the early diagnosis and treatment of cancer. In some embodiments of the present invention, at least a portion of the robotic bronchoscopy system is disposable. For example, the catheter portion may be designed to be disposable at a low cost while preserving surgical capability and functionality. Furthermore, the provided robotic bronchoscopy system is designed to have the ability to access hard-to-reach tissues, such as the bronchi and lungs, without introducing additional costs.

[0009] Adaptive search algorithms may be able to identify registrations or mappings between the coordinate frame of a 3D model (e.g., the coordinate frame of a CT scanner used to generate the model) and the coordinate frame of an EM field (e.g., an EM field generator) with on-the-fly update capabilities.

[0010] In one embodiment, a method for navigating a robotic endoscope device is provided. The method comprises: (a) generating a first transformation between the orientation of the robotic endoscope device and the orientation of a position sensor based at least partially on a first set of sensor data collected using a position sensor; (b) generating a second transformation between the coordinate frame of the robotic endoscope device and the coordinate frame of a model representing an anatomical lumen network based at least partially on the first transformation and a second set of sensor data; and (c) updating the second transformation using an update algorithm based at least partially on a third set of sensor data.

[0011] In some embodiments, the update algorithm includes a fast interval recalculation operation and a slow interval recalculation operation. In some cases, the fast interval recalculation operation includes (i) calculating a first set of associations using a subset of data sampled from a third set of sensor data, and (ii) combining the first set of associations with a second set of associations calculated to generate a second transformation of (b). For example, the method further includes the step of calculating a point cloud using the combined first set of associations and the second set of associations. In some cases, the slow interval recalculation operation includes a nearest neighbor algorithm. In some cases, the slow interval recalculation operation includes the step of updating the second transformation using only the third set of sensor data.

[0012] In some embodiments, the position sensor is an electromagnetic sensor. In some embodiments, a coordinate frame of a model representing an anatomical lumen network is generated using a preoperative imaging system.

[0013] In some embodiments, the robotic endoscope includes a disposable catheter assembly. In some embodiments, a position sensor is located at the distal tip of the robotic endoscope.

[0014] In another embodiment, a system for navigating a robotic endoscope is provided. The system comprises: a position sensor located at a distal end of the robotic endoscope; and one or more processors configured to execute a set of instructions to update the second transformation using an update algorithm based at least partially on the first set of sensor data collected using the position sensor, wherein the system comprises: (a) generating a first transformation between the orientation of the robotic endoscope and the orientation of the position sensor based at least partially on the first transformation and the second set of sensor data; (b) generating a second transformation between the coordinate frame of the robotic endoscope and the coordinate frame of a model representing an anatomical lumen network based at least partially on the first transformation and the second set of sensor data; and (c) one or more processors configured to execute a set of instructions to update the second transformation using an update algorithm based at least partially on the third set of sensor data.

[0015] In some embodiments, the update algorithm includes a fast interval recalculation operation and a slow interval recalculation operation. In some cases, the fast interval recalculation operation includes (i) calculating associations of a first set using data from a subset sampled from a third set of sensor data, and (ii) combining the associations of the first set with the associations of a second set calculated to generate the second transformation of (b). For example, the fast interval recalculation operation further includes the step of calculating a point cloud using the combined associations of the first set and the associations of the second set. In some cases, the slow interval recalculation operation includes a nearest neighbor algorithm. In some cases, the slow interval recalculation operation includes the step of updating the second transformation using only the third set of sensor data.

[0016] In some embodiments, the position sensor is an electromagnetic sensor. In some embodiments, a coordinate frame of a model representing an anatomical lumen network is generated using a preoperative imaging system. In some embodiments, the robotic endoscope device includes a disposable catheter assembly.

[0017] According to some aspects of the present disclosure, a robotic endoscope device is provided. The device may include a disposable slender member comprising a proximal end and a distal end, wherein the proximal end is removablely attached to a robotic arm. The distal end comprises a plurality of pull wires, the pull wires being integral with the wall of the slender member. The slender member may also be referred to as a bronchoscope or catheter, which may be interchangeably used throughout the specification.

[0018] In another aspect of the present disclosure, an improved registration algorithm or search method is provided. The registration algorithm allows for automatic registration with minimal user interaction and, advantageously, may allow for online registration updates that increase registration accuracy. It should be noted that the provided robotic system and / or registration algorithm may be used in various minimally invasive surgical procedures involving various types of tissue, including heart, bladder, and lung tissue, etc.

[0019] Further aspects and advantages of the present disclosure will be readily apparent to those skilled in the art from the following detailed description, in which only exemplary embodiments of the present disclosure are illustrated and described. Other and different embodiments of the present disclosure are possible as they are realized, and variations of some details thereof are possible in various obvious respects, all of which do not depart from the present disclosure. Accordingly, the drawings and description should be regarded as illustrative rather than limiting.

[0020] Reference Citation

[0021] All publications, patents, and patent applications mentioned in this specification are incorporated herein by reference to the same extent that each individual publication, patent, or patent application was specifically and individually indicated and cited by reference. To the extent that a publication and patent or patent application cited by reference conflicts with the disclosures contained herein, this specification is intended to replace and / or prioritize any such conflicting material. Brief explanation of the drawing

[0022] Novel features of the present invention are specifically set forth in the appended claims. The features and advantages of the present invention will be better understood by referring to the following detailed description and the accompanying drawings (also referred to herein as "Figure" and "FIG."), which specify exemplary embodiments in which the principles of the present invention are utilized. Figure 1 shows an exemplary workflow of standardized lung cancer diagnosis possible by the robotic bronchoscopy system described herein. FIG. 2a shows an example of a robotic bronchoscopy examination system according to some embodiments of the present invention. FIG. 2b shows different scenes of an exemplary robotic bronchoscopy system according to some embodiments of the present invention. Figure 3 shows an example of a fluorescence imaging (tomography) system. Figure 4 shows a C-arm fluoroscopic (tomometric synthesis) imaging system in different (rotated) positions while capturing images of an object. Figure 5 shows an example of a user interface for visualizing a virtual airway with the optimal path, the position of the catheter tip, and the lesion location superimposed. Figure 6 shows an exemplary portable robotic cone beam CT. FIG. 7 schematically shows an example of an automatic registration method according to some embodiments of the present invention. Figure 8 shows an example of a customized iterative nearest point (ICP) algorithm. Figure 9 shows an example of a classification result. Figure 10 shows an example of a data structure (spatial data structure). The specialized data structure may be a kd tree or an octree data structure. Specific details for implementing the invention

[0023] Although preferred embodiments of the present invention have been illustrated and described herein, it will be apparent to those skilled in the art that such embodiments are provided merely as examples. Various modifications, changes, and substitutions may be made by those skilled in the art without departing from the invention. It should be understood that various alternatives to the embodiments of the invention described herein may be adopted.

[0024] While exemplary embodiments may relate primarily to bronchoscopes, those skilled in the art will understand that this is not intended to be limiting, and that the device described herein may also be used in other therapeutic or diagnostic procedures and in anatomical areas of a patient's body such as the digestive system, including but not limited to the esophagus, liver, stomach, colon, and urinary tract, or the respiratory system, including but not limited to the bronchi and lungs, and various others. The registration method / algorithm of the present invention may be used to align the coordinate frame of a medical device with the coordinate frame of a 3D model (e.g., an airway model generated by a CT scanner or the patient's coordinate frame), regardless of the type of device or operation to be performed.

[0025] The embodiments disclosed herein may be combined in one or more ways to provide improved diagnosis and treatment to patients. The disclosed embodiments may be combined with existing methods and devices to provide improved treatment, such as, for example, in combination with known methods for lung diagnosis, surgery, and surgery on other tissues and organs. Any one or more of the structures and steps described herein may be combined with any one or more additional structures and steps of the methods and devices described herein, and it should be understood that the drawings and supporting notes provide an explanation according to the embodiments.

[0026] Although the treatment plan and definition of the diagnostic or surgical procedure described herein are presented in the context of lung diagnosis or surgery, the method and device described herein may be used to treat any tissue of the body and any organ and blood vessel of the body, such as soft tissues such as the brain, heart, lungs, intestines, eyes, skin, kidneys, liver, pancreas, stomach, uterus, ovaries, testes, bladder, ears, nose, mouth, and bone marrow, as well as hard biological tissues such as adipose tissue, muscle, glandular and mucosal tissues, spinal and nervous tissues, cartilage, teeth, bone, etc., as well as body lumens and passages such as sinuses, ureters, colon, esophagus, lung passages, blood vessels, and throat.

[0027] Whenever the terms “at least,” “greater than,” or “greater than or equal to” come before the first numeric value in a series of two or more numeric values, the terms “at least,” “greater than,” or “greater than or equal to” apply to each numeric value in the series of numeric values. For example, being greater than or equal to 1, 2, or 3 is equivalent to being greater than or equal to 1, greater than or equal to 2, or greater than or equal to 3.

[0028] Whenever the terms “not greater than,” “less than,” or “less than or equal to” come before the first numeric value in a series of two or more numeric values, the terms “not greater than,” “less than,” or “less than or equal to” apply to each numeric value in the series of numeric values. For example, being less than or equal to 3, 2, or 1 is equivalent to being less than or equal to 3, less than or equal to 2, or less than or equal to 1.

[0029] As used herein, a processor comprises one or more processors, e.g., a single processor, or, e.g., multiple processors of a distributed processing system. A control unit or processor as described herein generally comprises a tangible medium storing instructions for implementing steps of a process, and the processor may comprise, e.g., a central processing unit, programmable array logic, gate array logic, or field programmable gate array. In some cases, one or more processors may be a programmable processor (e.g., a central processing unit (CPU) or a microcontroller), a digital signal processor (DSP), a field programmable gate array (FPGA), and / or one or more advanced RISC machine (ARM) processors. In some cases, one or more processors may be operably coupled to a non-transient computer-readable medium. The non-transient computer-readable medium may store logic, code, and / or program instructions executable by one or more processor units to perform one or more steps. A non-transient computer-readable medium may include one or more memory units (e.g., removable media or external storage such as an SD card or random access memory (RAM)). One or more methods or operations disclosed herein may be implemented by hardware components, such as an ASIC, a special-purpose computer, or a general-purpose computer, or by a combination of hardware and software.

[0030] As used herein, the terms distal and proximal may generally refer to a position referenced from the device and may be as opposed to an anatomical reference. For example, the distal position of a bronchoscope or catheter may correspond to the proximal position of the patient's slender member, and the proximal position of a bronchoscope or catheter may correspond to the distal position of the patient's slender member.

[0031] Systems as described herein include slender parts or slender members, such as catheters. The terms “slender member,” “catheter,” and “bronchoscope” are used interchangeably throughout the specification unless the context otherwise indicates. A slender member may be placed directly in a body cavity or body cavity. In some embodiments, the system may further include a support device, such as a robotic manipulator (e.g., a robotic arm), for driving, supporting, positioning, or controlling the movement and / or operation of the slender member. Alternatively, or additionally, the support device may be a hand-held device or other control device that may or may not include a robotic system. In some embodiments, the system may further include peripheral devices and subsystems, such as an imaging system, that support and / or facilitate the exploration of the slender member to a target site in the subject’s body. Such exploration may require a registration process, which will be described later in this specification.

[0032] In some embodiments of the present disclosure, a robotic bronchoscopy system is provided to perform surgical procedures or diagnoses with improved performance at a low cost. For example, the robotic bronchoscopy system may include a maneuverable catheter that may be entirely disposable. This may advantageously reduce sterilization requirements, which can be costly or difficult to operate, but sterilization or hygienic treatment may not be effective. Furthermore, one problem with the bronchoscope is reaching the upper lobes of the lungs while navigating the airway. In some cases, the provided robotic bronchoscopy system may be designed to have the ability to navigate through airways with small curvatures in an autonomous or semi-autonomous manner. Autonomous or semi-autonomous navigation may require a registration process as described herein later. Alternatively, the robotic bronchoscopy system may be navigated by an operator via a control system with visual guidance capabilities.

[0033] The typical process for lung cancer diagnosis and surgical treatment can vary significantly depending on the technology used by healthcare providers, clinical protocols, and the clinical site. Inconsistent processes may lead to delays in the early diagnosis of lung cancer, high costs for healthcare systems and patients to diagnose and treat lung cancer, and a high risk of clinical and procedural complications. The robotic bronchoscopy system provided may allow for standardized early lung cancer diagnosis and treatment. FIG. 1 shows an exemplary workflow (100) of standardized lung cancer diagnosis made possible by the robotic bronchoscopy system described herein.

[0034] As exemplified in Fig. 1, preoperative imaging may be performed to identify lesions. Any suitable imaging technique, such as magnetic resonance (MR), positron emission tomography (PET), X-ray, computed tomography (CT), and ultrasound, may be used to identify lesions or regions of interest. For example, a patient suspected of having lung cancer may undergo a preoperative CT scan, and suspected lung nodules may be identified in the CT image. The preoperative imaging process may be performed prior to bronchoscopy.

[0035] Next, CT images may be analyzed to generate a map that guides the navigation of the robotic bronchoscope during bronchoscopy. For example, a lesion or region of interest (ROI) may be segmented in the image. When imaging of the lungs is performed, a pathway or path to the lesion may be highlighted in the reconstructed image to plan the navigation path. The reconstructed image may guide the navigation of the robotic bronchoscope to the target tissue or target site. In some cases, the navigation path may be pre-planned using 3D image data. For example, a catheter may be advanced toward the target site under the robotic control of the robotic bronchoscope system. The catheter may be maneuvered or advanced toward the target site in a manual, autonomous, or semi-autonomous manner. In one example, the movement of the catheter may be image-guided so that the insertion and / or maneuvering direction may be automatically controlled.

[0036] In some cases, the location of the lesion in preoperative imaging may be inaccurate due to patient movement or physical deviation. In such cases, the location of the lesion may be verified prior to the surgical procedure (e.g., biopsy or treatment). The precise location of the lesion may also be verified or updated with the assistance of a robotic bronchoscopy system. For example, the bronchoscopy system may provide an interface to imaging methods, such as fluoroscopy, to provide real-time in vivo imaging of the target area and surrounding regions to locate the lesion. In one example, a C-arm or O-arm fluoroscopy imaging system may be used to generate tomographic composite images to verify or update the location of the lesion. During surgical procedures such as a biopsy, various surgical tools, such as biopsy tools, brushes, and forceps, may be inserted into the working channel of the catheter, either manually or automatically, to perform the biopsy or other surgical procedures.

[0037] Next, a sample of the lesion or any other target tissue may be acquired by a tool inserted through the catheter's working channel. This system allows camera visualization to be maintained throughout the procedure, including during the insertion of the tool through the working channel. In some cases, the tissue may be rapidly evaluated at the site of sampling by a rapid on-site evaluation process to determine whether a repeat of the tissue sampling is necessary or to decide on further action. In some cases, the rapid on-site evaluation process may also provide a rapid analysis of the tissue sample to determine subsequent surgical treatment. For example, if the tissue sample is determined to be malignant as a result of the rapid on-site evaluation process, a manual or robotic treatment device may be inserted through the robotic bronchoscope's working channel to perform endotracheal intubation for lung cancer. This has the advantage of allowing diagnosis and treatment to be performed in a single session, thereby providing targeted, pain-free, and rapid treatment for early-stage lung cancer.

[0038] FIGS. 2a and 2b illustrate examples of a robotic bronchoscopy system (200, 230) according to some embodiments of the present invention. As illustrated in FIG. 2a, the robotic bronchoscopy system (200) may include a maneuverable catheter assembly (220) and a robotic support system (210) for supporting or carrying the maneuverable catheter assembly. The maneuverable catheter assembly may be a bronchoscope. In some embodiments, the maneuverable catheter assembly may be a disposable robotic bronchoscope. In some embodiments, the robotic bronchoscopy system (200) may include a device drive mechanism (213) attached to an arm of the robotic support system. The device drive mechanism may be provided by any suitable control unit device (e.g., a handheld control unit) which may or may not include the robotic system. The device drive mechanism may provide mechanical and electrical interfaces to the maneuverable catheter assembly (220). A mechanical interface may allow the maneuverable catheter assembly (220) to be disengaged from the device drive mechanism. For example, the handle portion of the maneuverable catheter assembly may be attached to the device drive mechanism via a quick attachment / removal means such as a magnet, a spring-loaded level, etc. In some cases, the maneuverable catheter assembly may be manually attached to or removed from the device drive mechanism without the use of tools.

[0039] A maneuverable catheter assembly (220) It may include a handle portion (223) that may include components configured to process video data, provide power, or establish communication with other external devices. For example, the handle portion (223) It may include circuits and communication elements that enable electrical communication between the maneuverable catheter assembly (220) and the device driving mechanism (213), and any other external system or device. In other examples, the handle portion (223) may include circuit elements such as a power source for providing power to the electronic devices of the endoscope (e.g., camera and LED light). In some cases, the handle portion may electrically communicate with the device driving mechanism (213) through an electrical interface (e.g., printed circuit board) so that image / video data and / or sensor data may be received by the communication module of the device driving mechanism and transmitted to another external device / system. Alternatively or additionally, the device driving mechanism (213) It may provide only a mechanical interface. The handle portion may also electrically communicate with a modular wireless communication device or any other user device (e.g., a portable / handheld device or a control unit) for transmitting sensor data and / or receiving control signals. Details regarding the handle portion will be described herein later.

[0040] The maneuverable catheter assembly (220) may include a flexible slender member (211) coupled to a handle portion. In some embodiments, the flexible slender member may include a shaft, a maneuverable tip, and a maneuverable section. The maneuverable catheter assembly may be a disposable robotic bronchoscope. In some cases, only the slender member may be disposable. In some cases, at least a portion of the slender member (e.g., shaft, maneuverable tip, etc.) may be disposable. In some cases, the entire maneuverable catheter assembly (220), including the handle portion and the slender member, may be disposable. The flexible slender member and the handle portion are designed so that the entire maneuverable catheter assembly can be disposed of at low cost. Details regarding the flexible slender member and the maneuverable catheter assembly are described herein later.

[0041] In some embodiments, the provided bronchoscope system may also include a user interface. As illustrated in the exemplary system (230), the bronchoscope system may include a treatment interface module (231) (on the user console side) and / or a treatment control module (233) (on the patient and robot side). The treatment interface module may allow an operator or user to interact with the bronchoscope during a surgical procedure. In some embodiments, the treatment control module (233) may be a handheld controller. In some cases, the treatment control module may include a dedicated user input device and one or more additional elements that are removablely coupled to an existing user device to improve the user input experience. For example, a physical trackball or roller may replace or supplement the function of at least one of the virtual graphic elements displayed in the graphical user interface (GUI) (e.g., navigation arrows displayed on a touchpad) by providing a function similar to the graphic element to be replaced. Examples of user devices may include, but are not limited to, mobile devices, smartphones / phones, tablets, personal information terminals (PDAs), laptops or notebook computers, desktop computers, media content players, etc. Details regarding user interface devices and user consoles will be described later in this document.

[0042] FIG. 2b shows different scenes of the bronchoscope system. The user console (231) may be mounted on the robot support system (210). Alternatively, or additionally, the user console or a part of the user console (e.g., a treatment interface module) may be mounted on a separate mobile cart.

[0043] Robot in-house platform

[0044] In one embodiment, a robotic in-tube platform is provided. In some cases, the robotic in-tube platform may be a bronchoscope platform. The platform may be configured to perform one or more operations consistent with the method described in FIG. 1. FIGS. 3 through 7 show various examples of a robotic in-tube platform and its components or subsystems according to some embodiments of the present invention. In some embodiments, the platform may include a robotic bronchoscope system and one or more subsystems that can be used in combination with the robotic bronchoscope system of the present disclosure.

[0045] In some embodiments, one or more subsystems may include an imaging system, such as a fluoroscopic (tomographic) imaging system, for providing real-time imaging of a target area (e.g., including a lesion). FIG. 3 shows an example of a fluoroscopic (tomographic) imaging system (300). For example, the fluoroscopic (tomographic) imaging system may perform accurate lesion location tracking or verification before or during a surgical procedure as described in FIG. 1. In some cases, the lesion location may be tracked based on location data for the fluoroscopic (tomographic) imaging system / station (e.g., C arm) and image data captured by the fluoroscopic (tomographic) imaging system. The lesion location may be registered using the coordinate frame of the robotic bronchoscopy system. The position or movement of a fluorescence imaging (tomography) system may be measured using any suitable motion / position sensor (310), such as an inertial measurement unit (IMU), one or more gyroscopes, a velocity sensor, an accelerometer, a magnetometer, a position sensor (e.g., a satellite positioning system (GPS) sensor), a field of view sensor (e.g., an imaging device capable of detecting visible light, infrared or ultraviolet light, such as a camera), a proximity or range sensor (e.g., an ultrasonic sensor, a lidar, a non-transparent or depth camera), an elevation sensor, an attitude sensor (e.g., a compass), and / or a field sensor (e.g., a magnetometer, an electromagnetic sensor, a radio sensor). One or more sensors for tracking the movement and position of a fluorescence imaging (tomography) station may be placed at the imaging station or located at a distance from the imaging station, such as a wall-mounted camera (320). FIG. 4 shows a C-arm fluorescence imaging (tomography) system in different (rotational) positions while capturing images of an object. As mentioned above, various postures may be captured by one or more sensors.

[0046] In some embodiments, the location of the lesion may be segmented from image data captured by a fluoroscopic (tomometric composite) imaging system with the help of a signal processing unit (330). One or more processors of the signal processing unit may be configured to further superimpose the treatment location (e.g., lesion) onto the real-time fluoroscopic image / video. For example, the processing unit may be configured to generate an augmentation layer containing augmentation information, such as the location of the treatment location or the location of the target site. In some cases, the augmentation layer may include a graphic marker indicating a path to this target site. The augmentation layer may be a substantially transparent image layer containing one or more graphic elements (e.g., a box, an arrow, etc.). The augmentation layer may be superimposed on an optical view of an optical image or video stream captured by the fluoroscopic (tomometric composite) imaging system and / or displayed on a display device. Because the augmentation layer is transparent, the user is able to view the optical image along with the graphic elements superimposed on it. In some cases, the segmented lesion image and the optimal path for the search of the slender member to reach the lesion may be superimposed on the real-time tomographic composite image. This allows the operator or user to visually confirm not only the precise location of the lesion but also the planned bronchoscope movement path. In some cases, the segmented and reconstructed image provided prior to the operation of the system described herein (e.g., CT image described elsewhere) may be superimposed on the real-time image.

[0047] In some embodiments, One or more subsystems of the platform may include a navigation and localization subsystem. The navigation and localization subsystem may be configured to construct a virtual airway model based on preoperative images (e.g., preoperative CT images or tomographic synthesis). The navigation and localization subsystem may be configured to identify segmented lesion locations in a 3D rendered airway model, and based on the location of the lesion, the navigation and localization subsystem may generate an optimal path from the main bronchus to the lesion with a recommended approach angle toward the lesion to perform a surgical procedure (e.g., biopsy).

[0048] In the registration phase, prior to driving the bronchoscope to the target site, the system may align a rendered virtual view of the airway to the patient's airway. Image registration may consist of a single registration phase or a combination of a single registration phase and real-time detection updates to the registration information. The registration process may include a step of finding a transformation that aligns the object (e.g., airway model, anatomical region) between different coordinate systems (e.g., EM sensor coordinates based on preoperative CT imaging and patient 3D model coordinates). Details regarding registration are described herein later.

[0049] Once registered, all airways may be aligned with the pre-operative rendered airways. While the robotic bronchoscope is driven toward the target site, the position of the bronchoscope within the airway may be tracked and displayed. In some cases, a positioning sensor may be used to track the bronchoscope's position relative to the airway. Other types of sensors (e.g., cameras) may also be used in place of or in conjunction with positioning sensors using sensor fusion technology. Positioning sensors, such as electromagnetic (EM) sensors, may be embedded in the distal tip of the catheter, or an EM field generator may be positioned next to the patient's torso during the procedure. The EM field generator may specify the position of the EM sensor in 3D space, or specify the position and orientation of the EM sensor in 5D or 6D space. This may also provide visual guidance to the operator when driving the bronchoscope toward the target site.

[0050] In real-time EM tracking, an EM sensor comprising one or more sensor coils embedded in a medical device (e.g., the tip of an endoscopic tool) at one or more positions and orientations measures changes in the EM field generated by one or more static EM field generators located close to the patient. Position information detected by the EM sensor is stored as EM data. An EM field generator (or transmitter) may be placed close to the patient and generates a low-intensity magnetic field that can be detected by the embedded sensor. The magnetic field induces a small current in the sensor coil of the EM sensor, and this current may be analyzed to determine the distance and angle between the EM sensor and the EM field generator. These distances and orientations may be registered in the patient's anatomical structure (e.g., a 3D model) during surgery to determine a registration transformation that aligns a single location in the coordinate system with the location of the preoperative model of the patient's anatomical structure.

[0051] FIG. 5 shows an example of a user interface for visualizing a virtual airway (509) in which the optimal path (503), the position of the catheter tip (501), and the lesion location (505) are superimposed. In this example, the position of the catheter tip is displayed in real time on the virtual airway model (509) to provide visual guidance. As illustrated in the example of FIG. 5, the optimal path (503) may be displayed and superimposed on the virtual airway model during the operation of the robotic bronchoscope. As previously mentioned, the virtual airway model may be constructed based on real-time fluoroscopic images / videos (and position data of the imaging system). In some cases, the real-time fluoroscopic image / video view (507) may also be displayed on the graphical user interface. In some cases, the user may also be allowed to access a camera view or image / video (511) captured in real time by the bronchoscope.

[0052] In some embodiments, One or more subsystems of the platform may include one or more sub-treatment systems, such as manual devices or robotic devices (e.g., biopsy needles, biopsy forceps, biopsy brushes) and / or manual treatment devices or robotic treatment devices (e.g., RF ablation devices, cryotherapy devices, microwave devices, etc.).

[0053] Conventional cone-beam CT machines may be equipped with transmitter and receiver panels on the same mechanical structure in a C-shape or O-shape. The size of the cone-beam CT can be increased by the connection between the transmitter and receiver panels. This extra-large design limits use cases and occupies a significant amount of space in somewhat cramped operating rooms.

[0054] The present invention describes a design that separates the mechanical connection between the transmitter and the receiver panel. Fig. 6 shows an example of a portable robot cone beam CT. The transmitter and receiver panel can be mounted separately on two individual robot arms, as shown in the example of Fig. 6. When in use, the two robots can move in the same coordinate system. A control algorithm may also ensure that the two robots move in synchronized motion.

[0055] Additionally, for the patient's gating movements, i.e., breathing, additional external sensors (i.e., IMU, EM, or imaging sensors) may be added to track the patient's movements. Changes in the patient's position can be tracked using sensors such as IMUs, EMs, or imaging sensors. Sensory signals can be used to issue commands to two robotic arms. In some cases, one or both of the robotic arms may move to track the patient's movements, which essentially ensures that the transmitter and receiver are fixed on the patient's movement relative to a region of interest (ROI) during tracking. The ROI may include a target area or target location that can be determined automatically by the system or manually by a physician. Tracking may also be performed using other mechanisms, such as, but not limited to, external cameras and one or more trackers on the patient's body.

[0056] Those skilled in the art will understand that cone beam (CT) is a non-limiting example. The design described herein may also be used in other imaging modes, such as fluoroscopy machines, classical CT machines, and MRI machines.

[0057] Automatic registration algorithm

[0058] The present disclosure provides an improved automatic registration algorithm capable of increasing registration accuracy while reducing registration time. The provided automatic registration algorithm can advantageously set up conversions or registrations that are improved over existing methods while minimizing user interaction. Additionally, the provided automatic registration algorithm may allow the algorithm to be updated immediately to adapt to real-time conditions that increase registration accuracy (e.g., automatic registration updates based on real-time detected registration errors) as the device operates inside the subject's body.

[0059] In some embodiments, the automatic registration algorithm may include three steps, or may be implemented in three steps. FIG. 7 schematically illustrates an example of an automatic registration method (700) according to some embodiments of the present invention. The automatic registration method may include one or more algorithms for updating registration information in real time without user input. The automatic registration algorithm may also include one or more algorithms for identifying an optimal registration / transformation matrix and increasing registration accuracy by utilizing time information.

[0060] As described above, an EM sensor may be coupled to the distal end of an endoscope to track the position within the EM field. The EM field is fixed to the EM field generator, and the coordinate frame of a 3D model of the lumen network (e.g., CT space) may be mapped to the coordinate frame of the EM field. As illustrated in FIG. 7, the automatic registration method (700) may include a first step, which may be an initial data collection phase (701). The initial data collection phase may include a step of reconstructing the distal end of the catheter and performing an initial translation transformation.

[0061] The initial data collection phase (701) may register the EM sensor orientation as a distal tip orientation. This may establish an association between the robotic endoscope device space and the position sensor space. For example, an electromagnetic coil located on the distal end may be used in conjunction with an electromagnetic tracking system to detect the position and orientation of the distal tip of the catheter while the distal tip of the catheter is positioned within an anatomical system (e.g., an anatomical lumen network). In some embodiments, the coil may be positioned at an angle to provide sensitivity of the electromagnetic field along different axes, thereby providing the disclosed search system with the ability to measure a total of 6 degrees of freedom (3 positional degrees of freedom and 3 angular degrees of freedom). The orientation / position of the EM field generator may be unknown, and the EM sensor orientation (e.g., the Z-axis of the EM coordinate frame) may be registered to the distal tip orientation (e.g., the Z-axis of the catheter tip) using sensor data collected through the initial data collection phase.

[0062] During the first phase, sensor data (e.g., EM sensor data) may be collected to register the z-axis of the magnetic field generator to the z-axis of the catheter tip. This may allow flexibility in positioning the magnetic field generator without needing to know the position and orientation in advance. The sensor data may include stream data collected when the catheter is driven in a known direction. The stream data may include, but is not limited to, various types of data such as time-series data, like spatiotemporal point measurements generated by the EM sensor. In some cases, time-series data may be collected when the catheter is driven in a known direction, which may or may not be inside the patient's body. For example, EM sensor data points may be collected when the distal tip is placed inside an object, such as a bronchus or airway, and moves along a known direction. In some cases, the z-axis of the catheter tip and the EM orientation may be corrected during assembly or manufacturing. In some cases, the z-axis of the catheter tip may be obtained from robotic data when driving the catheter tip in a known direction or with the help of CT imaging or other real-time imaging techniques. An initial translational transformation between the catheter tip and the EM field may also be obtained. These translational and rotational transformations may be continuously improved and modified during the first phase as more incoming sensor data is collected. In some cases, the translational transformations of the first phase may be updated and improved until a threshold is met. The threshold may be an accuracy threshold indicating that accuracy is satisfied. Alternatively, or additionally, updates may be performed continuously (e.g., errors may be calculated at time intervals), or updates may be stopped when tomographic image synthesis targeting is in progress.

[0063] In some cases, the collected sensor data may also provide a registration in the plane of translation transformation in CT space (e.g., tissue) and EM space (e.g., catheter). For example, a registration transformation between an EM field and a patient model (e.g., the coordinate frame of a luminal mesh model) may be identified. Such translation transformation in the second phase may include an initial translation transformation matrix that may be subsequently improved in the final phase as described herein.

[0064] A second phase (703) of the registration method may include a step of creating an association between EM space and a model (e.g., CT) space. During the second phase, an initial transformation matrix between EM space and CT space may be created based at least partially on real-time sensor data and movement information generated in the first phase. In some cases, data points (e.g., EM data) utilized in the second phase may be collected when the catheter is navigating inside the trachea and / or child branch along a predefined navigation path. The predefined navigation path may be defined in CT space. As described elsewhere in this invention, EM data may include information regarding orientation, position, and error data. A coordinate frame of CT space or an anatomical model may be created during a preoperative procedure or during surgery. These sensor data points establish a plane in space and may be used to establish a set of associations between EM space and CT space based on a set of directional distances (e.g., both orientation and length) traveled in EM space and CT space, respectively. The association set may also be generated based at least partially on time series data, such as time series data points generated by an EM sensor.

[0065] In some cases, a set of associations may be processed to identify the optimal transformation matrix. For example, to identify the optimal transformation matrix, the best fit of an EM curve (e.g., a point cloud in EM space) to a CT curve (e.g., a point cloud in CT space) set in 3D space may be determined. For example, the provided automatic registration method may employ a modified version of the iterative nearest point (ICP) algorithm using singular value decomposition and nearest neighbors to repeat calculations until an optimal solution is reached. It should be noted that other algorithms or variations suitable for identifying the optimal fit may be adopted.

[0066] Figure 8 shows an example of a custom iterative nearest neighbor (ICP) algorithm for generating an initial transformation between two data point clouds (e.g., an EM domain point cloud and a CT domain point cloud). The ICP is an iterative algorithm that alternates between the step of determining the nearest neighbors (NN) of all points of a feature based on the current transformation estimate and the step of updating the estimate based on the NN. An association solver may receive a computer-computed association set between two sets of data point clouds in CT space and EM space (e.g., Cn, Xn) and is executed to obtain the transformation that best matches the association set. As an example, the association set may be computed using the k-nearest neighbor (k-NN) algorithm. The best custom transformation may be computed using singular value decomposition during the error minimization step. Assumptions (e.g., proximity assumptions) may be modified to smoothly lock translation and rotation transformations into good initial values.

[0067] An example of a classification result is illustrated in Fig. 9. The provided method may further enhance the performance of the k-NN algorithm to reduce computation time by utilizing specialized data structures. Fig. 10 shows an example of a data structure (spatial data structure). The specialized data structure may be a kd tree or an octree data structure. A kd tree or octree data structure can allow for efficient implementation of NNS due to the regular partitioning of the search space and high branching factors, and coordinate queries are fast. Other data structures suitable for feature registration and storage efficiency may also be used.

[0068] Referring again to Fig. 8, the error minimization operation may include a step of calculating the best fitted transformation using singular value decomposition. For example, singular value decomposition may be applied to an association set to calculate the best transformation. The singular value decomposition method may also be used to calculate the solution to the total least squares minimization problem.

[0069] It should be noted that the algorithms mentioned above may be modified or replaced with other algorithms. For example, a coherent point drift algorithm may be employed to identify the best custom transformations based on probability rather than rigid point locations. The algorithm may also rotate the point cloud to traverse a stochastic optimization surface while expanding the point cloud (and / or transforming the point cloud using stochastic convergence) until the computed features, typically the center, align with other point clouds. Coherent point drift algorithms or other probability-based algorithms may also reduce the influence of noise on the dataset.

[0070] Referring again to FIG. 7, the third phase (705) of the automatic registration algorithm may include a step of updating the initial registration / transformation generated in the second phase. In some embodiments, the update method may include a weak real-time update and a strong real-time update for the initial transformation. As used interchangeably throughout the specification, the weak real-time update may also be referred to as a fast interval update, and the strong real-time update may also be referred to as a slow interval update. The update may be performed using real-time data without user intervention. In some cases, the update may utilize real-time EM sensor data collected from a large point cloud. In some cases, a subset of data points from the large point cloud may be selected / sampled to calculate the error and determine whether an update is needed based on the error.

[0071] In some cases, the update method may include weak real-time updates and strong real-time updates for the initial transformation. For example, the update method may include a fast interval improvement step and a slow interval recalculation step. In some cases, after each update, the error may be recalculated, and if the error decreases, the transformation matrix may be updated.

[0072] A rapid interval improvement task (e.g., rapid interval update) may include a step of randomly sampling a subset of data points from a large point cloud (e.g., data points collected in Phase 3). Next, the current transformation and nearest neighbor algorithm may be applied to the sampled set to generate a set of associations. This set of associations may be combined with the set of associations calculated in Phase 2 to generate a final point cloud. The final point cloud data may be fed into a modified ICP algorithm as described in Phase 2 to generate an updated transformation and thereby improve the current transformation.

[0073] The slow interval recalculation operation may include an update operation similar to the one described above, using only the nearest neighbor algorithm without utilizing initial transformation information or the association set generated in the second phase. For example, a set of associations may be calculated by applying the nearest neighbor algorithm to sensor data collected in the third phase, and the transformation matrix may be updated using only the association set without information from the second phase. In some cases, a subset of data may be randomly sampled from a large dataset. Alternatively, or additionally, a subset of data may be selected based on distance. For example, a distance-based time filter may sample points uniformly distributed along a time path and be used to select these points for the update calculation. The distance used for filtering may be based on CT scans and / or the patient's anatomical structure.

[0074] These slow-interval recalculations (i.e., strong updates) may be performed at larger time / point intervals because they may cause a "jump" to the next local minimum. In some cases, slow recalculations may be performed at fixed intervals (e.g., time intervals, a fixed number of data points, etc.). In some cases, slow recalculations may be triggered when the update error value begins to converge, signaling the reaching of a local or global minimum. As the initial transformation is continuously improved by weak updates (i.e., fast-interval recalculations) and through iterations of the ICP algorithm, a "jump" to the desired step or the correct direction can be guaranteed. It is advantageous to combine weak updates (e.g., fast-interval improvements) with strong updates, as well as to perform weak updates more frequently than strong updates, to guarantee a "jump" in the correct direction as the improvement step increases the distance to an incorrect local minimum while decreasing the distance to the global minimum.

[0075] Random sampling of data points during the update phase (e.g., rapid interval updates) can also advantageously prevent overfitting. Overfitting can occur due to noisy data and fitting data cylinders to a single line. Random sampling provides the ability to find and refine general solutions instead of jumping from one local minima to another, thereby allowing for smooth traversal of the error surface and potentially improving registration accuracy and stability. Random sampling of data points can also indirectly weight initial associations with estimated associations (calculated in rapid interval recalculations). Since high-accuracy data is computed in the second phase, and because the structure is complex and the gaps between surrounding airways are small, it is desirable to weight this calculation more than nearest neighbor calculations. Random data sampling during the update phase can further prevent getting stuck at local minima that could halt the entire registration process. Reaching a local minimum may be inevitable, primarily because even if an overall solution is found at iteration x, iteration x+1 may add data that causes the previous transformation to become a local minimum. Random sampling can advantageously facilitate iteration and error calculation.

[0076] In some embodiments, the automatic registration algorithm may also employ a preprocessing algorithm to improve performance by preprocessing the collected sensor data. In some cases, the Ramer-Douglas-Peucker algorithm may be used to preprocess large datasets of time-series data between two location points. The algorithm regressively finds points that preserve the shape of the entire dataset while simplifying the path. In some cases, algorithms such as exponential smoothing may be used to remove noise from the dataset while preserving small variations in the curve. In other examples, filters such as alpha-beta filtering may be utilized to remove noise from the dataset. In some cases, a distance-based time filter may be used to select a subset of data as described above.

[0077] Although preferred embodiments of the present invention have been illustrated and described herein, it will be apparent to those skilled in the art that such embodiments are provided merely as examples. Various modifications, alterations, and substitutions will now be made by those skilled in the art without departing from the invention. It should be understood that various alternatives to the embodiments of the invention described herein may be employed in practicing the invention. The claims below define the scope of the invention, and methods, structures, and equivalents within these claims are intended to be included within the scope of the claims.

Claims

Claim 1 A method for navigation of a robotic endoscope device comprising: (a) generating a first transformation between the orientation of the robotic endoscope device and the orientation of the position sensor based at least partially on a first set of sensor data collected using a position sensor; (b) generating a second transformation between the coordinate frame of the robotic endoscope device and the coordinate frame of a model representing an anatomical lumen network based at least partially on the first transformation and a second set of sensor data; and (c) updating the second transformation using an update algorithm based at least partially on a third set of sensor data, wherein the update algorithm comprises a fast interval recalculation operation and a slow interval recalculation operation. Claim 2 delete Claim 3 A method according to claim 1, wherein the fast interval recalculation operation comprises: (i) calculating a first set of associations using a subset of data sampled from the third set of sensor data; and (ii) combining the first set of associations with a second set of associations calculated to generate the second transformation of (b). Claim 4 A method according to claim 3, further comprising the step of calculating a point cloud using the combined first set of associations and the second set of associations. Claim 5 A method according to claim 1, wherein the slow interval recalculation operation includes a nearest neighbor algorithm. Claim 6 A method according to claim 1, wherein the slow interval recalculation operation includes the step of updating the second transformation using only the third set of sensor data. Claim 7 A method according to claim 1, wherein the position sensor is an electromagnetic sensor. Claim 8 A method according to claim 1, wherein the coordinate frame of the model representing the anatomical lumen network is generated using a preoperative imaging system. Claim 9 A method according to claim 1, wherein the robotic endoscope device comprises a disposable catheter assembly. Claim 10 A method according to claim 1, wherein the position sensor is located at the distal tip of the robotic endoscope device. Claim 11 A system for navigation of a robotic endoscope device, comprising: a position sensor located at the distal end of the robotic endoscope device; and one or more processors, wherein the one or more processors communicate with the position sensor and the robotic endoscope device and are configured to execute a set of commands, wherein the system generates a first transformation between the orientation of the robotic endoscope device and the orientation of the position sensor based at least partially on a first set of sensor data collected using the position sensor; (b) generates a second transformation between the coordinate frame of the robotic endoscope device and the coordinate frame of a model representing an anatomical lumen network based at least partially on the first transformation and a second set of sensor data; and (c) updates the second transformation using an update algorithm based at least partially on a third set of sensor data, wherein the update algorithm includes a fast interval recalculation operation and a slow interval recalculation operation. Claim 12 delete Claim 13 A system according to claim 11, wherein the rapid interval recalculation operation comprises: (i) calculating a first set of associations using a subset of data sampled from the third set of sensor data; and (ii) combining the first set of associations with a second set of associations calculated to generate the second transformation of (b). Claim 14 A system according to claim 13, wherein the rapid interval recalculation operation further comprises the step of calculating a point cloud using the combined first set of associations and the second set of associations. Claim 15 A system according to claim 11, wherein the slow interval recalculation operation includes a nearest neighbor algorithm. Claim 16 A system according to claim 11, wherein the slow interval recalculation operation includes the step of updating the second transformation using only the sensor data of the third set. Claim 17 A system according to claim 11, wherein the position sensor is an electromagnetic sensor. Claim 18 In claim 11, the system wherein the coordinate frame of the model representing the anatomical lumen network is generated using a preoperative imaging system. Claim 19 A system according to claim 11, wherein the robotic endoscope device comprises a disposable catheter assembly.

Images