Systems and methods for hybrid imaging and navigation

Abstract

To provide a method for navigating an endoscopic device through an anatomical luminal network of a patient.SOLUTION: The method comprises: (a) commanding a distal tip of an articulating elongate member to move along a pre-determined path; (b) concurrent with (a), collecting positional sensor data and kinematics data; and (c) computing an estimated roll angle based on the positional sensor data and the kinematics data.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] References

[0001] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 034,142, filed June 3, 2020, which is incorporated herein by reference. [Background technology]

[0002] Background of the Invention

[0002] Early diagnosis of lung cancer is crucial. The 5-year survival rate for lung cancer is around 18%, which is significantly lower than the next three most common cancers: breast cancer (90%), colorectal cancer (65%), and prostate cancer (99%). In 2018, a total of 142,000 deaths were recorded as being due to lung cancer.

[0003]

[0003] Robotics technology has the advantage that it can be incorporated into endoscopes for various applications, including bronchoscopy. For example, by utilizing soft, deformable structures capable of effectively navigating complex environments such as the interior of the main bronchi, pain and patient discomfort can be significantly reduced. However, guiding such robotic endoscopes can still be challenging due to insufficient precision and accuracy in sensing and detecting the complex and dynamic environment inside a patient's body. Summary of the Invention [Problem to be solved by the invention]

[0004] Various sensing modalities have been employed in lung biopsy for bronchoscope navigation. For example, electromagnetic (EM) navigation is based on registration with an anatomical model constructed using a preoperative CT scan. Live camera vision provides a direct view for the operator to navigate the bronchoscope, while image data is also used for localization by registering the image with the preoperative CT scan. Fluoroscopy from a mobile C-arm fluoroscopy system can be used to observe the catheter and anatomical structures in real time. Tomosynthesis, a partial 3D reconstruction based on X-ray images at various angles, can reveal lesions, which can be superimposed on live fluoroscopic images during navigation or targeting. Endobronchial ultrasound (EBUS) has been used to visualize lesions. Robotic kinematics is useful for determining the location of the bronchoscope tip as the catheter is robotically controlled. However, each technique may not provide sufficient localization precision to reliably navigate the bronchoscope to reach small lesions within the lung.

[0005]

[0005] Recognized herein is a need for a minimally invasive system that enables surgical or diagnostic procedures to be performed with improved sensing and localization capabilities. The present disclosure provides systems and methods that enable early lung cancer diagnosis and treatment with improved localization accuracy and reliability. Specifically, the present disclosure provides a bronchoscope device with multimodal sensing capabilities by combining multiple sensing modalities using a unique fusion framework. The bronchoscope uses a dynamic fusion framework to combine electromagnetic (EM) sensors, direct imaging devices, kinematic data, tomosynthesis, and ultrasound imaging, specifically to enable small lung modules outside the airways to be identified and automatically steer the bronchoscope toward the target. In some cases, multiple sensing modalities are dynamically fused based on real-time confidence scores or uncertainties associated with each modality. For example, when the camera field of view is obstructed or the quality of the sensor data is not good enough to identify the location of an object, the corresponding modality may be assigned a low confidence score. In some cases, when electromagnetic (EM) systems are used, real-time imaging (e.g., tomosynthesis, EBUS, live camera) can be employed to provide corrections to the EM navigation, thereby improving localization accuracy.

[0006] Additionally, conventional endoscopic systems may lack the capability for recovering the orientation of the scope or for roll sensing. The present disclosure provides a method and system with real-time roll detection for recovering the orientation of the scope. Specifically, a roll detection algorithm is provided for detecting the orientation of an imaging device disposed at the distal end of a flexible catheter. The roll detection algorithm may utilize real-time registration and fluoroscopic image data. This may advantageously avoid the use of a six-degree-of-freedom (DOF) EM sensor. In an alternative method, roll detection may be achieved by using radiopaque markers on the distal end of the catheter and real-time radiography, such as fluoroscopy. [Means for solving the problem]

[0007] In one aspect, a method is provided for navigating an endoscopic device through a patient's anatomical lumen network, the method including: (a) commanding a distal tip of an articulating elongate member to move along a predetermined path; (b) simultaneously collecting position sensor data and kinematic data; and (c) calculating an estimated roll angle of the distal tip based on the position sensor data and the kinematic data.

[0008]

[0008] In some embodiments, the predetermined path comprises a linear trajectory. In some embodiments, the predetermined path comprises a non-linear trajectory.

[0009] In some embodiments, the position sensor data is captured by an electromagnetic (EM) sensor. In some embodiments, the EM sensor does not measure roll orientation. In some embodiments, the position sensor data is obtained from an imaging modality.

[0010]

[0010] In some embodiments, calculating the estimated roll angle comprises applying a registration algorithm to the position sensor data and the kinematic data. In some embodiments, the method further comprises assessing the accuracy of the estimated roll angle.

[0011] In another aspect, a method for navigating an endoscopic device through a patient's anatomical lumen network is provided, the method including (a) attaching a radiopaque marker to a distal end of the endoscopic device, (b) capturing fluoroscopic image data of the endoscopic device while the endoscopic device is moving, and (c) reconstructing an orientation of the distal end of the endoscopic device by processing the fluoroscopic image data with a machine learning algorithm-trained model.

[0012]

[0012] In some embodiments, the orientation includes a roll angle of the distal end of the endoscopic device. In some embodiments, the machine learning algorithm is a deep learning network. In some embodiments, the distal end of the endoscopic device is articulatable and rotatable.

[0013] In one aspect, a method is provided for navigating an endoscopic device through a patient's anatomical lumen network using a multimodal framework, the method including: (a) receiving input data from multiple sources, including position sensor data, camera-captured image data, fluoroscopic image data, ultrasound image data, and kinematic data; (b) determining a confidence score for each of the multiple sources; (c) generating input feature data based at least in part on the confidence scores and the input data; and (d) processing the input feature data with a machine learning algorithm-trained model to generate a navigation output for steering a distal end of the endoscopic device.

[0014] In some embodiments, the position sensor data is captured by an EM sensor attached to the distal end of the endoscopic device. In some embodiments, the camera is integrated into the distal end of the endoscopic device. In some embodiments, the fluoroscopic image data is acquired using tomosynthesis techniques.

[0015] In some embodiments, the input data is acquired simultaneously from multiple sources and aligned in time. In some embodiments, the ultrasound image data is captured by an array of ultrasound transducers. In some embodiments, the kinematic data is acquired from a robotic control unit of an endoscopic device.

[0016]

[0016] In some embodiments, the navigation output includes a control command to an actuation unit of the endoscopic device. In some embodiments, the navigation output includes navigation guidance to be presented to an operator of the endoscopic device. In some embodiments, the navigation output includes a desired navigation direction.

[0017] In another aspect, a method for compensating for respiratory motion while navigating an endoscopic device through a patient's anatomical lumen network is provided, the method including: (a) acquiring position data while navigating the endoscopic device through the anatomical lumen network, (b) creating a respiratory motion model based on the position data with the aid of a machine learning algorithm-trained model, where the respiratory motion model is created by distinguishing respiratory motion from navigation motion of the endoscopic device, and (c) generating commands to steer a distal portion of the endoscopic device by compensating for the respiratory motion using the created respiratory motion model.

[0018]

[0018] In some embodiments, the position data is captured by an EM sensor located at a distal portion of the endoscopic device. In some embodiments, the machine learning algorithm is a deep learning network. In some embodiments, the position data is smoothed and decimated.

[0019]

[0019] It should be noted that the provided endoscopic systems can be used in a variety of minimally invasive surgical, therapeutic, or diagnostic procedures involving various types of tissue, including cardiac, bladder, and lung tissue, as well as within other anatomical regions of a patient's body, such as the digestive system, including but not limited to the esophagus, liver, stomach, colon, and urinary tract, or the respiratory system, including but not limited to the bronchi, lungs, and various others.

[0020]

[0020] Additional aspects and advantages of the present disclosure will be readily apparent to those skilled in the art from the following detailed description. In the detailed description, only exemplary embodiments of the present disclosure are shown and described. As will be recognized, the present disclosure is capable of other and different embodiments, and its several details can be modified in various obvious respects, all without departing from the present disclosure. Accordingly, the drawings and description should be regarded as illustrative in nature, and not as limiting.

[0021] Incorporation by Reference

[0021] All publications, patents, and patent applications referred to in this specification are incorporated by reference herein to the same extent as if each individual publication, patent, or patent application was specifically and individually indicated to be incorporated by reference. To the extent that the publications, patents, or patent applications incorporated by reference conflict with the disclosure contained herein, it is intended that the present specification supersede and / or take precedence over any such conflicting matter.

[0022] The novel features of the invention are set forth with particularity in the appended claims. A better understanding of the features and advantages of the present invention will be obtained by reference to the following detailed description that sets forth illustrative embodiments, in which the principles of the invention are utilized, and the accompanying drawings (also referred to herein as "Figure" and "FIG.") [Brief explanation of the drawings]

[0023] [Figure 1] FIG. 1 shows an example of a rotating frame. [Figure 2]

[0024] Figure 2 shows an example of the calibration procedure. [Figure 3]

[0025] Figure 3 shows the results of an example calibration process. [Figure 4]

[0026] Figure 4 shows the scope inside the tube lumen in the experimental setup. [Figure 5]

[0027] Figure 5 shows an example of a radiopaque marker attached to the tip of a catheter for pose estimation. [Figure 6]

[0028] Figure 6 shows a schematic diagram of an intelligent fusion framework for a multimodal navigation system. [Figure 7]

[0029] FIG. 7 shows an example of calculating compensation for respiratory motion. [Figure 8]

[0030] FIG. 8 shows an example of a robotic endoscopic system supported by a robotic support system. [Figure 9]

[0031] FIG. 9 shows an example of an instrument drive mechanism that provides a mechanical interface to the handle portion of a robotic endoscope. DETAILED DESCRIPTION OF THE INVENTION

[0024] Detailed Description of the Invention

[0032] While various embodiments of the present invention have been shown and described herein, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Numerous variations, changes, and substitutions will occur to those skilled in the art without departing from the invention. It is understood that various alternatives to the embodiments of the invention described herein may be employed.

[0025]

[0033] While the exemplary embodiments will be directed primarily to bronchoscopes, those skilled in the art will understand that this is not intended to be limiting and that the devices described herein may be used for other therapeutic or diagnostic procedures and within other anatomical regions of a patient's body, such as the digestive system, including but not limited to the esophagus, liver, stomach, colon, urinary tract, or the respiratory system, including but not limited to the bronchi, lungs, and various others.

[0026]

[0034] The embodiments disclosed herein can be combined in one or more of many ways to provide improved diagnosis and treatment to patients. The embodiments of the present disclosure can be combined with existing methods and devices to provide improved treatment, such as in combination with known methods of pulmonary diagnosis, surgery, and surgery of other tissues and organs. It should be understood that any one or more of the structures and steps as described herein can be combined with any one or more of the additional structures and steps of the methods and devices as described herein, and that the figures and supporting text provide a description of the embodiments.

[0027]

[0035] Although the treatment plans and definitions of diagnostic or surgical procedures as described herein are presented in the context of bronchoscopy, pulmonary diagnostic or surgical procedures, the methods and devices as described herein can be used to treat any tissue of the body and any organ and duct of the body, such as the brain, heart, lungs, intestines, eyes, skin, kidneys, liver, pancreas, stomach, uterus, ovaries, testes, bladder, ears, nose, mouth, soft tissues such as bone marrow, adipose tissue, muscle, glandular and mucosal tissue, spinal cord and nerve tissue, cartilage, etc., biological hard tissues such as teeth, bones, and the like, and body lumens and passageways such as sinuses, ureters, colon, esophagus, pulmonary passageways, blood vessels, and throat, etc.

[0028]

[0036] Whenever the terms "at least," "greater than," or "greater than or equal to" precede the first number in a series of two or more numbers, the term "at least," "greater than," or "greater than or equal to" applies to each and every number in the series. For example, greater than or equal to 1, 2, or 3 is equivalent to 1 or more, 2 or more, or 3 or more.

[0029]

[0037] Whenever the terms "no more than," "less than," or "less than or equal to" precede the first number in a series of two or more numbers, the term "no more than," "less than," or "less than or equal to" applies to each and every number in the series. For example, less than or equal to 3, 2, or 1 is equivalent to 3 or less, 2 or less, or 1 or less.

[0030]

[0038] As used herein, a processor encompasses one or more processors, e.g., a single processor, or multiple processors, e.g., in a distributed processing system. A controller or processor as described herein generally includes a tangible medium for storing instructions for performing process steps, and a processor may include, for example, one or more of a central processing unit, programmable array logic, gate array logic, or field programmable gate array. In some cases, the one or more processors may be a programmable processor (e.g., a central processing unit (CPU), a graphics processing unit (GPU), or a microcontroller), a digital signal processor (DSP), a field programmable gate array (FPGA), and / or one or more Advanced RISC Machine (ARM) processors. In some cases, the one or more processors may be operably coupled to a non-transitory computer-readable medium. The non-transitory computer-readable medium may store logic, code, and / or program instructions executable by one or more processor units to perform one or more steps. The non-transitory computer-readable medium may include one or more memory units (e.g., removable media or external storage, such as an SD card or random access memory (RAM)). One or more methods or operations disclosed herein may be implemented in the form of hardware components, or a combination of hardware and software, such as, for example, an ASIC, a special-purpose computer, or a general-purpose computer.

[0031]

[0039] As used herein, the terms distal and proximal may generally refer to a location referenced from the device, as opposed to an anatomical reference. For example, a distal location on a bronchoscope or catheter may correspond to a proximal location on the patient's elongate element, and a proximal location on a bronchoscope or catheter may correspond to a distal location on the patient's elongate element.

[0032]

[0040] An endoscopic system as described herein includes an elongated portion or member, such as a catheter. The terms “elongated member” and “catheter” are used interchangeably throughout this specification unless the context suggests otherwise. The elongated member can be positioned directly within a body lumen or cavity. In some embodiments, the system may further include a support device, such as a robotic manipulator (e.g., a robotic arm), for driving, supporting, positioning, or controlling the movement and / or motion of the elongated member. Alternatively, or in addition, the support device may be a handheld device or other control device that may or may not include a robotic system. In some embodiments, the system may further include peripheral devices and subsystems, such as an imaging system, that may assist and / or facilitate navigation of the elongated member to a target site within a subject's body.

[0033]

[0041] In some embodiments, the systems and methods provided herein may include a multimodal sensing system that may implement a position sensing system, such as at least an electromagnetic (EM) sensor, an optical fiber sensor, and / or other sensors for registering and displaying a medical instrument with a preoperatively recorded surgical image, thereby positioning the distal portion of the endoscope relative to the patient's body or a global reference frame. The position sensor may be a component of an EM sensor system that includes one or more conductive coils that may be exposed to an externally generated electromagnetic field. Each coil of the EM sensor system used to implement the position sensor system then produces an induced electrical signal having characteristics that depend on the coil's position and orientation relative to the externally generated electromagnetic field. In some cases, the EM sensor system used to implement the position sensing system may be configured and positioned to measure at least three degrees of freedom, e.g., three position coordinates X, Y, and Z. Alternatively, or in addition, the EM sensor system may be configured and positioned to measure five degrees of freedom, e.g., three position coordinates X, Y, and Z, and two orientation angles indicating the pitch and yaw of a fiducial. In some cases, the roll angle may be provided by including a MEMS-based gyro sensor and / or accelerometer, but if a gyroscope or accelerometer is not available, the roll angle may be recovered by a proprietary roll detection algorithm, as described later in this specification.

[0034]

[0042] The present disclosure provides various algorithms and methods for roll detection, or catheter pose estimation. The provided methods or algorithms may advantageously enable catheter pose estimation without a 6DOF sensor. Additionally, the provided methods and algorithms can be easily integrated or applied to any existing system or device lacking roll detection capabilities without requiring additional hardware or modifications to the underlying system.

[0035] Role Detection Algorithm

[0043] The present disclosure provides algorithms for real-time scope orientation measurement and roll detection. The algorithms provided herein can be used to detect roll orientation for any robotically actuated / controlled flexible device. In some embodiments, the algorithms can include a "wiggle" method for generating instantaneous roll estimates for the catheter tip. The roll detection algorithm can include a protocol for automated catheter tip movement while the robotic system collects EM sensor data and kinematic data. In some cases, the kinematic data can be obtained from the robotic control unit of the endoscopic device.

[0036]

[0044] 1 shows an example of a rotating frame 100 for a catheter tip 105. In the example shown, a camera 101 and one or more illumination devices (e.g., LED fiber-based lights) 103 may be incorporated within the catheter tip. The camera may include imaging optics (e.g., lens elements), an image sensor (e.g., CMOS or CCD), and illumination (e.g., LED or fiber-based lights).

[0037]

[0045] In some embodiments, the catheter 110 can include a shaft 111, an articulating (flexing) section 107, and a steerable distal portion or catheter tip 105. The articulating section (flexing section) 107 connects the steerable distal portion to the shaft 111. For example, the articulating section 107 can be connected to the distal tip portion at a first end and to the shaft portion at a second end or at a base 109. The articulating section can be articulated by one or more pull wires. For example, the distal ends of one or more pull wires can be fixed to or integral with the catheter tip 105, such that manipulation of the pull wires by a control unit can apply a force or tension to the catheter tip 105, thereby steering or articulating the distal portion (e.g., flexible section) of the catheter (e.g., up, down, pitch, yaw, or any direction in between).

[0038]

[0046] The rotation frame and rotation matrices used in the roll detection algorithm are shown in Figure 1 and are defined as follows:

number

number

number

number

number

[0039]

[0051] As mentioned above, the relative orientation of the EM sensor "s" with respect to the catheter tip frame "ct", i.e.,

number

number

number

number

number

number

[0040]

[0058] FIG. 2 shows an example of a calibration procedure. The catheter tip is moved (e.g., articulated) while EM and kinematic data are collected. In some cases, the calibration procedure may be performed autonomously without human intervention. For example, articulation of the catheter tip may be performed automatically by executing a predetermined calibration program. Alternatively, or additionally, a user may be allowed to move the catheter tip via a controller. The alignment algorithm, as described above, is applied to calculate the relative rotation between the tip-located EM sensor with respect to the kinematic tip frame.

[0041]

[0059] FIG. 3 shows the results of an example calibration process. The calibration process may be illustrated with a visualization to provide real-time visualization of the alignment procedure. The calibration process / results can be presented to the user in various formats. As shown in the figure, the visualization may be a plot that shows the calibration process provides accurate, real-time calibration results. For example, the plot shows that the z-axis of the endoscope base frame is in the approximate direction of scope tip travel (301). A second observation 303 shows that the x-axis of the endoscope base frame points away from the EM frame. This is an expected result because the scope tip is oriented so that the camera is closer to the EM field generator. A third observation 305 shows that the x-axis of the “s” frame is properly aligned with the scope tip travel direction. In some cases, a visual indicator (e.g., a textual description or visual indicator) of the calibration observation or result as described above may be displayed to the user on the user interface.

[0042]

[0060] In some embodiments, the roll detection algorithm may include an algorithm based on point coordinate registration. Similar to the calibration procedure described above, this algorithm relies on simple point coordinate registration. In some cases, instead of swinging the catheter tip around within its workspace (i.e., along a non-linear trajectory), calibration can be performed by commanding the tip to translate along a linear trajectory. The algorithm may enable calibration using a linear trajectory (instead of swinging along a non-linear trajectory), which advantageously reduces the time required for calibration. In an exemplary process, the algorithm may include the following operations:

[0043]

[0061] (1) Tip motion is performed by moving the catheter tip back and forth within its workspace. EM sensor data and kinematic data are collected while the catheter tip is moved along a predetermined path, such as by swinging the tip back and forth or by following commands to move along a path (e.g., translating along a short, straight trajectory).

[0044]

[0062] (2) Apply the alignment algorithm to the EM sensor data to obtain the rotation matrix

number

number

number

number

[0045]

[0063] (3) Using the EM sensor data, we reconstruct the expected kinematic catheter tip frame.

number

number

number

number

[0046]

[0064] The expected kinematic catheter tip frame can be estimated using the following equation:

number

[0047]

[0066] By mapping the above orientations (estimated using the registration output), the relative orientation between the endoscope tip frame and the endoscope base frame can be recovered using the following equation:

number

[0048]

[0068] The expected or estimated kinematic catheter tip frame is expressed relative to the kinematic base frame. Such expected kinematic catheter tip frame or estimated rotation of the catheter tip can only be obtained using position information, i.e., the registration process.

[0049]

[0069] The method involves kinematic mapping

number

number

number

[0050]

[0071] The roll error in the reconstruction of the kinematic frame from the EM sensor data can be calculated by decomposing the roll offset into an axis and angle representation. The angle represents the error in the reconstruction of the kinematic frame from the EM sensor data. The error angle can be obtained using the following formula:

number

[0051]

[0073] Next, the error angle is projected onto the axis of the endoscope's travel direction, and the pure roll error θ r get:

number

[0052]

[0075] In some cases, alternative methods can be used to calculate the roll error in the final step. The roll error can be calculated using geometric methods by projecting the reconstructed catheter tip coordinate frame onto a plane defined by the direction of travel of the endoscope tip. That is, the direction of travel of the endoscope is orthogonal to the plane. The x-axis of the reconstructed catheter tip can be calculated, and the roll error can be defined as the angle between the reconstructed x-axis and the x-axis at the kinematic catheter tip using the following equation:

number

number

number

number

number

[0053] experiment

[0079] The experiment was performed by inserting a scope into the tube lumen to simulate the effect of the scope being within the lumen. Figure 4 shows the scope within the tube lumen in the experimental setup. 0 The proposed algorithm was evaluated using five datasets with a mean calculated roll error of 14.8 ± 9.1. The last two experiments had a much larger error than those in the first three experiments. 0 The proposed algorithm is evaluated using five data sets with an average calculated roll error of .

[0054]

[0080] Below is a table of raw data collected or generated from the experiment shown in Figure 4. r2 is the roll angle calculated using an alternative method (i.e., the geometric method).

[0055] [Table 1]

[0056]

[0081] Other methods for calculating roll orientation may also be employed, similar to the registration process described above, in which two sets of position data are utilized: (1) the position of the kinematic tip frame relative to the kinematic base frame, and (2) the position of the EM sensor embedded within the endoscope tip relative to the EM field generator coordinate system. In some embodiments, the EM sensor may be rigidly (if not physically) fixed within the endoscope tip, similar to the kinematic tip frame, and a registration process may be used to calculate the relative orientation between the base frame of both sets of position data.

[0057]

[0082] In some cases, instead of using EM sensor data, other sensor data can also be employed to calculate roll orientation. Non-kinematic position information does not necessarily need to come from an electromagnetic tracking system. Instead, for example, fluoroscopic image information can be used to capture position information. In conjunction with the registration methods described above (e.g., point coordinate registration or other coordinate registration algorithms), the relative orientation between the endoscope kinematic frame and the reference fluoroscopic coordinate system can be calculated. For example, roll motion can be recovered by mapping motion from fluoroscopic image data to motion in the kinematics obtained from the motion of the drive mechanism (e.g., calculating kinematic data and scope tip position based on fluoroscopic image data). In some cases, when an imaging modality (e.g., an imaging modality providing position data to replace EM sensor data) actually explicitly provides position information within a known coordinate system, an additional step of mapping image artifacts to coordinate positions can be performed.

[0058] Catheter pose estimation using radiopaque materials

[0083] In some embodiments, roll measurement or pose estimation can be achieved using object recognition of radiopaque materials. For example, by placing a radiopaque pattern on the catheter tip, fluoroscopic imaging and image recognition can be used to recover the catheter's orientation.

[0059]

[0084] The method may have the ability to measure the roll angle along the catheter tip axis when viewed under fluoroscopic imaging. This may advantageously enable catheter pose estimation without a 6DOF sensor. Additionally, the provided method may not require user interaction, as the catheter orientation can be calculated automatically with the aid of fluoroscopic imaging.

[0060]

[0085] Fluoroscopy is an imaging modality that acquires real-time motion images of a patient's anatomy, medical devices, and any radiopaque markers within an imaging field of view using X-rays. Fluoroscopy systems may include C-arm systems that provide positional flexibility and are capable of orbital, horizontal, and / or vertical movement via manual or automatic control. Non-C-arm systems are stationary and provide less motion flexibility. Fluoroscopy systems generally use either an image intensifier or a flat-panel detector to generate two-dimensional real-time images of a patient's anatomy. Biplane fluoroscopy systems simultaneously capture two fluoroscopic images, each from a different (often orthogonal) perspective. In the presented method, a radiopaque marker disposed at the tip of a catheter can be made visible by fluoroscopic imaging and analyzed to estimate the pose of the catheter or camera.

[0061]

[0086] FIG. 5 shows an example of a radiopaque marker 503 attached to a catheter tip 501 for pose estimation. As shown, a radiopaque pattern is placed on the tip of the endoscope and imaged using fluoroscopic imaging. In some cases, the radiopaque marker may be integrally coupled to the outer surface of the tip of the elongate member. Alternatively, the radiopaque marker may be removably coupled to the elongate member. Fluoroscopic image data may be captured while the endoscopic device is in motion. The radiopaque pattern is visible in the fluoroscopic image data. The fluoroscopic image data may be processed to recover the orientation of the catheter tip, such as using computer vision, machine learning, or other object recognition methods to recognize and analyze the shapes of markers in the fluoroscopic images.

[0062]

[0087] The radiopaque marker can have any pattern, shape, or geometric feature that is useful for recovering the 3D orientation of the catheter tip. For example, the pattern can be asymmetric with at least three points. In the illustrated example, the radiopaque marker has an "L" shape, which is not intended to be limiting. Markers of many shapes and sizes can be employed. In some cases, the marker can have an asymmetric shape or pattern with at least three distinct points.

[0063]

[0088] Computer vision (CV) techniques or systems are used to process the 2D image data to construct the 3D orientation or pose of the object. Any other suitable optical or image processing techniques can be used to recognize and isolate the pattern and associate it with one of the rotation angles. For example, the orientation of the camera or catheter tip can be obtained using methods including object recognition, stereoscopic vision, monocular shape-from-motion, shape-from-shading, and simultaneous localization and mapping (SLAM), or other computer vision techniques such as optical flow, computational stereo approaches, predictive models, machine learning approaches, iterative methods combined with predictive filtering, or any non-rigid registration method.

[0064]

[0089] In some cases, optical techniques for predicting catheter pose or roll angle may employ one or more trained predictive models. In some cases, input data to be processed by the predictive models may include image or optical data. Image or video data may be captured by a fluoroscopy system (e.g., a C-arm system), and the roll orientation may be recovered in real time while the image or optical data is being collected.

[0065]

[0090] The one or more predictive models can be trained using any suitable deep learning network. For example, the deep learning network may employ a U-Net architecture, which is essentially a multi-scale encoder-decoder architecture with skip connections that directly transfer each encoder layer output to the corresponding decoder layer output. In one example of a U-Net architecture, unsampling in the decoder is performed using a pixel shuffle layer, which helps reduce gridding artifacts. Merging of encoder features with decoder features is performed using pixel-by-pixel addition operations, resulting in reduced memory requirements. Residual connections between the central input frame and the output are introduced to accelerate the training process.

[0066]

[0091] The deep learning model can employ any type of neural network model, such as a feedforward neural network, a radial basis function network, a recurrent neural network, a convolutional neural network, a deep residual learning network, and the like. In some embodiments, the deep learning algorithm can be a convolutional neural network (CNN). The model network can be a deep learning network, such as a CNN, which can include multiple layers. For example, a CNN model can include at least an input layer, multiple hidden layers, and an output layer. A CNN model can include any total number of layers and any number of hidden layers. The simplest architecture of a neural network starts with an input layer, followed by a series of intermediate or hidden layers, and ends with an output layer. The hidden or intermediate layers can act as trainable feature extractors, while the output layer can output an improved image frame. Each layer of a neural network can include multiple neurons (or nodes). A neuron receives input, either directly from input data (e.g., low-quality image data, etc.) or from the output of other neurons, and performs a specific operation, such as summation. In some cases, connections from inputs to neurons are associated with weights (or weighting coefficients). In some cases, a neuron may sum the products of all pairs of inputs and their associated weights. In some cases, the weighted sum is offset using a bias. In some cases, the output of a neuron may be gated using a threshold or activation function. The activation function may be linear or nonlinear. The activation function may be, for example, a rectified linear unit (ReLU) activation function, or other functions such as saturated hyperbolic tangent, identity, binary step, logistic, arcTan, soft sine, parametric rectified linear unit, exponential linear unit, soft plus, bent identity, softExponential, sinusoid, sinc, Gaussian, sigmoid function, or any combination thereof.During the training process, the weights or parameters of the CNN are adjusted to approximate the ground truth data, thereby learning a mapping from input raw image data to desired output data (e.g., the orientation of an object in a 3D scene).

[0067] Hybrid Imaging and Navigation

[0092] The endoscopic systems of the present disclosure may combine multiple sensing modalities to provide enhanced navigation capabilities. In some embodiments, the multimodal sensing system may include at least position sensing (e.g., an EM sensor system), direct vision (e.g., a camera), ultrasound imaging, and tomosynthesis.

[0068]

[0093] As mentioned above, electromagnetic (EM) navigation is based on registration with an anatomical model constructed using a preoperative CT scan. Live camera vision provides a direct view for the operator to navigate the bronchoscope, while image data is also used for localization by registering the image with the preoperative CT scan. Fluoroscopy from a mobile C-arm fluoroscopy system can be used to observe the catheter and anatomical structures in real time. Tomosynthesis, a partial 3D reconstruction based on X-ray images at various angles, can reveal lesions, which can be superimposed on live fluoroscopic images during navigation or targeting. Endobronchial ultrasound (EBUS) has been used to visualize lesions. Robot kinematics is useful for determining the location of the bronchoscope tip when the catheter is robotically controlled. In some cases, kinematic data can be obtained from the robotic control unit of the endoscopic device.

[0069]

[0094] In some cases, the endoscope system may implement a position sensing system, such as an electromagnetic (EM) sensor, fiber optic sensor, and / or other sensor for registering and displaying the medical instrument with preoperatively recorded surgical images, thereby positioning the distal portion of the endoscope relative to the patient's body or a global reference frame. The position sensor may be a component of an EM sensor system including one or more conductive coils that may be exposed to an externally generated electromagnetic field. Each coil of the EM sensor system used to implement the position sensor system then produces an induced electrical signal having characteristics that depend on the coil's position and orientation relative to the externally generated electromagnetic field. In some cases, the EM sensor system used to implement the position sensing system may be configured and positioned to measure at least three degrees of freedom, e.g., three position coordinates X, Y, and Z. Alternatively, or in addition, the EM sensor system may be configured and positioned to measure six degrees of freedom, e.g., three position coordinates X, Y, Z and three orientation angles indicating the pitch, yaw, and roll of a fiducial, or five degrees of freedom, e.g., three position coordinates X, Y, Z and two orientation angles indicating the pitch and yaw of a fiducial.

[0070]

[0095] The direct view may be provided by an imaging device such as a camera. The imaging device may be located at the distal tip of the catheter or elongate member of the endoscope. In some cases, the directional vision system may include an imaging device and an illumination device. In some embodiments, the imaging device may be a video camera. The imaging device may include optical elements and an image sensor for capturing image data. The image sensor may be configured to generate image data according to wavelengths of light. Various image sensors for capturing image data may be employed, such as complementary metal oxide semiconductor (CMOS) or charge-coupled device (CCD). The imaging device may be a low-cost camera. In some cases, the image sensor may be provided on a circuit board. The circuit board may be an imaging printed circuit board (PCB). The PCB may include multiple electronic elements for processing the image signal. For example, circuitry for a CCD sensor may include an analog-to-digital converter and an amplifier for amplifying and converting the analog signal provided by the CCD sensor. Optionally, the image sensor may be integrated with an amplifier and a converter for converting the analog signal to a digital signal, so that a circuit board may not be required. In some cases, the output of the image sensor or circuit board may be image data (digital signals) that may be further processed by the camera's camera circuitry or processor. In some cases, the image sensor may include an array of optical sensors. As described later in this specification, the imaging device may be located at the distal tip of a catheter or a separate hybrid probe that is attached to an endoscope.

[0071]

[0096] The illumination device may include one or more light sources positioned at the distal tip of the endoscope or catheter. The light sources may be light-emitting diodes (LEDs), organic LEDs (OLEDs), quantum dots, or any other suitable light source. In some cases, the light sources may be miniature LEDs for compact designs, or dual-tone flashing LED illumination.

[0072]

[0097] Provided endoscopic systems may use ultrasound to help guide a physician to locations outside the airway. For example, a user may use ultrasound to identify the location of a lesion in real time and guide the endoscope to where a computed tomography (CT) scan reveals the approximate location of a solitary pulmonary nodule. The ultrasound may be linear endobronchial ultrasound (EBUS), also known as convex probe EBUS, which may image to the side of the imaging endoscopic device, or radial probe EBUS, which images 360° radially. For example, a linear endobronchial ultrasound (EBUS) transducer or transducer array may be located at the distal portion of the endoscope.

[0073]

[0098] The multimodal sensing features of the present disclosure may include combining multiple sensing modalities using a unique fusion framework. Using a dynamic fusion framework, a bronchoscope may combine electromagnetic (EM) sensors, direct imaging devices, tomosynthesis, kinematic data, and ultrasound imaging, specifically enabling small lung modules outside the airways to be identified and automatically steering the bronchoscope toward the target. In some cases, multiple sensing modalities are dynamically fused based on real-time confidence scores or uncertainties associated with each modality. In some cases, when an electromagnetic (EM) system is used, real-time imaging (e.g., tomosynthesis, EBUS, live camera) may be employed to provide corrections to the EM navigation, thereby improving localization accuracy.

[0074]

[0099] The provided systems and methods may include a multimodal navigation system that utilizes machine learning and AI techniques to optimize the fusion of multimodal data. In some embodiments, the multimodal navigation system may combine four or more different sensing modalities, namely, position sensing (e.g., EM sensor systems), direct vision (e.g., cameras), ultrasound imaging, kinematic data, and tomosynthesis, via an intelligent fusion framework.

[0075]

[0100] The intelligent fusion framework may include one or more predictive models and may be trained using any suitable deep learning network, as described above. The deep learning model may be trained using supervised learning or semi-supervised learning. For example, to train the deep learning network, a pair of datasets having input image data (i.e., images captured by a camera) and desired output data (e.g., navigation direction, pose, or location of the catheter tip) may be generated by a training module of the system as training datasets.

[0076]

[0101] Alternatively, or in addition, handcrafted rules can be utilized by the fusion framework, for example, confidence scores can be generated for each of the different modalities, and multiple data can be combined based on real-time conditions.

[0077]

[0102] 6 schematically illustrates an intelligent fusion framework 600 for dynamically controlling a multimodal navigation system, fusing and processing real-time sensory data and robot kinematic data, and generating outputs for navigation and various other purposes. In some embodiments, intelligent fusion framework 600 may include a position sensor 610, an optical imaging device (e.g., a camera) 620, a tomosynthesis system 630, an EBUS imaging system 640, a robot control system 650 for providing robot kinematic data, a sensor fusion component 660, and an intelligent navigation direction estimation engine 670. Position sensor 610, optical imaging device (e.g., a camera) 620, a tomosynthesis system 630, an EBUS imaging system 640, and robot kinematic data 650 may be the same as those described above.

[0078]

[0103] In some embodiments, the output 613 of the navigation engine 670 may include a desired navigation direction or steering control output signal for steering the robotic endoscope in real time. In some cases, when the robotic endoscope system is in autonomous mode, the multimodal navigation system may utilize artificial intelligence algorithms (e.g., deep machine learning algorithms) to process the multimodal input data and provide a predicted steering direction and / or steering control signal as an output for steering the distal tip of the robotic endoscope. In some cases, for example, in fully automatic mode, the multimodal navigation system may be configured to guide the advancing endoscope with little or no input from a surgeon or other operator. The output 613 may include a desired direction that is converted by a controller of the robotic endoscope system into a control signal for controlling one or more actuation units. Alternatively, the output may directly include control commands for one or more actuation units. In some cases, for example, in semi-automatic mode, the multimodal navigation system may be configured to provide assistance to a surgeon who is actively guiding the advancing endoscope. In such cases, the output 613 may include guidance for the operator of the robotic endoscope system.

[0079]

[0104] The output 613 may be generated by a navigation engine 670. In some embodiments, the navigation engine 670 may include an input feature generation module 671 and a trained predictive model 673. The predictive model may be a trained model or may be trained using a machine learning algorithm. The machine learning algorithm may be any type of machine learning network, such as a support vector machine (SVM), a naive Bayes classification, a linear regression model, a quantile regression model, a logistic regression model, a random forest, a neural network, a convolutional neural network (CNN), a recurrent neural network (RNN), a gradient boosting classifier or repressor, or another supervised or unsupervised machine learning algorithm (e.g., a generative adversarial network (GAN), Cycle-GAN, etc.).

[0080]

[0105] The input feature generation module 671 may generate input feature data to be processed by the trained predictive model 673. In some embodiments, the input feature generation module 671 may receive data from the position sensor 610, the optical imaging device (e.g., camera) 620, the tomosynthesis system 630, the EBUS imaging system 640, and the robot kinematics data 650, extract features, and generate the input feature data. In some embodiments, the data received from the position sensor 610, the optical imaging device (e.g., camera) 620, the tomosynthesis system 630, and the EBUS imaging system 640 may include raw sensor data (e.g., image data, EM data, tomosynthesis data, ultrasound images, etc.). In some cases, the input feature generation module 671 may preprocess (e.g., data alignment) raw input data generated by multiple different sensing systems (e.g., sensors may capture data at different frequencies) or from different sources (e.g., third-party application data). For example, data captured by a camera, a position sensor (e.g., an EM sensor), ultrasound image data, tomosynthesis data may be aligned with respect to time and / or identified features (e.g., lesions). In some cases, multiple sources of data may be captured simultaneously.

[0081]

[0106] The data received from the various data sources 610, 620, 630, 640, 650 may include processed data. For example, data from a tomosynthesis system may include reconstructed data or information about lesions identified from the raw data.

[0082]

[0107] In some cases, the data 611 received from the multimodal data sources may be adaptable to real-time conditions. The sensor fusion component 660 may be operatively coupled to the data sources to receive their respective output data. In some cases, the output data produced by the data sources 610, 620, 630, 640, and 650 may be dynamically adjusted based on real-time conditions. For example, multiple sensing modalities may be dynamically fused based on real-time confidence scores or uncertainties associated with each modality. The sensor fusion component 660 may evaluate the confidence scores for each data source and determine the input data to be used to infer navigation direction. For example, when a camera field of view is obstructed or the quality of the sensor data is not good enough to identify the location of an object, the corresponding modality may be assigned a low confidence score. In some cases, the sensor fusion component 660 may weight the data from multiple sources based on the confidence scores. The multiple data may be combined based on real-time conditions. In some cases, when electromagnetic (EM) systems are used, real-time imaging (e.g., tomosynthesis, EBUS, live camera) can be employed to provide corrections to the EM navigation, thereby improving localization accuracy.

[0083] Respiratory Compensation for Electromagnetic (EM)-Based Navigation

[0108] While traversing the lung structures, the bronchoscope may be moved by a certain offset (e.g., up to 2 centimeters) due to respiratory motion, and there is a need to compensate for the respiratory motion by allowing smooth navigation and improved alignment with the target site (e.g., lesion).

[0084]

[0109] The present disclosure can improve navigation and location tracking by creating a real-time adaptable model that predicts respiratory motion. In some embodiments, the respiratory motion model can be generated based on position sensor (e.g., EM sensor) data. Figure 7 shows an example of calculating compensation for respiratory motion.

[0085]

[0110] The sensor data for building the model can be acquired while a device with EM sensors is positioned inside the patient's body without user interaction, so that the detected motion is substantially the patient's respiratory motion. Alternatively, or in addition, the sensor data for building the model can be collected while the device is being driven or operated, so that the collected sensor data can indicate motion resulting from both respiratory motion and active device motion. In some cases, the motion model can be a relatively low-order parametric model that can be created by using autocorrelation of the sensor signal to identify periodic motion frequencies and / or using a filter to extract low-frequency motion. Alternatively, or in addition, the model can be created using a reference signal. For example, position sensors placed on the patient's body, a rubber band, or a ventilator, or an acoustic signal from the ventilator's operation, can be used to provide a reference signal for distinguishing respiratory motion from raw sensor data.

[0086]

[0111] The method may include preprocessing the position sensor data by smoothing, decimating, and dividing the position sensor data into dimensional components. The type, form, or format of the time-series position data may depend on the type of sensor. For example, when the time-series data is collected from a 6DOF EM sensor, the time-series data may be decomposed into X, Y, and Z axes. In some cases, the time-series data may be preprocessed and arranged into a three-dimensional numerical array.

[0087]

[0112] A respiratory motion model may be constructed by dimensionally fitting a defined function to the preprocessed sensor data. The constructed model may be used to calculate an offset to be applied to the incoming sensor data to compensate for respiratory motion in real time. In some cases, the respiratory motion model may be calculated and updated as new sensed data is collected and processed, and the updated respiratory motion model may be deployed for use.

[0088]

[0113] In some cases, static information from lung segmentation may be utilized to distinguish user actions from respiratory motion, thereby increasing prediction accuracy. In some cases, the model may be created using machine learning techniques. In some cases, the respiratory motion model is created by distinguishing respiratory motion from navigation motion of the endoscopic device with the help of machine learning techniques. Various deep learning models and frameworks as described elsewhere herein may be used to train the respiratory model. In some cases, the EM sensor data may be preprocessed (e.g., smoothed and decimated), and the preprocessed EM sensor data may be used to generate input features to be processed by the trained model.

[0089]

[0114] The respiratory motion model can be used to plan tool trajectories and / or navigate the endoscope. For example, by compensating for respiratory motion, commands can be generated to deflect the distal tip of the scope to follow the path of the structure being inspected, thereby minimizing frictional forces on the surrounding tissue. In another example, it is beneficial to time a surgical task or subtask (e.g., needle insertion) for a pause between inspiration and expiration.

[0090]

[0115] In some embodiments, the endoscopic device may be a disposable robotic endoscope. In some cases, only the catheter may be disposable. In some cases, at least a portion of the catheter may be disposable. In some cases, the entire robotic endoscope may be detached from the instrument drive mechanism and disposed of.

[0091]

[0116] The robotic endoscopes described herein may include suitable means for deflecting the distal tip of the scope to follow the path of the structure under inspection while minimizing deflection or frictional forces on the surrounding tissue. For example, a control or pull cable may be carried within the endoscope body to connect an articulating section adjacent the distal end to a set of control mechanisms (e.g., a handle) at the proximal end of the endoscope or to a robotic support system. The orientation (e.g., roll angle) of the distal tip may be recovered by the methods described above. Navigation control signals may be generated by a navigation system as described above, and control of the robotic endoscope's motion may have respiratory compensation capabilities as described above.

[0092]

[0117] The robotic endoscope system can be removably coupled to an instrument drive mechanism. The instrument drive mechanism can be attached to the arm of the robotic support system or to any actuated support system. The instrument drive mechanism can provide a mechanical and electrical interface to the robotic endoscope system. The mechanical interface can allow the robotic endoscope system to be removably coupled to the instrument drive mechanism. For example, the handle portion of the robotic endoscope can be attached to the instrument drive mechanism via a quick attachment / detachment means, such as a magnet and spring-loaded level. In some cases, the robotic endoscope can be manually attached to or detached from the instrument drive mechanism without the use of tools.

[0093]

[0118] FIG. 8 shows an example of a robotic endoscope system supported by a robotic support system. In some cases, the handle portion may electrically communicate with the instrument drive (e.g., instrument drive 820) via an electrical interface (e.g., a printed circuit board), allowing image / video data and / or sensor data to be received by the instrument drive's communications module and transmitted to other external devices / systems. In some cases, the electrical interface may establish electrical communication without cables or wires. For example, the interface may include pins soldered onto an electronic board such as a printed circuit board (PCB). For example, a receptacle connector (e.g., a female connector) may be provided on the instrument drive as a mating interface. This may advantageously allow the endoscope to be quickly plugged into the instrument drive or robotic support without utilizing a special cable. This type of electrical interface may also serve as a mechanical interface, establishing both a mechanical and electrical connection when the handle portion is plugged into the instrument drive. Alternatively, or in addition, the instrument drive may provide only the mechanical interface. The handle portion may be in electrical communication with a modular wireless communication device or any other user device (e.g., a portable / handheld device or controller) for transmitting sensor data and / or receiving control signals.

[0094]

[0119] As shown in FIG. 8 , the robotic endoscope 820 can include a handle portion 813 and a flexible elongate member 811. In some embodiments, the flexible elongate member 811 can include a shaft, a steerable tip, and a steerable section, as described elsewhere herein. The robotic endoscope can be a disposable robotic endoscope. In some cases, only the catheter can be disposable. In some cases, at least a portion of the catheter can be disposable. In some cases, the entire robotic endoscope can be detached from the instrument drive mechanism and disposed of. The endoscope can include various stiffness levels along its shaft to improve functional operation.

[0095]

[0120] The robotic endoscope can be removably coupled to an instrument drive 820. The instrument drive 820 can be mounted on an arm of a robotic support system or on any actuated support system, as described elsewhere herein. The instrument drive can provide a mechanical and electrical interface to the robotic endoscope 820. The mechanical interface can allow the robotic endoscope 820 to be removably coupled to the instrument drive. For example, the handle portion of the robotic bronchoscope can be attached to the instrument drive via a quick attachment / detachment means, such as a magnet and spring-loaded level. In some cases, the robotic bronchoscope can be manually attached to or detached from the instrument drive without the use of tools.

[0096]

[0121] 9 shows an example of an instrument drive mechanism 920 that provides a mechanical interface to a handle portion 913 of a robotic endoscope. As shown in this example, the instrument drive mechanism 920 can include a set of motors that are actuated to rotationally drive a set of pull wires of the catheter. The handle portion 913 of the catheter assembly can be mounted on the instrument drive mechanism, which causes its pulley assembly to be driven by the set of motors. The number of pulleys can vary based on the configuration of the pull wires. In some cases, one, two, three, four, or more pull wires can be utilized to articulate the catheter.

[0097]

[0122] The handle portion can be designed to enable the robotic endoscope to be disposable at reduced cost. For example, classic manual endoscopes and robotic endoscopes may have cables within the proximal end of the endoscope handle. The cables often include illumination fibers, camera video cables, and other sensor fibers or cables, such as electromagnetic (EM) sensors or shape-sensing fibers. Such complex cables can be expensive and increase the cost of the bronchoscope. The provided robotic endoscope may have an optimized design that can employ simplified structures and components while maintaining mechanical and electrical functionality. In some cases, the handle portion of the robotic endoscope may employ a cable-free design while providing a mechanical / electrical interface to the catheter.

[0098]

[0123] In some cases, the handle portion may house or include components configured to process image data, provide power, or establish communication with other external devices. In some cases, the communication may be wireless communication. For example, wireless communication may include Wi-Fi, radio communication, Bluetooth, IR communication, or other types of direct communication. Such wireless communication capabilities may enable the robotic bronchoscope to function in a plug-and-play manner and may be conveniently disposed of after a single use. In some cases, the handle portion may include circuitry elements such as a power supply for powering electronics (e.g., a camera and LED light source) disposed within the robotic bronchoscope or catheter.

[0099]

[0124] The handle section can be designed in conjunction with the catheter to eliminate cables or fibers. For example, the catheter section can employ a design with a working channel that allows instruments to pass through a robotic bronchoscope, a vision channel that allows a hybrid probe to pass through, and low-cost electronics such as a chip-on-tip camera, an illumination source such as a light-emitting diode (LED), and an EM sensor optimally positioned according to the mechanical structure of the catheter. This can enable a simplified design of the handle section. For example, by using LEDs for illumination, termination in the handle section can be based solely on electrical soldering or wire crimping. For example, the handle section can include a proximal board on which the camera cable, LED cable, and EM sensor cable terminate, while the proximal board connects to an interface in the handle section and establishes an electrical connection to the instrument drive mechanism. As described above, the instrument drive mechanism is attached to the robot arm (robot support system) and provides mechanical and electrical interfaces to the handle section. This can advantageously improve assembly and packaging efficiency and simplify the manufacturing process and costs. In some cases, the handle section can be disposed of after a single use with the catheter.

[0100]

[0125] The robotic endoscope may have a compact configuration of electronic elements disposed in the distal portion. Designs for the distal tip / portion design, as well as navigation systems / methods, may include those described in International Application No. PCT / US2020 / 65999, entitled "Systems and Methods for Robotic Bronchoscopy," which is incorporated by reference herein in its entirety.

[0101]

[0126] While preferred embodiments of the present invention have been shown and described herein, it will be obvious to those skilled in the art that such embodiments are provided by way of example only. Numerous variations, changes, and substitutions will now occur to those skilled in the art without departing from the invention. It is understood that various alternatives to the embodiments of the invention described herein may be employed in practicing the invention. The appended claims define the scope of the invention, and it is intended that methods and structures within the scope of these claims and their equivalents be covered thereby.

Claims

1. 1. A method for navigating an endoscopic device through a network of anatomical lumens in a patient, the method comprising: (a) receiving, by one or more processors coupled to the endoscopic device, fluoroscopic image data including a radiopaque marker disposed at a distal end of the endoscopic device while the endoscopic device is moving; (b) the one or more processors reconstructing an orientation of the distal end of the endoscopic device by processing the fluoroscopic image data with a machine learning algorithm trained model; and A method comprising:

2. The method of claim 1 , wherein the orientation reconstructed by the one or more processors includes a roll angle of the distal end of the endoscopic device.

3. The method of claim 1 , wherein the machine learning algorithm comprises a deep learning network executed by the one or more processors.

4. The method of claim 3 , wherein the deep learning network has a U-Net architecture and is trained to estimate the orientation of the distal end of the endoscopic device.

5. The method of claim 1 , wherein the fluoroscopic image data includes an asymmetrically shaped radiopaque marker disposed at the distal end of the endoscopic device.

6. The method of claim 1 , wherein the distal end of the endoscopic device is articulatable and rotatable under the control of the one or more processors.

7. 1. An endoscopy system comprising one or more processors coupled to an endoscopy device, the one or more processors: (a) receiving fluoroscopic image data including a radiopaque marker disposed at a distal end of the endoscopic device while the endoscopic device is moving; (b) reconstructing an orientation of the distal end of the endoscopic device by processing the fluoroscopic image data with a machine learning algorithm trained model; An endoscopy system comprising:

8. The endoscopic system of claim 7 , wherein the orientation reconstructed by the one or more processors includes a roll angle of the distal end of the endoscopic device.

9. The endoscopic system of claim 7 , wherein the machine learning algorithm comprises a deep learning network executed by the one or more processors.

10. The endoscopic system of claim 9, wherein the deep learning network has a U-Net architecture and is trained to estimate the orientation of the distal end of the endoscopic device.

11. The endoscopic system of claim 7 , wherein the fluoroscopic image data includes an asymmetrically shaped radiopaque marker disposed at the distal end of the endoscopic device.

12. The endoscopic system of claim 7 , wherein the distal end of the endoscopic device is articulatable and rotatable under the control of the one or more processors.

Images