Self-Steering Intraluminal Devices Using Dynamically Deformable Luminal Maps

Abstract

The present invention relates to a self-steering endoluminal system comprising: an endoluminal device comprising a steerable elongate body; and a computer memory storage medium comprising one or more modules, the one or more modules including: a navigation module comprising instructions for generating navigation operations to be performed by the steerable elongate body of the endoluminal device to reach a desired location selected within a digital endoluminal map; a deformation module comprising instructions for evaluating potential deformations to one or more lumens caused by the navigation operations performed; a stress module comprising instructions for evaluating potential stress levels on the lumens caused by the navigation operations; and a higher level module comprising instructions for receiving information from one or more of the navigation module, deformation module, and stress module and generating instructions for driving, and optionally further driving, the steerable elongate body of the endoluminal device accordingly.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 242,101, filed September 9, 2021, and U.S. Provisional Patent Application No. 63 / 340,512, filed May 11, 2022, the contents of which are incorporated by reference in their entireties herein.

[0002] The present invention, in some embodiments thereof, relates to systems and methods for navigating one or more endoluminal devices, and more particularly, but not exclusively, to systems and methods for navigating one or more self-steering endoluminal devices.

[0003] In certain interventional procedures, the physician needs to reach a specific target tissue inside an endoluminal structure, e.g., the pulmonary bronchial tree, or e.g., the cerebral vasculature, or e.g., the digestive system, in order to obtain a biopsy sample or apply a localized treatment. To achieve this, it is standard technology to use an endoluminal tool, e.g., an intrapulmonary bronchoscope, or e.g., a catheter insertion kit for the vasculature. This is manually guided through the branching lumens according to real-time direct visual imaging, e.g., direct vision, or e.g., an angiogram. This is a tedious task, especially when the target is in a peripheral location and / or when the path to reach the location is a tortuous path. One obvious difficulty in driving the tool in such cases is the mechanical difficulty. For example, in the lungs, a standard bronchoscope is usually relatively thick (e.g., 6 mm in diameter) compared to the peripheral airways (e.g., 1 mm in diameter) through which it must be forced. Another problem concerns navigation. In direct visualization navigation, the physician must understand the location of the bronchoscope in the lungs based only on a video image. However, due to the fractal nature of airways, as they become smaller and smaller, it is often difficult to distinguish one airway from the other, increasing the likelihood that even experienced physicians will mistakenly choose the wrong turn and ultimately not reach the desired target.Similarly, in the vasculature, e.g., the cerebral vasculature, or, e.g., the hepatic vasculature, the structures are delicate, narrowed, and tortuous, making the use of standard angiograms to guide microcatheters and guidewires challenging and requiring years of training and specialization.

[0004] In recent years, it has become more common for bronchoscopists to use navigation bronchoscopy for peripheral vascular interventions within the lungs. Such procedures are typically performed using systems that provide 2D and / or 3D navigation renderings of the lungs based on CT or other near real-time imaging, on which instrument location references are displayed. Such systems thus assist the physician in guiding instruments such as bronchoscopes, endoscopes, or common catheters (with or without a camera) to the target location. Such guided instruments typically have the advantage of being small in diameter (e.g., 3-4 mm or less) compared to standard bronchoscopes. Such instruments typically have a working channel (e.g., 2 mm or more in diameter) that is wide enough to allow the physician to introduce biopsy and / or treatment tools into the target tissue once the desired location inside the anatomy has been reached.

[0005] Further background art includes European Patent EP2849669B1, which discloses a medical system comprising a processor and a surgical device including a tracking system disposed along a length of an elongated flexible body. The processor receives a first model of an anatomical passageway of a patient's anatomy. The first model includes a set of model passageways representing proximal and distal branches. The processor also receives from the tracking system a shape of the elongated flexible body located within the proximal and distal branches. The processor determines a set of forces acting on the patient's anatomy in response to the surgical device located within the proximal and distal branches based on the shape of the elongated flexible body. The processor also generates a second model by deforming the first model based on the set of forces and displays the second model and a representation of the elongated flexible body within the second model.

[0006] US10499993B2 discloses a processing system including a processor and a memory having computer readable instructions stored thereon. The computer readable instructions, when executed by the processor, cause the system to receive a reference 3D volumetric representation of a branched anatomical structure in a reference state, and obtain a reference tree of nodes and linkages based on the reference 3D volumetric representation. The computer readable instructions also cause the system to obtain a reference 3D geometric model based on the reference tree, and detect deformations of the branched anatomical structure due to anatomical motion based on measurements from a shape sensor. The computer readable instructions also cause the system to obtain a deformed tree of nodes and linkages based on the detected deformations, create a 3D deformation field representing the detected deformations of the branched anatomical structure, and apply the 3D deformation field to the reference 3D geometric model.

[0007] US10610306B2 discloses a method that includes determining a shape of a device that is at least partially located within an anatomical passageway. The method further includes determining a set of deformation forces for a plurality of sections of the device. Determining the set of deformation forces includes determining a stiffness of each section of the plurality of sections of the device. The method further includes generating a composite material model that indicates a position of the device relative to the anatomical passageway based on the shape of the device, the set of deformation forces (including an effect of each section of the plurality of sections on a respective portion of the anatomical passageway), and anatomical data describing the anatomical passageway.

[0008] US Patent No. 10,524,641 B2 discloses navigational guidance provided to an operator of an endoscope by determining the current position and shape of the endoscope relative to a reference frame, generating an endoscope computer model according to the determined position and shape, and displaying the endoscope computer model together with a patient computer model referenced to the reference frame for viewing by the operator while maneuvering the endoscope within the patient.

[0009] US Patent Application No. 20180193100A1 discloses an apparatus including a surgical instrument that can be attached to a robotic manipulator. The surgical instrument includes an elongated arm. The elongated arm includes an actively controlled bendable region including at least one joint region, a passively bendable region including a distal end coupled to the actively controlled bendable region, a drive mechanism extending through the passively bendable region and coupled to the at least one joint region to control the actively controlled bendable region, and a channel extending through the elongated arm. The surgical instrument also includes an optical fiber located within the channel. The optical fiber includes a fiber optic bend sensor within at least one of the passively bendable region or the actively controlled bendable region.

[0010] US9839481B2 discloses a system including a handpiece body configured to couple to a proximal end of a medical instrument and a manual actuator mounted within the handpiece body. The system further includes a plurality of drive inputs mounted within the handpiece body. The drive inputs are configured for releasable mating with the motorized drive mechanism. A first drive component is operably coupled to the manual actuator and to one of the plurality of drive inputs. The first drive component controls movement of the distal end of the medical instrument in a first direction. A second drive component is operably coupled to the manual actuator and to another one of the plurality of drive inputs. The second drive component controls movement of the distal end of the medical instrument in a second direction.

[0011] U.S. Patent No. 9,763,741 B2 discloses an intraluminal robotic system that gives the surgeon the ability to drive a robotically driven endoscopic device to a desired anatomical location within a patient without the need for skilled movements and positions, while enjoying improved image quality from a digital camera mounted on the endoscopic device.

[0012] US Patent Application No. US20110085720A1 discloses that alignment between a digital image of a branching structure and a real-time indicia representing the location of a sensor within the branching structure is achieved by using the sensor to "paint" the digital image of the structure's interior. Once sufficient location data is collected, alignment is achieved. The alignment is "automatic" in the sense that navigation through the branching structure necessarily results in the collection of further location data, resulting in continuous refinement of the alignment. Summary of the Invention

[0013] Below is a non-exhaustive list including some example embodiments of the present invention. The present invention also includes embodiments including fewer than all of the features in an example, and embodiments that use features from more than one example, even if not explicitly listed below.

[0014] Example 1. A method for generating a steering plan for a self-steering endoluminal system, comprising: a. selecting a location accessible through one or more lumens in a digital endoluminal map that a self-steering endoluminal device needs to reach; b. generating a navigation movement for the endoluminal device to reach the location; and c. assessing potential deformations to one or more lumens resulting from the navigational movements performed by the endoluminal device; and d. updating the steering plan according to the results of evaluating the potential deformations while the self-steering intraluminal system is reaching the location.

[0015] Example 2. The method of example 1, further comprising performing the navigation action until the location is reached.

[0016] Example 3. The method of example 1 or example 2, wherein the updating of the maneuver plan is performed in real time.

[0017] Example 4. The method of any one of Examples 1-3, wherein the method further comprises assessing a potential stress level on the lumen caused by the navigational motion performed by the endoluminal device.

[0018] Example 5. The method of example 4, wherein the method is performed until the potential stress level falls below a predetermined threshold.

[0019] Example 6. The method of any one of Examples 1-5, further comprising providing said plan to said self-steering intraluminal system.

[0020] Example 7. The method of any one of Examples 1-6, further comprising generating the digital endoluminal map including the one or more lumens based on the image.

[0021] Example 8 The method of example 7, wherein the image is a CT scan.

[0022] Example 9. The method of example 7, wherein the image is an angiogram.

[0023] Example 10. The method of any one of Examples 1-9, wherein generating a navigation behavior includes performing a first simulation of the navigation behavior.

[0024] Example 11. The method of any one of Examples 1-10, wherein evaluating the potential deformations includes performing a second simulation of the potential deformations.

[0025] Example 12. The method of Example 11, further comprising updating the digital endoluminal map according to the potential deformations simulated in the second simulation.

[0026] Example 13. The method of example 4, wherein assessing the potential stress level includes performing a simulation of the potential stress level.

[0027] Example 14. The method of example 13, further comprising updating the navigation behavior to reduce the potential stress level.

[0028] Example 15. The method of any one of Examples 1 to 14, wherein assessing the potential deformation further comprises assessing deformation caused by breathing, heartbeat, and other external factors.

[0029] Example 16. A self-steering intraluminal system, comprising: a. an intraluminal device comprising a self-steerable elongate body; b. a computer memory storage medium including one or more modules, said one or more modules comprising: i. a navigation module including instructions for generating navigation operations to be performed by the steerable elongate body of the endoluminal device to reach a desired location selected within a digital endoluminal map; and ii. a deformation module comprising instructions for evaluating potential deformations to one or more lumens resulting from the navigational actions performed by the steerable elongate body of the endoluminal device; and and iii. a higher level module containing instructions for receiving information from one or more of the navigation module and the deformation module and driving the steerable elongate body of the endoluminal device accordingly.

[0030] Example 17. The system described in Example 16, wherein the computer memory storage medium further includes a stress module including instructions for evaluating potential stress levels on the lumen caused by the navigational motion performed by the steerable elongate body of the intraluminal device.

[0031] Example 18. The system of Example 17, wherein the high level module further includes instructions for receiving information from the stress module and driving the steerable elongate body of the intraluminal device accordingly.

[0032] Example 19. The system of any one of Examples 16-18, wherein the intraluminal device includes one or more sensors for monitoring a location of the intraluminal device during the navigation operation.

[0033] Example 20. The system of example 19, further comprising an external transmitter for enabling said monitoring.

[0034] Example 21. The system of any one of Examples 16-20, wherein the navigation module includes instructions for generating navigation operations to be performed by the steerable elongate body of the endoluminal device to assist in reaching a desired location selected within a digital endoluminal map.

[0035] Example 22. A system described in any one of Examples 16-21, wherein the high-level module further includes instructions for generating a maneuver plan based on the received information.

[0036] Example 23. A system described in any one of Examples 16 to 22, wherein the high-level module further includes instructions for generating the digital endoluminal map including the one or more lumens based on an image.

[0037] Example 24. The system of Example 23, wherein the image is a CT scan.

[0038] Example 25. The system of example 23, wherein the image is an angiogram.

[0039] Example 26. The system of any one of Examples 16-25, wherein the navigation module further includes instructions for performing a first simulation of the navigation operation.

[0040] Example 27. The system of any one of Examples 16-26, wherein the deformation module further includes instructions for performing a second simulation of the potential deformation.

[0041] Example 28. The system of Example 27, further comprising updating the digital endoluminal map according to the potential deformations simulated in the second simulation.

[0042] Example 29. The system of example 17, wherein the stress module further includes instructions for performing a third simulation of the potential stress levels.

[0043] Example 30. The system of example 29, further comprising updating the navigation behavior to reduce the potential stress level.

[0044] Example 31. A system described in any one of Examples 16 to 30, wherein evaluating the potential deformations further includes evaluating deformations caused by breathing, heartbeat, and other external factors.

[0045] Example 32. The system of any one of Examples 16-31, wherein the intraluminal device includes one or more steering mechanisms including one or more pull wires, one or more pre-curved shafts, the one or more shafts having variable stiffness along a body of the one or more shafts, and one or more coaxial tubes.

[0046] Example 33. The system of Example 32, wherein one or more of the one or more pre-curved shafts and the one or more shafts having variable stiffness along a body of the one or more shafts are integral with one another.

[0047] Example 34. The system of Example 32, wherein the one or more steering mechanisms are configured to induce one or more steering actions including rotating a shaft, advancing / retreating a shaft, deflecting a tip of the device, and deflecting a portion of a shaft of the device.

[0048] Example 35. A method of generating a steering plan for a self-steering endoluminal system, comprising: a. selecting a location accessible through one or more lumens in a digital endoluminal map that a self-steering endoluminal device needs to reach; b. generating a navigation movement for the endoluminal device to reach the location; and c. assessing potential deformations to one or more lumens resulting from the navigational movements performed by the endoluminal device; and d. assessing potential stress levels on the lumen caused by the navigational motion performed by the endoluminal device; and e. performing steps b through d until the potential stress level falls below a predetermined threshold.

[0049] Example 36. The method of Example 35, further comprising providing the plan to the self-steering endoluminal system.

[0050] Example 37. The method of Example 35, further comprising generating the digital endoluminal map including the one or more lumens based on the image.

[0051] Example 38 The method of Example 37, wherein the image is a CT scan.

[0052] Example 39. The method of Example 37, wherein the image is an angiogram.

[0053] Example 40. The method of example 35, wherein generating a navigation behavior includes performing a first simulation of the navigation behavior.

[0054] Example 41. The method of example 35, wherein evaluating the potential deformations includes performing a second simulation of the potential deformations.

[0055] Example 42. The method of Example 41, further comprising updating the digital endoluminal map according to the potential deformations simulated in the second simulation.

[0056] Example 43. The method of example 35, wherein assessing the potential stress level includes performing a simulation of the potential stress level.

[0057] Example 44. The method of example 43, further comprising updating the navigation behavior to reduce the potential stress level.

[0058] Example 45. The method of Example 35, wherein assessing the potential deformation further comprises assessing deformation caused by breathing, heartbeat, and other external causes.

[0059] Example 46. A self-steering intraluminal system, comprising: a. an intraluminal device comprising a steerable elongate body; b. a computer memory storage medium including one or more modules, said one or more modules comprising: i. a navigation module including instructions for generating navigation operations to be performed by the steerable elongate body of the endoluminal device to reach a desired location selected within a digital endoluminal map; and ii. a deformation module comprising instructions for evaluating potential deformations to one or more lumens resulting from the navigational actions performed by the steerable elongate body of the endoluminal device; and iii. a stress module including instructions for assessing potential stress levels on the lumen caused by the navigational motion performed by the steerable elongate body of the endoluminal device; and iv. the self-steering endoluminal system comprising: a computer memory storage medium comprising: a high-level module comprising instructions for receiving information from one or more of the navigation module, deformation module, and stress module and driving the steerable elongate body of the endoluminal device accordingly.

[0060] Example 47. The system of Example 46, wherein the intraluminal device includes one or more sensors for monitoring a location of the intraluminal device during the navigation operation.

[0061] Example 48. The system of Example 47, further comprising an external transmitter for enabling said monitoring.

[0062] Example 49. The system of Example 46, wherein the navigation module includes instructions for generating navigation operations to be performed by the steerable elongate body of the endoluminal device to assist in reaching a desired location selected within a digital endoluminal map.

[0063] Example 50. The system of Example 46, wherein the high-level module further includes instructions for generating a maneuver plan based on the received information.

[0064] Example 51. The system of Example 46, wherein the high-level module further includes instructions for generating the digital endoluminal map including the one or more lumens based on an image.

[0065] Example 52. The system of Example 51, wherein the image is a CT scan.

[0066] Example 53. The system of Example 51, wherein the image is an angiogram.

[0067] Example 54. The system of Example 46, wherein the navigation module further includes instructions for performing a first simulation of the navigation operation.

[0068] Example 55. The system of Example 46, wherein the deformation module further includes instructions for performing a second simulation of the potential deformation.

[0069] Example 56. The system of Example 55, further comprising updating the digital endoluminal map according to the potential deformations simulated in the second simulation.

[0070] Example 57. The system of example 46, wherein the stress module further includes instructions for performing a third simulation of the potential stress levels.

[0071] Example 58. The system of Example 57, further comprising updating the navigation behavior to reduce the potential stress level.

[0072] Example 59. The system of Example 46, wherein evaluating the potential deformations further includes evaluating deformations caused by breathing, heartbeat, and other external factors.

[0073] Unless otherwise defined, all technical and / or scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this invention belongs. In the practice or testing of embodiments of the present invention, methods and materials similar or equivalent to those described herein can be used, but exemplary methods and / or materials are described below. In case of conflict, the patent specification, including definitions, will control. In addition, the materials, methods, and examples are illustrative only and are not intended to be necessarily limiting.

[0074] Some embodiments of the present invention will now be described, by way of example only, with reference to the accompanying drawings, in which: - specific reference will now be made to the drawings in detail, stressing that the particulars shown are for the purpose of illustrating, by way of example, embodiments of the invention; - in this connection, the description made with the drawings will make apparent to those skilled in the art how embodiments of the invention may be practiced; [Brief description of the drawings]

[0075] [Figure 1] 1 is a schematic diagram of an exemplary endoluminal system, according to some embodiments of the present invention. [Diagram 2] 1 is a schematic diagram of an exemplary intraluminal device, according to some embodiments of the present invention. [Figure 3a] FIG. 2 is a schematic diagram of an exemplary digital / virtual 3D volumetric image provided to NavNN, according to some embodiments of the present invention. [Figure 3b] FIG. 2 is a schematic diagram of an exemplary digital / virtual 3D volumetric image including camera sensor images provided to NavNN, according to some embodiments of the present invention. [Figure 4] 14A-14E are schematic diagrams of a typical sequence of driving actions based on real-time localization images generated in real-time during a procedure and processed by the NavNN module according to some embodiments of the present invention. [Diagram 5] 1 is a schematic diagram of an exemplary volumetric tessellation of a catheter using 3D pyramid primitives, according to some embodiments of the present invention. [Figure 6] 1A-B are schematic diagrams of exemplary 3D localization images centered according to different objects, according to some embodiments of the present invention; [Figure 7] 1A-B are schematic diagrams of exemplary undistorted and distorted localization images, according to some embodiments of the present invention. [Figure 8] 1 is a flowchart of an exemplary method for displaying correct 2D / 3D system views to reflect lumen deformation, according to some embodiments of the present invention. [Figure 9] 5a-d are schematic diagrams of exemplary operations performed by the DeformNN module, according to some embodiments of the present invention. [Figure 10] 1 is a schematic diagram of an exemplary endoluminal device with a tracking and navigation system, according to some embodiments of the present invention. [Figure 11] 4 is a flowchart of an exemplary method of using a system according to some embodiments of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0076] The present invention relates in some embodiments to systems and methods for navigating one or more intraluminal devices, such as, for example, an endoscope, or, for example, a miniature intraluminal robotic device, or, for example, an intravascular catheter, or, for example, an intravascular guidewire, and more particularly, but not exclusively, to systems and methods for navigating one or more self-steering intraluminal devices. In some embodiments, when two or more devices are used, they navigate simultaneously. In the following paragraphs, the invention is described using a single device, but it should be understood that the same description applies when multiple devices are used simultaneously.

[0077] In some embodiments, the instrument is tracked in real-time or near real-time to display the location of the endoluminal device on the navigation map. In some embodiments, various methods can be used to identify the location of the instrument and display its location on the navigation map, including electromagnetic single sensor, multi-sensor, fiber optics, fluoroscopic visualization, and the like. For example, in some embodiments, the instrument has a single tracking sensor (e.g., electromagnetic sensor) at the tip of the catheter to provide 6-DOF position and orientation (also referred to as "location", hereafter meaning both position and orientation) to the navigation system. The terms "catheter", "endoscope", and "endoluminal device" mean the same thing, a device used inside a lumen, and are used interchangeably herein. The term "navigation map" refers to a representation of anatomy that may be based on various modalities or detection methods (e.g., CT, CTA, angiogram, MR scan, ultrasound, 3D ultrasound reconstruction, fluoroscopic imaging, tomosynthesis reconstruction, OCT, etc.). In some embodiments, the tip location is displayed in the navigation 2D / 3D view in registration with the patient's anatomy. The term "registration" refers to the process of transforming different data sets into one coordinate system unless otherwise indicated. Thus, in some embodiments, the physician can see a representation of the tip of the catheter located inside, for example, the lung, or inside, for example, the cerebral blood vessels, and can steer the catheter to the desired target, which is usually also displayed in the presented view. In some embodiments, the shape of the catheter is sensed using a "shape sensor" that may be based on optical fibers. In some embodiments, monitoring of the catheter's shape is performed using other means, for example, using RFID technology that allows monitoring of the device without the need for active transmission from within the endoluminal device, or by reconstructing its 3D shape using one or more fluoroscopic projections in near real time. In some embodiments, reconstructing the 3D shape of the device from the fluoroscopic projections is performed by:Identifying the tip and / or full curve of the device in multiple fluoroscopic 2D projections, identifying the location of the fluoroscope in some reference coordinate system (e.g. using optical fiducials), finding the 3D location and / or shape of the device by optimization so that the back projected 2D device curve fits the 2D curve observed from the fluoroscopic projections. In some embodiments, the shape of the catheter is shown to the physician in a 2D / 3D view, registered to the patient's anatomy. In some embodiments, the catheter may include multiple position sensors (e.g. electromagnetic) that allow tracking of the position and absolute shape of the full catheter relative to some reference transmitter. In some embodiments, the catheter may not include any sensors. In some embodiments, it may be a passive catheter visible under fluoroscopy. In some embodiments, the shape of the catheter is tracked using fluoroscopy with a reconstruction method using one or more fluoroscopic projections. In some embodiments, the shape and location of the catheter is then displayed to the physician, registered to the patient's anatomy. In some embodiments, a combination of these methods is used.

[0078] In some embodiments, various 2D / 3D views are used to display the catheter location relative to a navigation map. In some embodiments, the views are used by the physician to determine how to steer the catheter to reach the target. In some embodiments, the views optionally display a pre-planned path from the entry point to the target. In some embodiments, during the intervention, the physician articulates and drives the catheter tip close to the target, following the path while watching the instrument movement tracked in real time on the displayed views.

[0079] In some embodiments, various mechanisms can be used to drive the instrument to the desired location. In some embodiments, the mechanism is manually driven and operated by the physician, with one or more levers providing articulation of the tip of the catheter. In some embodiments, the catheter may be manually inserted with a fixed curve at the distal end. In some embodiments, the catheter is attached to a robotic drive mechanism controlled by a remote control panel. In some embodiments, at any point, particularly when reaching the desired target, the robotic drive mechanism may fix the catheter in space or in the anatomy, eliminating the need to hold the catheter and allowing stable insertion of tools through the working channel without changing the position and orientation of the catheter. In some embodiments, a potential advantage of fixing the catheter in the anatomy is that in some cases, fixing the catheter "in space" is not sufficient because the anatomy moves relative to any fixed point in space (e.g., when the patient is breathing), and therefore it is potentially useful to fix the catheter "in the anatomy", i.e., to move it automatically in space and maintain its position relative to the anatomical target, regardless of patient movement / breathing or tissue movement, deflection or deformation.

[0080] There are many cases where pushing an instrument past a lumen wall can be harmful or at least require caution. In the case of a manual instrument, when pushed against a barrier, a resistance force is transmitted back to the handle of the catheter, from where it is sensed by the physician. A trained physician will be aware of the risks and will manipulate the catheter carefully. If the resistance force increases beyond what the physician determines to be excessive, the physician may relive the pressure and pull the catheter. Once the catheter is retracted, the physician may change the direction of the tip and slowly push it forward towards the target. However, in the case of mechanical, electromechanical, and / or power-driven catheters, or in the case of very long catheters, the physician may not be able to sense these forces, increasing the risk of injuring the patient or damaging the catheter. Thus, in some embodiments, the system includes mechanisms to replace the lost natural force feedback, for example using force sensors and mechanical tracking.

[0081] overview An aspect of some embodiments of the present invention relates to a system and method for navigating an endoluminal device, e.g., a bronchoscope, or e.g., an endovascular device, e.g., a guidewire, or e.g., a microcatheter, or e.g., a catheter, or e.g., an embolus retrieval tool, or e.g., a coiling tool, using a virtual dynamically deformable luminal map. In some embodiments, the navigation is performed automatically by the system using a self-steering endoluminal device. In some embodiments, the navigation and navigation updates are performed in real time while the endoluminal device is moving forward towards the desired location. In some embodiments, the deformations are tracked in real time by a deformation-aware tracking system as a product of real-time tracking of the complete location and / or shape of the endoluminal device inside the patient and converted into a virtual dynamically deformable map. In some embodiments, an informative 3D localization image is generated in real time from the fully tracked endoluminal device or from multiple fully tracked endoluminal devices and the virtual dynamically deformable map (including the current real-time location and complete shape of the device). In some embodiments, the localization image encodes all the information necessary for a competent human and / or an intelligent machine (AI) to determine the optimal driving action (e.g., steering, forward or reverse movement) required at any location to reach the target. In some embodiments, the localization image can be processed by a Navigation Neural Network (NavNN) module to generate an intelligent driving action. In some embodiments, the undeformed localization image may be used first and a Deformation Neural Network (DeformNN) module may be used to find the deformations and thus generate a deformed localization image for navigation. In some embodiments, the system and / or method is versatile and can be used, for example, to perform a completely autonomous navigation from start to finish, or in another example, the navigation may be divided into smaller human-supervised steps (e.g., controlled by an intuitive "tap-to-drive" user interface).The user interface performs autonomous navigation, for example, from a current location in the anatomy to a commanded location (e.g., “tapped” on a touch screen interface). In some embodiments, the system and / or method may be used to display recommended navigation instructions to a human physician. In some embodiments, the system and / or method may be used in a self-steering endoscope, where the tip of the endoscope is automatically aligned with a path to the target, and the physician only needs to advance the tip distally or proximally along the patient's airway. In some embodiments, the system and / or method may be used with any intravascular device (e.g., catheter, guidewire, tool, or other) with an attached drive with self-steering capabilities, where the drive automatically aligns the tip of the intravascular device with a pre-planned path, and the physician only needs to wilt and advance the tip distally or proximally inside the blood vessel, either manually or using the drive. In some embodiments, the system and / or method is suitable for collecting training data to improve AI performance (e.g., teaching one or more neural network modules, as described further below). In some embodiments, autonomous driving operations are monitored by additional safety mechanisms to ensure safe operation of the device within the body.

[0082] An aspect of some embodiments of the invention relates to a system that rasterizes 3D pyramid primitives onto a 3D render target and uses it to render real-time 3D localization images in a navigation procedure. In some embodiments, the method is optionally implemented in a GPU ASIC / FPGA. In some embodiments, the method is optionally exposed to developers through OpenGL extensions or by DirectX. In some embodiments, the method is optionally used to render real-time 3D composite data for processing by a 3D neural network. In some embodiments, the method is optionally used to render real-time 3D images of tracked hands and fingers.

[0083] Aspects of some embodiments of the invention relate to systems and / or methods for encoding and optionally displaying navigation data in a 3D multi-channel localization image, optionally in a virtual 3D multi-channel localization image. In some embodiments, optionally one of the channels includes a segmented luminal structure. In some embodiments, optionally the segmented luminal structure is binary. In some embodiments, optionally the segmented luminal structure is a scalar likelihood map. In some embodiments, optionally the segmented luminal structure is deformed using a deformation neural network module. In some embodiments, optionally the segmented luminal structure is represented by its skeleton. In some embodiments, optionally one of the channels includes raw CT data, raw MRI data, raw angiogram data, and any combination thereof. In some embodiments, optionally one or more channels include a catheter at its estimated location inside the body. In some embodiments, optionally the catheter is represented as a full curve or a partial curve. In some embodiments, optionally the catheter is represented only by its tip. In some embodiments, optionally the catheter renders in its deformed position within the anatomical structure. In some embodiments, optionally the catheter renders in its non-deformed position within the anatomical structure. In some embodiments, optionally one of the channels includes a path to the target. In some embodiments, optionally one of the channels includes a segmented target. In some embodiments, optionally one of the channels includes a target sphere. In some embodiments, optionally one of the channels includes an image of an endoscopic camera. In some embodiments, optionally the image is 2D and is rendered in the 3D localized image using backprojection along corresponding rays. In some embodiments, optionally the image includes a depth channel and is rendered in the 3D localized image using the depth channel as a 3D surface.In some embodiments, the localization image optionally has a special position and arrangement. In some embodiments, the localization image is optionally centered on the tip of the catheter. In some embodiments, the localization image is optionally centered on the path. In some embodiments, the localization image is optionally centered on the closest path point. In some embodiments, the localization image is optionally aligned with the catheter tip direction. In some embodiments, the X-axis of the localization image is optionally aligned with the catheter tip direction. In some embodiments, the X-axis of the localization image is optionally aligned with the path direction. In some embodiments, the Z-axis of the localization image is optionally aligned with the normal vector of the next branch point. In some embodiments, the 3D localization image input is optionally generated in real time. In some embodiments, the localization image is optionally rendered using a 3D pyramid tessellation technique. In some embodiments, the segmented lumen structure is optionally rendered in its deformed state in real time as calculated by the deformation-aware localization system. In some embodiments, optionally the segmented lumen structure is rendered in its deformed state in real time using a deformation neural network. In some embodiments, optionally the catheter position is rendered in its position as calculated by the tracking system. In some embodiments, optionally the catheter position is rendered in its anatomical deformation compensated position using a deformation neural network module.

[0084] Aspects of some embodiments of the invention relate to systems and / or methods for generating automatic navigation driving behavior. In some embodiments, the localization images are optionally processed using a Navigation Neural Network (NavNN) module. In some embodiments, the localization images are optionally processed using a 3D Convolutional Neural Network (3D CNN). In some embodiments, the localization images are optionally processed using a 3D Recurrent Neural Network (3D RNN). In some embodiments, the localization images optionally include a camera channel for generating better driving behavior. In some embodiments, the NavNN optionally possesses memory. In some embodiments, the NavNN optionally carries a state vector between predictions. In some embodiments, a high-level module optionally operates the NavNN. In some embodiments, the high-level module optionally selects an optimal driving behavior by selecting a maximum output of the NavNN. In some embodiments, the high-level module optionally automatically activates a motor based on the NavNN output. In some embodiments, optionally the high level module periodically generates random drive motions to add exploration to the navigation and avoid local extreme points in the NavNN output. In some embodiments, optionally the high level module automatically rolls the catheter at certain predefined time intervals. In some embodiments, optionally hysteresis is used in the NavNN output to prevent "jumps" between different output drive motions. In some embodiments, optionally a safety mechanism is enforced in the NavNN output to prevent harmful drive motions. In some embodiments, optionally the catheter does not push if a certain force is exerted on the patient. In some embodiments, optionally the catheter automatically pulls back if a certain force is exerted on the patient. In some embodiments, optionally the exerted force is calculated by analyzing the complete catheter curve inside the segmented lumen structure.In some embodiments, the applied force is optionally sensed by a force sensor in the catheter handle or along the catheter body. In some embodiments, the NavNN is optionally trained in supervised training using 3D localized image inputs labeled with corresponding driving behavior. In some embodiments, the labeled samples are optionally generated using a realistic simulator module. In some embodiments, the labeled samples are optionally collected from a real robot navigation procedure. In some embodiments, the labeled samples are optionally collected from an actual manual navigation procedure. In some embodiments, the operator's manual driving behavior is optionally classified automatically. In some embodiments, the driving behavior is optionally classified using proximal and distal catheter sensors. In some embodiments, the catheter handle optionally includes one or more sensors for classifying the operator's actions. In some embodiments, the NavNN is optionally trained in unsupervised learning using 3D localized image inputs. In some embodiments, the NavNN is optionally trained in a realistic simulator module using reinforcement learning.

[0085] Aspects of some embodiments of the invention relate to systems and / or methods for locating an anatomical location of a catheter inside a deformed luminal structure (see below for further description of "deformed luminal structure"). In some embodiments, the localization images are optionally processed using a Deformation Neural Network (DeformNN) module. In some embodiments, the localization images are optionally processed using a 3D CNN. In some embodiments, the localization images are optionally processed using a 3D RNN. In some embodiments, the localization images are optionally processed using a 3D U-Net. In some embodiments, the localization images optionally include a camera channel for improved accuracy. In some embodiments, the DeformNN optionally possesses memory. In some embodiments, the DeformNN optionally carries a state vector between predictions. In some embodiments, the DeformNN optionally outputs an image of a deformation compensated luminal structure. In some embodiments, the DeformNN optionally outputs an image of a catheter in an anatomical location inside an input luminal structure. In some embodiments, DeformNN optionally outputs images of one or more virtual catheters in anatomical locations inside the luminal structure along with their corresponding confidence levels. In some embodiments, DeformNN optionally outputs a single probability for each catheter reflecting the confidence of the input catheter at its location in the input luminal structure. In some embodiments, the deformation of the luminal structure is optionally searched to maximize the output probability of DeformNN. In some embodiments, a high-level module optionally operates DeformNN. In some embodiments, if a deformed luminal structure is output, the input and output luminal structures are optionally aligned to calculate a deformation vector. In some embodiments, if a deformed catheter curve is output, the input and output catheters are optionally aligned to calculate a deformation vector. In some embodiments, the deformation vector is optionally applied to the complete luminal structure or catheter positions to display a deformation-compensated system view.In some embodiments, the part localization image output of DeformNN is optionally manipulated by missing channels and input to NavNN to generate the automated driving behavior. In some embodiments, DeformNN is optionally trained in supervised training using 3D localization image input labeled with corresponding deformation compensated output images. In some embodiments, the labeled samples are optionally generated using a realistic simulator module. In some embodiments, the deformation of the luminal structure is optionally simulated in the simulator module using a realistic deformation model. In some embodiments, the deformation of the luminal structure is optionally simulated in the simulator using polynomial, spline, or rigid 3D transformation. In some embodiments, the labeled samples are optionally collected from an actual manual navigation procedure. In some embodiments, one or more catheters are optionally inserted into known anatomical locations (e.g., peripheral locations), the anatomical structure is deformed by applying internal and external forces, and the deformation of the luminal structure is recorded. In some embodiments, a trackable sensor is optionally placed inside the organ to record the deformation. In some embodiments, multiple CBCT (Cone Beam CT) scans are optionally performed and registered using deformable registration to calculate deformation vectors. In some embodiments, DeformNN is optionally further trained on the luminal anatomy of a particular patient prior to the procedure.

[0086] Aspects of some embodiments of the invention relate to systems and / or methods for displaying multiple catheter hypotheses in a navigation procedure. In some embodiments, two or more catheter hypotheses are optionally displayed inside the luminal structure on a 2D / 3D view with different opacity or intensity based on confidence level. In some embodiments, a single catheter is optionally displayed up to a point where it splits into different directions for different hypotheses. In some embodiments, shared segments of the catheter hypotheses are optionally displayed normally while splitting segments are displayed with different colors, intensities or opacities. In some embodiments, the screen is optionally split into multiple independent displays of different catheter hypotheses when the catheter position is ambiguous. In some embodiments, optionally, a screen reverts to a single winning hypothesis, with the winning half of the screen "pushing" the losing half of the screen out of view.

[0087] Aspects of some embodiments of the invention relate to systems and / or methods for calculating a risk estimate of a catheter force inside a luminal structure. In some embodiments, the force risk estimate is optionally calculated using a fully tracked position of the catheter inside the luminal structure. In some embodiments, the force risk estimate is optionally calculated by estimating contact forces and internal catheter forces. In some embodiments, the force risk estimate is optionally calculated using StressNN by providing a 3D localization image visualizing the catheter inside the luminal structure. In some embodiments, the StressNN is optionally trained with labeled examples generated using a realistic simulator module. In some embodiments, the force risk estimate is optionally calculated in the simulator module using physically simulated force estimates.

[0088] An aspect of some embodiments of the invention relates to a system and method for self-steering an endoluminal device, optionally wirelessly, optionally disposable, using real-time 3D localization images. In some embodiments, the device optionally wirelessly pairs with the patient in a pre-procedure pairing process. In some embodiments, the patient's data (segmented luminal structure, target plan, etc.) is optionally transferred to the device using NFC or any other wireless method. In some embodiments, the device optionally applies deformation compensation to the segmented luminal structure or catheter. In some embodiments, the deformation compensation is optionally performed using a skeletal deformation model and optimization method. In some embodiments, the deformation compensation is optionally performed using DeformNN. In some embodiments, the device optionally uses NavNN to generate accurate automated driving movements and feedback. In some embodiments, the device optionally uses a small motor in the handle to automatically rotate the catheter based on the NavNN output (particularly useful when utilizing passive J-catheters) to align the catheter to a path to the target. In some embodiments, the device optionally uses a small actuator (e.g., in the handle) to automatically push or pull the intraluminal portion of the device to advance the device in either direction relative to the target. In some embodiments, the device optionally uses LED or vibration motor feedback to provide instructions to the operator during navigation. In some embodiments, the device is optionally handheld, with the pushing / pulling motion carried by the operator following the device's commands. In some embodiments, the device is optionally mounted in a robotic drive mechanism to drive autonomously without human mechanical intervention. In some embodiments, the automatic navigation optionally stops based on a force risk estimate.

[0089] An aspect of some embodiments of the invention relates to a system and / or method for controlling an intraluminal device driven by indicating a destination. In some embodiments, the driving function is accomplished, for example, by using an electromechanical device. In some embodiments, the intraluminal device is advanced within the lumen using other driving methods, for example, by applying a magnetic field to a magnetized device, or, for example, by using pneumatic or hydraulic pressure to drive the device, or by other methods. In some embodiments, the operator optionally navigates the instrument tip to a location within the anatomy by indicating a desired final position and orientation of the instrument tip. In some embodiments, the destination is optionally marked by tapping a point in a 3D map representing the organ displayed on a touch screen. In some embodiments, the destination is optionally marked by clicking a mouse pointer at a location on a computer screen displaying an anatomical imaging, for example, a CT slice, or, for example, an angiogram, or, for example, an ultrasound, or, for example, an MRI. In some embodiments, the destination is optionally marked by selecting a predefined location from a menu or other user interface (UI) element. In some embodiments, the destination is optionally automatically suggested by the system. In some embodiments, the destination is optionally indicated by issuing a voice command. In some embodiments, the destination is optionally indicated on a multi-waypoint curved plan view map similar to a progress bar. In some embodiments, the waypoints are optionally obtained by performing limited operations sequentially according to their order on the map. In some embodiments, a "magnifying glass" view is optionally used to indicate the exact destination within the area of ​​interest. In some embodiments, a "first person" view is optionally used to indicate the exact destination within the area of ​​interest. In some embodiments, the system is optionally triggered to stop moving forward according to a predetermined maximum travel distance. In some embodiments, a deadman switch is optionally used to stop operation of the device.In some embodiments, a "stabilize in anatomical structure" mechanism is optionally used to actively prevent the tip from crossing a determined proximity to the determined structure using motorized micromovements and adjustments.

[0090] Before describing at least one embodiment of the invention in detail, it is to be understood that the invention is not necessarily limited in its application to the details of construction and arrangement of components and / or methods set forth in the following description and / or illustrated in the drawings and / or examples. The invention is capable of other embodiments or of being practiced or carried out in various ways.

[0091] Typical Endoluminal System Referring now to FIG. 1, a schematic diagram of an exemplary endoluminal system according to some embodiments of the present invention is shown. In some embodiments, the endoluminal system 100 includes an endoluminal device 102 (e.g., an endoscope) configured for endoluminal intervention. In some embodiments, the endoluminal device 102 is connected to a computer 104. The computer 104 is configured to monitor and control operations performed by the endoluminal system 100, including, in some embodiments, self-steering operations of the endoluminal device 102. In some embodiments, the endoluminal system 100 further includes a transmitter 106. The transmitter 106 is configured to generate an electromagnetic field used by the endoluminal system 100 to monitor the location of the endoluminal device 102 inside the patient 108. In some embodiments, the endoluminal system 100 further includes a display unit 110. The display unit 110 is configured to show a dedicated image to an operator, which potentially assists the operator during navigation of the endoluminal device 102 during endoluminal intervention. In some embodiments, the endoluminal system 100 optionally further includes one or more sensors 112 configured to monitor movement of the patient 108 during the endoluminal intervention. In some embodiments, the patient movement is used to aid in navigation of the endoluminal device 102 within the patient 108. Each of the aforementioned exemplary portions of the endoluminal system 100, and exemplary methods thereof, are further described below.

[0092] Exemplary Intraluminal Device 102 and Its Tracking System 2, a schematic diagram of an exemplary endoluminal device is shown, in accordance with some embodiments of the present invention. In some embodiments, the endoluminal system 100 includes an endoluminal device 102 (e.g., an endoscope). In some embodiments, the endoluminal device 102 includes a handle 202 and an elongated body 204. In some embodiments, the endoluminal device 102 includes a plurality of sensors 206 along the elongated body 204. The plurality of sensors 206 are configured to detect transmitted signals from the transmitter 106. In some embodiments, the endoluminal system 100 monitors the location of the elongated body 204 using the plurality of sensors 206.

[0093] In some embodiments, the number of sensors 206 is one or more of a 3-axis accelerometer, a 3-axis gyroscope, and a 3-axis magnetometer. In some embodiments, the number of sensors 206 is a digital sensor. In some embodiments, the number of sensors 206 is an analog sensor that includes an additional A2D element to transmit sensed analog data in a digital data format. In some embodiments, the number of sensors 206 is a combination of digital and analog sensors.

[0094] In some embodiments, the elongated body 204 includes a flexible printed circuit board (PCB) disposed within and / or along the elongated body 204. Further information can be found in International Publication No. WO2021048837, the contents of which are incorporated herein in their entirety. In some embodiments, the PCB is communicatively connected to the microcontroller by the same data bus, for example, including a small number of wire lines, for example, 2-4 wires. For example, an Inter-Integrated Circuit (I2C) is used as a digital connection interface between the microcontroller and the multiple sensors 206 located along the flexible PCB. In some embodiments, the intraluminal device requires only two wires to exchange data between the sensors and the microcontroller. In some embodiments, a potential advantage of having such a small number of wires is the ability to keep the number of wires in the catheter, which needs to be kept small.

[0095] In some embodiments, there may be 8, 5, 10, or any suitable number of sensors mounted on the flexible PCB, e.g., all connected to the same I2C bus (e.g., serial data and serial clock lines). In some embodiments, the microcontroller is connected to the flexible PCB using a 4-wire shielded cable (e.g., including a voltage and / or ground wire). In some embodiments, the microcontroller provides voltage and / or ground for the digital sensors, in addition to two data lines, e.g., for reading digital measurements by the sensors. In some embodiments, the microcontroller reads the sensors, e.g., sequentially and / or simultaneously, and sends the sensor readings to the computer 104, e.g., via wired and / or wireless communication.

[0096] In some embodiments, the design of and / or the positioning of the sensor thereon provides the location and / or orientation of the sensor, e.g., when the PCB is straight. In some embodiments, e.g., during a manufacturing process, the PCB is mounted within and / or along the elongated body 204 in a manner that determines the location and / or orientation of the sensor 206 relative to the elongated body 204. In some embodiments, the computer 104 may be calibrated with the initial 6DOF orientation and / or position of the sensor, e.g., the 6DOF orientation and / or position of the sensor when the elongated body 204 is straight. In some embodiments, the initial 6DOF orientation and / or position data is incorporated into the catheter localization algorithm as a geometric constraint, along with information regarding the stiffness and / or flexibility limitations of the elongated body 204. In some embodiments, e.g., based on the incorporated geometric constraint, two adjacent sensors cannot point in opposite directions. In some embodiments, a potential advantage of utilizing geometric constraints in the calculations is that more advanced localization algorithms that take geometric constraints into account are potentially provided, potentially making the system 100 both compact and robust. In some embodiments, solving for the positions and / or orientations of all 6 DOF of sensors while imposing physical shape constraints on the overall curvilinear shape of the elongated body 204 potentially reduces the number of parameters of the motion model, thus potentially preventing, for example, overfitting of the measurement data. In some embodiments, a further potential advantage of using shape constraints is that the computer 104 may stop miscalculating the position and / or orientation of any sensor due to noisy or distorted measurements, because the position and / or orientation solutions should, for example, conform to the position and / or orientation solutions of adjacent sensors, for example, so that they collectively describe a smooth, physically plausible elongated body 204.

[0097] In some embodiments, the computer 104 takes into account dynamic electromagnetic distortions, for example by incorporating the distortions into the localization algorithm to obtain an accurate solution. Different methods used to compensate for dynamic magnetic distortions are described in International Patent Publication WO2021048837, the contents of which are incorporated herein by reference in their entirety.

[0098] Exemplary Methods for Monitoring Intraluminal Devices As mentioned above, exemplary methods for monitoring intraluminal devices are described elsewhere, for example, in International Patent Publication No. WO2021048837, the contents of which are incorporated herein by reference in their entirety.

[0099] Briefly, in some embodiments, the computer 104 receives data from the transmitter 106 regarding the instantaneous phase of the generated alternating electromagnetic field. In some embodiments, the computer 104 receives local magnetic field sensed by a number of sensors 206 along the elongated body 204 that sense the magnetic field generated by the transmitter 106. In some embodiments, the number of sensors 206 sense the generated magnetic field in its local coordinate system, and thus in some embodiments rotate the magnetic field reading according to its orientation relative to the transmitter 106. In some embodiments, the computer 104 then correlates between the transmitter data and the magnetic field sensed from the sensors. In some embodiments, the computer 104 then calculates the position and orientation of the number of sensors that gave the sensed magnetic field value, e.g., the 6DOF or 5DOF localization of each of the sensors, and / or the overall position, orientation, and / or curvature of the elongated body 204 based on the transmitter data and the sensed magnetic field from the sensors. In some embodiments, the computer 104 optionally uses accelerometer and / or gyroscope readings of corresponding sensors in the plurality of sensors 206 for localization calculations. In some embodiments, the electromagnetic field frequency of the transmitter 106 is limited to about 10 Hz to about 100 Hz. Optionally, about 10 Hz to about 200 Hz. Optionally, about 10 Hz to about 500 Hz. Optionally, about 10 Hz to about 1000 Hz. In some embodiments, the computer 104 utilizes a mathematical model to describe the movement of the elongated body 204. In some embodiments, the computer 104 tracks each of the plurality of sensors 206 independently. For example, the computer 104 is configured to predict the state of each of the plurality of sensors 206 in the next time frame, for example, based on the state in the current time frame and / or based on bundle measurements of an inertial measurement unit (IMU) (which provides information on the movement and attitude of the device) that may be used to correct the prediction.In some embodiments, the computer 104 utilizes known structural relationships between the multiple sensors 206 in its catheter localization algorithm to calculate an estimate of the position, orientation, and / or curvature of the elongate body 204 as a whole, for example, rather than calculating the position and / or orientation for each of the multiple sensors 206 separately.

[0100] Typical principles of advanced monitoring systems In some embodiments, the present invention relates to a system that utilizes an advanced monitoring system that provides guidance, and in some embodiments, auto-steering (described further below), to the endoluminal device.

[0101] During a navigational bronchoscopy procedure, the physician needs to select the optimal driving action to perform on the catheter according to the data shown in the system view in order to bring the catheter closer towards the target. Using both handheld as well as robotic driving mechanisms, the physician either manually or remotely manipulates (e.g., pushes / pulls / rolls / curves) the handle of the catheter to try to "improve" the state of the catheter shown in the view. The term "state" in this context refers to the relative position of the device along a predefined path towards the desired location inside the patient's body. The more "on track" the device is, the better the "state" of the device is with respect to the desired target location. As a simple example, when the catheter is located in front of the main carina (the first carina that connects the left and right lungs), assuming that the desired target is located in the left lung, the physician needs to articulate (e.g., by rolling / curving) the tip of the catheter in the correct direction to push the catheter to the left lung. After doing so, the catheter is displayed in the left lung in the real-time system view, so that the state of the catheter is actually improved. Alternatively, if the physician mistakenly pushes the catheter to the right lung, the catheter status has worsened because the catheter appears in the right lung and is further away from the desired target path as displayed in the system view. The physician realizes that the catheter is now further away from the path to the target and pulls the catheter back and re-navigates to the correct lung, improving the status of the catheter relative to the destination target.

[0102] Being able to estimate the state of the catheter relative to the destination target and the path of the target may seem a trivial task, but it may prove to be very tricky whenever the state, map or direction of the catheter is not fully known. There are several reasons why these important factors are unknown, not least of all the dynamic deformation of the tissue. Dynamic deformation of the tissue may be caused by many forces (organic or inorganic). For example, bending the catheter may exert forces on the tissue, causing dynamic deformations and moving the airway with the catheter. It is noted that some systems are not able to compensate for this dynamic deformation. In these systems, the airway map displayed is fixed from the beginning of the procedure and does not take into account changes due to breathing, forces dynamically applied during the procedure (such as in the described case), atelectasis induced by anesthesia, cardiac motion, pneumothorax, etc.

[0103] Even when decisions are made according to perfect map conditions, problems with maneuver execution may prevent navigation, necessitating repeated trial and error. For example, frictional forces may prevent the catheter from advancing into the desired airway. It is then up to the skilled physician to carefully interact with the catheter, closely watch the real-time catheter movement in the system view, and try to redirect the catheter tip in different directions, pulling and pushing the catheter until it advances in the correct airway toward the target. In another example, the physician may wish to advance the catheter toward the upper lobe. To do so, the catheter tip needs to be articulated at a sharp angle. However, when pushed, the catheter may slip and advance forward toward the middle lobe, missing the turn. The skilled physician may then pull back and advance the catheter, trying different levels of curvature in different directions (some of which are significant) and speeds, until it somehow enters the upper lobe.

[0104] As can be seen by the above examples, the problem of selecting the optimal drive motion is not trivial and does not necessarily correspond to simple geometric reasoning. What may seem to be an optimal choice of motion from a geometric point of view may turn out to be unhelpful in practice, requiring the skilled physician to repeat different motions until the catheter is correctly advanced. It is also important to note that in all the above examples, it was assumed that the physician would be shown a complete 2D / 3D view by the system that would allow the physician to fully understand in real time the 3D status of the catheter relative to the surrounding 3D airway and the path to the target. However, the view is inherently limited by the fact that the human eye can only sense a 2D projection image shown, for example, by a 2D monitor. In some cases, stereoscopic views are generated and displayed separately to each eye using special headsets or glasses that create the effect of 3D perception, but the displayed data is still essentially 2D and is merely a 2D projection (or multiple projections) of the raw 3D data observed by a virtual camera in a 3D world. Because the system view is essentially a 2D projection of the raw 3D data, it may suffer from problems such as occlusion (e.g., one airway occludes another from the virtual camera's perspective), false depth perception (e.g., the distance between two features appears much smaller in the projection than it actually is). To overcome this, the view is designed to allow a skilled physician to "complete the picture" using his imagination and 3D perception. For example, to overcome occlusion, an automatic camera positioning algorithm is used to place the virtual camera in an optimal position with minimal occlusion, and by automatically moving the camera, the observer perceives the 3D position to some extent. However, the ultimate understanding of the true 3D structure of the displayed features (e.g., the catheter, the surrounding airways, and the path to the target) relies on the skilled physician's 3D perception capabilities, making the system less usable for general users.

[0105] In some embodiments, the system of the present invention includes a self-steering endoscope, which may be, for example, handheld. In some embodiments, the physician holds the endoscope and slides it into the patient's airway. In some embodiments, the tip of the endoscope is automatically steered to align with the next bifurcation point, and the physician only needs to push the endoscope forward, optionally at a specific and pre-defined speed. In some embodiments, the auto-steering of the endoscope is powered by a Navigation Neural Network (NavNN) module, which is fed with a virtual dynamic localization image and generates output driving actions / commands.

[0106] For further explanation of NavNN, see below.

[0107] In some embodiments, the roll and deflection drive commands are translated into mechanical manipulations using small motors or other actuators inside the handle of the endoscope. In some embodiments, the user is then optionally given navigational feedback (e.g., push / pull back) and, with the aid of the NavNN, the user can safely and easily reach the desired target. In some embodiments, the catheter may be optionally attached to a fully robotic drive mechanism and navigated to the target using a tap-to-drive user interface. In some embodiments, instead of manually steering the catheter with a remote control, the physician is provided with a screen showing the catheter at its location along the path to the target. In some embodiments, the physician then taps the next closest branch or waypoint along the path and the robot performs the necessary drive movements to advance the catheter from its current position to the next waypoint based on the output from the NavNN. In some embodiments, the maneuvers performed are relatively short and can be monitored by the physician operator. In some embodiments, once the next waypoint is reached, the physician then commands the robot to perform the next maneuver sequentially until the target is reached. In some embodiments, in a fully autonomous navigation scenario, the physician may command the robot to automatically perform two consecutive maneuvers or perform all remaining maneuvers to reach the target.

[0108] In some embodiments, the system further includes a catheter stress detection algorithm that uses the position and shape of the fully tracked catheter at the anatomical location to estimate catheter stress inside the patient's lumen, represented using a force risk estimate.

[0109] For further discussion of the catheter stress detection algorithm, see below.

[0110] In some embodiments, the algorithm examines the shape of the catheter and issues warnings if the catheter is about to break or starts exerting too much force on the airway, etc. In some embodiments, these warnings can be used to monitor the robot's drive operation as well as to issue warnings in the handheld case for patient safety and system stability. In some embodiments, the algorithm can be based on pure geometric considerations as well as a dedicated Stress Neural Network (StressNN).

[0111] For further explanation of the Stress Neural Network (StressNN), see below.

[0112] In some embodiments, a force sensor may be incorporated inside the drive mechanism to predict the force applied by the device to the lumen (as a physician would do with a handheld catheter), but another option is to utilize device tracking information for the advance distance performed by the robot to estimate the stress of the device inside the lumen. In some embodiments, as described herein, the full tracked curve of the device is analyzed in its local state inside the anatomical structure to accurately predict the stress level of the device inside the lumen. In general, if the device follows a smooth path, it is most likely to be relaxed and not likely to damage the tissue. If the device begins to build a curved shape inside a fairly straight lumen and loops begin to form, the stress level of the device is considered high and the robotic drive mechanism is stopped. In some embodiments, in such cases, the device is pulled and released, or a warning is otherwise activated. In some embodiments, a potential advantage of combining the proposed stress detection mechanism with an external or internal force sensor is that it potentially provides better protection for the robotic driven catheter.

[0113] In some embodiments, optionally, the virtual lumen map used for navigation is actively deformed according to real-time deformation of the luminal structure to increase the accuracy of the system. In some embodiments, a potential advantage of deforming the virtual lumen map is that it potentially avoids displaying the device in its incorrect anatomical position, even potentially outside the lumen boundary, which may result in erroneous navigation decisions. In some embodiments, the deformation is tracked in real time by a deformation-aware tracking system, for example, based on a skeletal model of the luminal structure. In some embodiments, the skeletal model is deformed using an optimization method under certain shape constraints to find the true position of the device perfectly tracked within the deformed anatomical structure. In some embodiments, the deformation of the luminal structure is found in real time based on the perfectly tracked position of the device using a dedicated Deformation Neural Network (DeformNN) module based on many training samples.

[0114] For further explanation regarding the Deformation Neural Network (DeformNN) module, see below.

[0115] In some embodiments, the NavNN module is fed with the most accurate deformation-compensated localization image, whether generated by a dedicated deformation neural network based on a non-deformed localization image, or generated by a production deformation-aware tracking system in general, to determine the optimal driving action.

[0116] Exemplary generation of virtual / digital dynamically deformable luminal maps In some embodiments, as described above, the system utilizes a virtual / digital dynamically deformable luminal map to navigate the device inside the patient's body. In some embodiments, as a starting step in the generation of the virtual / digital dynamically deformable luminal map, the system is provided with, for example, a CT image (or an MRI image or an angiogram, etc.) of the patient in question, or, for example, an angiogram. In some embodiments, the system is configured to analyze the image and generate a virtual / digital 3D volumetric image of the patient. In some embodiments, the virtual / digital 3D volumetric image is the image used by the system to perform the navigation. In some embodiments, the digital 3D volumetric image is the image provided to a Navigation Neural Network (NavNN) module and / or a Deformation Neural Network (DeformNN) module and / or a Stress Neural Network (StressNN) module.

[0117] In some embodiments, during the procedure, the system is configured to correlate actual measured locations of the catheter inside the patient and incorporate those measured locations into a virtual / digital 3D volumetric image.

[0118] A typical Navigation Neural Network (NavNN) module In some embodiments, a Navigation Neural Network (NavNN) module is provided that "sees" the real-time system view (3D localization image) and determines optimal driving behavior based on this view. However, in some embodiments, instead of being displayed with a 2D projection, as a user would, the localization image encodes all relevant navigation information as raw 3D data. In some embodiments, the system is configured to overcome the inherent problems of displaying 2D or 3D images to a human user by allowing the NavNN module to analyze the relevant information as raw 3D data (human users cannot process 3D raw data) so that the user can analyze and decide which path to take. In some embodiments, this information does not suffer from 2D projection problems such as occlusion and depth misperception (as occurs to human users). In some embodiments, the NavNN processes the data in 3D based on trained weights to generate output driving behavior. For example, each NN includes "weights" such as convolution filter coefficients, thresholds, etc., and in some embodiments, these weights are found during the training process of the NN and are used for further predictions through the model. In some embodiments, these actions are then displayed to the user as driving recommendations (e.g., without limitation: (a) PUSH the shaft (catheter) forward / PULL back; (b) ROTATE the shaft (catheter) clockwise / counterclockwise; (c) DEFLECT joint #1 or deflect segment #1 up / down / right / left; (d) ROTATE joint #2 clockwise / counterclockwise; (e) DEFLECT joint #3 or deflect segment #3 up / down / right / left, etc.) or used automatically in an autonomous or semi-autonomous navigation system. In some embodiments, the NavNN is trained on data from a physical and realistic simulation module (see below) or on annotated recordings using supervised or unsupervised methods. For example, the physical simulation mimics a realistic endoluminal navigation procedure.For example, the simulation may show all 2D / 3D views available to the user during a navigational bronchoscopy, except that the tracked endoscope displayed is not real; instead, it is a physically simulated virtual endoscope placed inside the patient's CT scan (or MRI scan, or angiogram, etc.). In some embodiments, all interactions between the endoscope and the patient are physically simulated in software.

[0119] Referring now to FIG. 3a, a schematic diagram of an exemplary digital / virtual 3D volumetric image provided to the NavNN is shown, according to some embodiments of the present invention. In some embodiments, as previously described, the localized image provided to the NavNN is a digital / virtual 3D volumetric image of a particular resolution and scale derived, for example, from a pre-operative CT (or MRI scan, or angiogram, etc.) of the patient. In some embodiments, for example, the image may be a 100x100x100 multi-channel voxel image, where each voxel represents a region of interest such that the image spans a total spatial volume of 5x5x5cm. 3 The size is 0.5 mm to cover 33a , where 3 is a cube. In some embodiments, each of the channels in the localization image represents a different navigation function. In some embodiments, for example, the first channel represents the segmented lumen structure 302 (derived from the patient's pre-operative CT / MRI / angiogram / etc., as described above), the second channel represents the path 304 to the target, and the third channel represents the complete catheter curve 306 (inside the region of interest (ROI) localization image box, in this case only a single catheter is used) being tracked by a real-time tracking system, as shown in FIG. 3a . Optionally, a fourth channel is added with pre-operative raw (unsegmented) CT data (or MRI data, or angiogram data, etc.), not shown in FIG. 3a . In some embodiments, a potential advantage of providing raw unsegmented CT data (or MRI data, or angiogram data, etc.) is that it potentially allows the NavNN to base its navigation decisions not only on segmented airway structures, but also on non-segmented airways (which may be present in the CT scan and traversed by the catheter). In some embodiments, instead of using a binary segmented image, the first channel 302 representing the lumen structure may include a scalar image reflecting the likelihood that each voxel is inside the lumen, for example, as output by a lumen segmentation neural network or any other non-binary lumen segmentation algorithm. In some embodiments, in this case, the NavNN module is presented with richer information describing the complete lumen structure, including very small lumen tubes that would potentially have been dropped by applying a binary threshold to the segmentation. In some embodiments, the NavNN module can then base its navigation decisions not only on binary segmented airway structures, but also on "soft segmented" airways (those with small possibilities). In some embodiments, the second channel 304 optionally also includes a segmented or spherical target 308 at the end of the path to the target, or the target is included in a dedicated separate channel.In some embodiments, optionally, the first channel 302 represents a skeleton of the segmented luminal structure. The value of each skeleton voxel may be equal to the radius of the segmented luminal structure at that voxel.

[0120] Referring now to FIG. 3b, a schematic diagram of an exemplary digital / virtual 3D volumetric image including a camera sensor image provided to a NavNN is shown, according to some embodiments of the present invention. In some embodiments, a fifth channel 310 may be optionally added, including data from an image sensor placed at the tip of the catheter, for example as illustrated in FIG. 3b. In some embodiments, the image may be a 2D frame, for example at VGA resolution (640x480 pixels). In some embodiments, the frame is 2D, but the localization image is 3D, so it is important to specify how to render the 2D frame inside the 3D localization image at a sensible position that results in effective use of the camera image by the NN in its training and prediction, as described below. In some embodiments, the depth of each pixel (meaning its distance from the camera sensor) is typically unknown, so it may be located at any point along a ray that extends from the 3D camera position (known from the 3D tracking of the catheter) to a 3D direction determined by that pixel (according to its x,y position inside the camera sensor). In some embodiments, each 2D pixel is rendered using a backprojection starting from the 3D camera position and extending in the 3D direction of the pixel from the camera to the space in front of it until it collides with the boundary of the localized image, for example as illustrated in FIG. 3b. In some embodiments, if a depth channel is available for the camera image (e.g., by using a stereo camera, or by 3D reconstruction techniques, or by LiDAR, or by any other suitable method), the depth value is used to render each camera pixel at its exact 3D position in space, resulting in a 2D surface rendered in 3D, rather than backprojecting each pixel along a full ray. In some embodiments, a potential advantage of combining image sensor data inside the 3D localized image is that it can potentially improve the performance of the NavNN.In some embodiments, the NavNN module is configured to identify lumen passages in the image (relative to the 3D position of the catheter in space) and improve its output-driving behavior by using the identified lumens. Note that in some embodiments, the order of the channels is not important to the NavNN as long as it is consistent between training and prediction. In some embodiments, the localization image optionally includes additional channels with other navigation functions similar to the channels listed above or of other nature. In some embodiments, the results of a previously performed training process are optionally used to determine which data of the input channels to use based on its contribution to the success of the NavNN in predicting the output.

[0121] In some embodiments, to determine the navigation driving action, the digital / virtual 3D localization image is input into the NavNN module, which may consist of, for example, a 3D convolutional neural network (3D CNN). In some embodiments, the NavNN module processes the localization image in a "deep" multi-layer scheme, for example using multiple sigmoid activation functions in its output layer, until it outputs a probability for each possible driving action. In some embodiments, the high-level module then selects the driving action with the highest output probability as the choice for the next navigation driving action, as described above, and either mechanically executes the driving action using an automatic motor or displays the proposed driving action to the physician. In some embodiments, the high-level module may filter and / or improve and / or refine the output of the NavNN module. In some embodiments, for example, if the maximum output probability is not much better than the rest, the high-level module may randomly select between two comparable outputs to introduce some useful randomness (exploration) into the system. In some embodiments, a potential benefit of this randomness is that it can potentially help avoid local extrema points in the navigation system, where the system may oscillate back and forth around the same point in space. Alternatively, in some embodiments, the high-level module may enforce some hysteresis in the output probabilities to avoid rapid transitions between different driving behaviors, thereby smoothing out the driving process.

[0122] Referring now to Figs. 4a-e, schematic diagrams of an exemplary sequence of driving actions based on real-time localization images generated in real-time during a procedure and processed by a NavNN module are shown, according to some embodiments of the present invention. In some embodiments, the output driving actions are optionally performed by an automatic motor. For the description of Figs. 4a-e, it is assumed that the catheter is a passive "J" catheter and the driving system is a two-action system, namely ROLL and PUSH. Furthermore, the luminal structure is marked as 402, the path to the target is marked as 404, and the catheter is marked as 406. Fig. 4a shows that the catheter 406 is pointing left towards an airway 408 that does not lead to the target 410, indicated by the last sphere of the path. In some embodiments, the NavNN module processes the localization images to output the highest probability for a ROLL action. In some embodiments, the high-level module performs a motorized action to rotate the catheter, resulting in the catheter as illustrated in Fig. 4b. In some embodiments, presented with a localization image as shown in Figure 4b, the NavNN module then outputs its highest probability for a PUSH action, resulting in an image as shown in Figure 4c. In some embodiments, the NavNN module then outputs ROLL again, leading to an image as shown in Figure 4d, with the catheter pointing to the target. In some embodiments, all that remains is to push the catheter into the small left airway towards the target, as indicated by the PUSH output from the NavNN module, resulting in the final state as shown in Figure 4e, where the target is reached.

[0123] In some embodiments, the NavNN module generates real-time navigation instructions. In some embodiments, the 3D localization image is a multi-channel volumetric image including important navigation features (although, as mentioned above, it may be a 2D view). In some embodiments, some of the features may be considered static (e.g., the segmented luminal structure), while others may and / or may change rapidly during the procedure. For example, the fully tracked catheter position changes rapidly in real time, and the 3D localization image needs to be updated accordingly. Furthermore, in some embodiments, the segmented luminal structure may be considered static, but it is highly preferred to use a dynamic structure, e.g., one that approximates the true deformation state of the luminal structure (or at least the virtually calculated deformation state of the luminal structure) during the procedure. In some embodiments, the virtual real-time deformation is tracked or virtually calculated during the procedure (e.g., using a skeleton-based model or using a deformation neural network as described below and further described in International Patent Application No. PCT / IL2021 / 051475, the contents of which are incorporated herein by reference in their entirety). This modifies the luminal structure according to the tracked / virtually calculated deformations and updates the localized image accordingly to reflect the virtual real-time deformation state of the lumen.

[0124] In some embodiments, one or more techniques are used to generate real-time 3D volumetric images based on known structures. In some embodiments, the luminal structures and paths to the targets are static and may be generated once, while the fully tracked catheter is live and drawn on top of the static luminal map and paths using 3D line rasterization techniques, all ignoring deformations. In some embodiments, the luminal structures and paths to the targets may be dynamically modified to approximate real-time deformations of the luminal structures. In some embodiments, in this scenario, these features are updated in real-time, optionally requiring more extensive computational techniques. In some embodiments, a novel approach is to use a GPU to render 3D localization images in real-time. In some embodiments, in this setup, the 3D localization images are combined as a 3D render target, and each of the navigation structures is rendered by partitioning it into a set of volumetric pyramids. In some embodiments, in this novel proposed setup, instead of using planar triangles, 3D volumetric features such as luminal structures are volumetrically "tessellated" using 3D pyramid primitives. In some embodiments, an optimized GPU algorithm then processes the set of pyramids in a manner similar to processing standard 3D surface triangles, rasterizing them onto a 3D render target, essentially filling every voxel inside the pyramid until the entire 3D volumetric structure is rendered within the voxel. In some embodiments, modern GPU hardware does not support rendering pyramid primitives into a 3D render target as described above, but can be extended to do so using a dedicated GPU program. This is done, for example, by implementing an optimized GPU 3D rasterization algorithm using NVIDIA's Compute Unified Device Architecture (CUDA) or Open Computing Language (OpenCL).In some embodiments, alternatively, dedicated GPU hardware implemented in an ASIC or FPGA can be used to render 3D primitives. In some embodiments, rasterization of 3D volumetric primitives (pyramids) onto a 3D render target can be performed efficiently as rendering of 3D surface primitives (triangles) onto a 2D render target. This can be done, for example, using bucket rendering techniques in a parallel computing setting, as can be implemented in CUDA / OpenCL or ASIC / FPGA. In some embodiments, the developer can then use OpenGL extensions or DirectX to access the added functionality. For example, when using OpenGL, instead of creating a 2D framebuffer, the developer can generate and bind a 3D framebuffer for a GL_TEXTURE_3D render target, and instead of drawing primitives of type GL_TRIANGLES, the developer draws primitives of type GL_PYRAMIDS (a new GLenum type) consisting of four vertices per primitive. When using DirectX, developers can create and bind a 3D render target texture with the D3D11_BIND_RENDER_TARGET bind flag, and instead of drawing primitives of topology D3D_PRIMITIVE_TOPOLOGY_TRIANGLELIST, developers will draw primitives of topology D3D_PRIMITIVE_TOPOLOGY_PYRAMIDLIST with 4 vertices per primitive.

[0125] Referring now to FIG. 5, a schematic diagram of an exemplary volumetric tessellation of a catheter using 3D pyramid primitives is shown, according to some embodiments of the present invention. In some embodiments, using 3D pyramid tessellation to represent 3D structures (e.g., lumen structure, path to target, and fully tracked catheter - an exemplary catheter shown in FIG. 5) allows great flexibility to move and deform them in real time, thus reducing the complexity of generating a real-time 3D composite localization image for the deformation-aware NavNN module. In some embodiments, to update the localization image, it is only necessary to update the vertices that make up the navigation features, e.g., the catheter vertices are updated according to the fully tracked catheter position reported by the tracking system. The lumen structure and path to the target are potentially updated according to the real-time deformation tracking system by updating their vertices according to their association to the original lumen segmentation or skeleton.

[0126] It should be noted that the above-mentioned method can be considered as a general method for generating real-time 3D composite data using a dedicated GPU program or ASIC / FPGA to be processed by a 3D neural network for general use. For example, in some embodiments, the method can be used to render real-time composite volume images of a car traveling on a road for autonomous driving or for real-time prediction of potential car accidents. As another example, the method can be used to render a human hand and fingers, which may be tracked by multiple sensors, into a 3D volumetric image in real time. The 3D composite image can then be processed by the NN for real-time gesture recognition or any other suitable application.

[0127] In some embodiments, the NavNN module trains using several supervised and unsupervised methods. In some embodiments, in the supervised case, it utilizes a realistic navigation simulator module. In some embodiments, the module may model the catheter using finite element methods and may use position-based dynamics to simulate the physics of the catheter and handle collisions between the catheter and the luminal structure. In some embodiments, the luminal structure may be represented using its skeletal model or using its raw segmented volume as segmented from a CT scan (or MRI scan, or angiogram, etc.). In some embodiments, a distance transform may be applied to the segmented luminal volume, which may be processed to create a 3D gradient field of the luminal structure in 3D space, simplifying collision detection between the simulated catheter and the luminal structure. In some embodiments, the catheter tip and / or curves may be shown inside the luminal structure using a navigation view, for example, as performed in an actual navigation bronchoscopy procedure. In some embodiments, the operator may then navigate the simulated catheter toward an arbitrarily selected target using a keyboard, remote control, or any suitable method inside the luminal structure. In some embodiments, a record of the simulated navigation may then be collected. In some embodiments, at each timestamp, the simulated state of the complete catheter inside the luminal structure is fully known by the simulator. In some embodiments, the simulator module may generate the aforementioned localized images of the catheter inside the luminal structure along with the path to the target based on the known simulated state. In some embodiments, if a camera channel is to be included, the virtual camera images may be rendered using ray tracing techniques, which resemble real camera images for a particular camera specification (e.g., as performed in virtual bronchoscopy).In some embodiments, the camera image may be used as a 2D frame without depth information or may include a depth channel and may be calculated by the simulator. In some embodiments, the localized image may then be associated with the operator's driving command. Thus, in some embodiments, the operator's command is considered as a label for the NavNN module for each localized image generated in time. In some embodiments, the collected localized images, along with their supervised labels (operator's command), are then used in a supervised training process for the NavNN module. In some embodiments, as a result of the training process, the NavNN module tries to mimic the operator's command. In some embodiments, in the worst case scenario, the system only mimics the "average" operator's decision, and in the best case scenario, the system provides further generalization on top of the operator's command. In some embodiments, the simulator module may be given to multiple operators, each of whom may navigate to multiple different targets inside the simulated luminal structures of multiple patients. In some embodiments, during this process, a large amount of labeled samples are generated for training the NavNN nodes, which makes the training more robust and error-resistant.

[0128] In some embodiments, instead of using a simulator module, the labeled training samples may be collected from actual navigation procedures performed on real patients and / or mechanical simulated models, e.g., plastic or silicon models of luminal structures, and / or, e.g., preserved lungs (inflated in a vacuum chamber). In some embodiments, the navigation procedures may be "robotic" in the sense that an operator drives the system using a remote control and commands the drive mechanism to perform one of several possible drive actions (e.g., PUSH / PULL, ROLL, DEFLECT). In some embodiments, in the robotic case, the labeled training samples are collected by associating each localization image generated in real time with the operator's robotic command (e.g., PUSH / ROLL / DEFLECT). In some embodiments, a potential advantage of using data from an actual navigation procedure is that the catheter physics is realistic, whereas in the simulated case, the catheter physics is only an approximation of reality. In some embodiments, the labeled localization images may be collected from multiple procedures performed with different systems on many patients. In some embodiments, the data collection does not interfere with the normal procedure. The procedural software can be used to record the system status and the robot driving motions, because it runs in the background and may be executed offline in the post-processing stage of the procedure. In some embodiments, the procedural software may record only the system data and state (e.g., complete catheter position, selected target, deformation state of the luminal structure, camera video, and robot driving motion) over time. In some embodiments, the post-processing stage then generates corresponding localization images based on the recorded system state and labels them with the recorded robot driving motions with the same timestamp. In some embodiments, the labeled localization images can then be sent back to a dedicated server via a local network or the Internet, or can be collected manually by a field engineer.In some embodiments, the collected data is used to train from scratch or to improve the training of the NavNN module. In some embodiments, the NavNN module then mimics the driving behavior and navigation decisions of multiple physicians, potentially making the NavNN module as good as or better than the most skilled physicians.

[0129] In some embodiments, if the navigation procedure is fully manual or semi-manual (i.e., the catheter is hand-held and manually manipulated by the physician without the aid of a complete drive system), it may be more difficult to label the localization images based on the manual manipulation of the catheter. In some embodiments, in the case of manual manipulation, the manipulation of the catheter is not clearly a selection from a set of several drive actions as in the case of a robotic system, but rather is the result of the physician's hand, wrist, and arm manipulation. In some embodiments, in this case, a label can still be associated with each localization image by classifying each manual manipulation into a limited set of drive actions as described above. For example, the most proximal tracked sensor (the one closest to the catheter handle) of a fully tracked catheter may be used to classify the instantaneous handle manipulation, since it most efficiently reflects the action performed on the catheter handle (as the robot would have performed). As an example, if the physician pushes the catheter forward into the luminal structure, the most proximal tracked sensor is most likely pushed forward, and therefore the instantaneous manipulation is classified as a PUSH action. Conversely, the most distal catheter sensor (at the tip of the catheter) may not move at all, for example due to frictional forces. This illustrates why the proximal portion of the catheter is highly favorable in identifying the nature of manual handle manipulation. As another example, if the physician rotates the handle, the proximal sensor will most likely rotate with the catheter handle, and the momentary manipulation will be identified as a ROLL motion, while the distal sensor may again remain in place. In some embodiments, the catheter handle may be tracked using a dedicated sensor in the handle (e.g., a 6DOF tracked sensor, an IMU sensor (accelerometer, gyroscope, magnetometer, or any combination), or any other suitable sensor). In some embodiments, in the case of single or multi-joint deflectable catheters, the distal sensor may be used to detect catheter deflection and provide appropriate labeling for the NavNN module.In some embodiments, the deflection of the catheter tip is alternatively performed by pushing and pulling the steering wire inside the catheter handle, so a special sensor can be placed in the handle to track the state of the steering wire and detect the DEFLECT motion to the NavNN module. Thus, in some embodiments, it is possible to collect records of the manually manipulated catheter in the actual navigation procedure and label them in a post-processing stage by classifying each instantaneous manipulation of the catheter handle into a limited set of drive motions as required by the NavNN module training, for example using the most proximal tracked part of the catheter.

[0130] In some embodiments, instead of using the supervised training methods described above, a software simulator may be used for unsupervised learning using reinforcement learning. In some embodiments, in this case, the NavNN module has full control of a simulated catheter, and its goal is to drive the catheter to a randomly selected destination target in a random patient simulation. In some embodiments, the NavNN module is rewarded for each significant advance on the path towards the target and penalized for ineffective moves. In some embodiments, the training goal is to maximize the total reward of the NavNN module. In some embodiments, a potential advantage of such unsupervised learning is that the NavNN module can be trained in parallel across thousands of simulations of different patients and targets without the need for a human operator.

[0131] In some embodiments, the system is provided with dedicated commands that allow a level of randomness or "exploration" to the navigation. In some embodiments, a potential benefit of providing such apparent freedom to the system is that it potentially avoids the risk of getting caught in an extreme point of local probability, for example, the NavNN module endlessly outputting PUSH / PULL actions back and forth around the same anatomical point, which leads to a navigation "dead end" that the NavNN module cannot escape when using a stateless neural network (i.e., one with no "memory") such as a 3D CNN on a single localized image input. In some embodiments, this certain level of randomness (or "exploration") may be introduced into the navigation by, for example, a high-level behavioral module. In some embodiments, the high-level behavioral module may prefer a random driving action with a certain probability over the action output from the NavNN module. In some embodiments, the high-level behavioral module may also detect "loops" (situations where the NavNN module oscillates around an extreme point of local probability) and kick the NavNN module out of the loop by forcing a random exploration. For example, the high-level module may force the drive mechanism to perform a ROLL motion every 100 ms. In some embodiments, this motion is harmless to the navigation process and may allow the NavNN module to escape from a local extreme point if it gets stuck in one.

[0132] In some embodiments, the NavNN module utilizes previously recorded catheter states. In this case, the NavNN module is no longer completely "instantaneous". Instead, in some embodiments, the NavNN module-based output is based on history, not just on the current localization image input. Thus, in some embodiments, instead of training the NavNN module on a single randomly shuffled localization image, it trains on a time sequence of localization images. In some embodiments, the NavNN module is then input with the localization image as before, along with the output state of the previous prediction, and outputs the updated state for the next prediction. In some embodiments, the NavNN module is equipped with a memory that allows the NavNN module to "remember" that it has already attempted a particular operation and "see" that it was unsuccessful, thus avoiding loops by trying different techniques rather than repeatedly attempting the same operation. In some embodiments, in a more typical setting, the NavNN module is input with a short sequence of past localization images (e.g., including the 30 last frames) and their output actions along with the current output actions, and thus its output is based on history without the use of a dedicated state vector. In some embodiments, the NavNN module may be implemented using a 3D CNN over a short sequence of past localization images, or using a 3D Recurrent Neural Network (3D RNN) with a state vector, or by any other suitable method, with or without memory.

[0133] 6a-b, schematic diagrams of typical 3D localization images centered according to different objects are shown, according to some embodiments of the present invention. In some embodiments, the NavNN module is given an image (localization image) without knowing where the catheter is located inside this image or in which direction the catheter is pointing, so the NavNN module may then be forced to look for the catheter inside the image, which is a wasted effort, since the information about the complete position of the catheter is already known to higher level modules. In some embodiments, the task of the NavNN module is "relaxed" by providing it with an input image in which the tip of the catheter is centered 602, for example, as shown in FIG. 6a, and the x-axis of the image is aligned with the catheter tip direction. In some embodiments, the NavNN module then learns that the catheter is always centered in the image and pointing towards the x-axis, and can focus the rest of the navigation function to determine the optimal driving behavior. In some embodiments, the localization image may be centered 604 and oriented according to the closest point along the path to the target relative to the tip of the catheter, for example, as shown in FIG. 6b. In some embodiments, it may be oriented such that the x-axis of the image is aligned with the path direction to the target and the z-axis of the image may be aligned with the normal vector of the next branch point or with an interpolated normal vector between the previous branch point and the next branch point. In this scenario, the localization image maintains a fairly stable center and orientation along the path to the target despite manipulation of the tip of the catheter, because it is no longer constrained to the tip of the catheter, but is instead tied to the path to the target. In some embodiments, several other options for centering and orienting the localization image may be used, which may be a combination of the options described above. For example, the localization image may be centered on the tip of the catheter but oriented according to the path to the target, or vice versa.In some embodiments, the size of the localization image may be increased or decreased, and the resolution may be changed as well. In some embodiments, any such configuration may be used for training and prediction in the NavNN module, among others.

[0134] A typical DeformNN module 7a-b, which show schematic diagrams of exemplary undeformed and deformed localization images according to some embodiments of the present invention. In some embodiments, as previously described, accurate real-time localization images of the device are provided to the NavNN module to potentially generate better actuation motion. In some embodiments, the localization images include, in addition to the lumen map, a fully tracked catheter over the luminal structure in a separate channel. In some embodiments, to provide more information and increase the accuracy of DeformNN, the localization images include additional channels of additional tracked catheters. In some embodiments, deformation inputs are provided to the NavNN module to increase the accuracy of the NavNN module and / or to allow the catheter to be placed in its correct location inside the anatomical structure. The deformation inputs include information regarding real-time based information on the actual organ deformation, which is translated into deformations in the luminal structure as shown in the localization images. In some embodiments, this is achieved by placing a skeletal model of the luminal structure, which is used to find the organ deformation based on the fully tracked catheter using optimization methods. This is also further described in International Patent Application No. PCT / IL2021 / 051475. The contents of this document are incorporated herein by reference in their entirety. In some embodiments, an undeformed image may be constructed using an undeformed luminal structure (one without any deformation compensation), for example as shown in FIG. 7a. In FIG. 7a, the catheter may appear to cross 702 the lumen boundary. In some embodiments, the downside of feeding the NavNN module with an undeformed localization image is that it potentially degrades the performance of the NavNN module since it does not provide an accurate image of the catheter inside the lumen. In some embodiments, the deformation tracking algorithm provides either an adjustment of the catheter's position relative to the luminal structure, or vice versa, so that the catheter appears inside an acceptable tube, as it would in reality. In some embodiments, in a skeleton-based deformation tracking algorithm, the luminal structure is modeled as a skeleton with branches of specific radii and connecting branch points.In some embodiments, the skeleton is deformed according to a particular deformation model in order to return the catheter to the interior of the lumen under the imposed organ shape constraints.

[0135] In some embodiments, we propose a new method for finding luminal deformation based on an AI statistical approach. In some embodiments, instead of explicitly modeling the luminal structure with a skeletal model and finding the deformation based on an optimization method, we follow an AI approach where the deformation is implicitly solved using a neural network. In some embodiments, similar to the NavNN module, the DeformNN module inputs localized images that can be of the same size and / or centered and / or oriented as described above. However, in some embodiments, the DeformNN module does not necessarily input the path to the target as one of its input channels, since this information is more relevant for navigating to the target and less relevant for finding the luminal deformation. In some embodiments, in addition, the NavNN module preferably inputs a deformed localized image (with deformation compensation), while the input to the DeformNN module is a non-deformed localized image, as shown for example in FIG. 7a. In some embodiments, the localized image input to the DeformNN module can further include a camera channel, as shown for example in FIG. 3b. In some embodiments, the DeformNN module utilizes the camera channel to determine the most likely deformation of the luminal structure. For example, when the luminal structure is deformed, as shown, for example, in FIG. 7a, the camera image may teach, for example, the correct catheter position inside the anatomical structure to localize the tip of the catheter relative to the visual bifurcation. In some embodiments, the DeformNN module may learn to use these features to better find the correct anatomical location of the catheter within the deformed luminal structure. In some embodiments, the DeformNN module is responsible for taking the undeformed localization image (luminal structure and catheter position) and converting it into an accurate deformed localization image of the same size, as shown, for example, in FIG. 7b. In some embodiments, this can be achieved using, for example, a 3DU-Net neural network architecture.In some embodiments, the output deformed localization image can then be rigged with additional channels (path to the target with applied deformations) and input to the NavNN module to generate more reliable drive motion and accurately guide the catheter towards the target. In some embodiments, the output of the DeformNN module may be used for display purposes to correct the 2D / 3D system view to reflect the lumen deformations, as described further below and shown, for example, in the flow chart of FIG.

[0136] Referring now to FIG. 8, a flow chart of an exemplary method for displaying a correct 2D / 3D system view to reflect lumen deformation is shown, according to some embodiments of the present invention. In some embodiments, the system generates an undeformed localization image 802. The term "undeformed localization image" refers to a localization image where the image has not been modified and / or compensated for potential and / or estimated and / or calculated deformations (either due to catheter movement, patient movement, etc.). In some embodiments, the system then generates a deformed localization image using a DeformNN module 804. The term "deformed localization image" refers to a localization image where the image has been modified and / or compensated for potential and / or estimated and / or calculated deformations (either due to catheter movement, patient movement, etc.). In some embodiments, the system view is updated with the newly generated deformed localization image (806). In some embodiments, the newly generated deformed localization image is then fed into a NavNN module (808). In some embodiments, the NavNN module then provides the necessary driving operations to be performed by the system (810).

[0137] In some embodiments, instead of outputting a deformed version of the undeformed localization image input, the DeformNN module may simply output one or more probabilities that indicate that the input catheter is in its correct position within the lumen, as in the case of the input localization image. In this scenario, a high level optimization (e.g., based on a skeletal model) is used to search for lumen deformations, as in the case of deformation tracking algorithms based on a skeletal approach. In some embodiments, when outputting a single probability, instead of optimizing based on energy minimization of a more standard energy function (e.g., one that encodes branch point angle constraints, etc.), the optimization is performed to maximize the output probability of the DeformNN module. The deformation state is searched for such that it has the greatest probability of being the correct one output by the DeformNN module. In some embodiments, DeformNN then serves as a metric to evaluate the proposed deformations, but the deformations themselves are performed externally in an optimization algorithm using any suitable deformation model.

[0138] In some embodiments, the DeformNN module may be designed and trained to output the position of the catheter in a fully deformed localization image, as described above. In some embodiments, the DeformNN module takes an input catheter position on the undeformed luminal structure and renders it on its output inside the undeformed luminal structure (where it would have been rendered if the luminal structure had not been deformed). In some embodiments, the DeformNN module outputs a single channel that renders the modified catheter position. This differs from that shown in FIG. 7a-b, where the DeformNN module modifies the luminal structure from an undeformed state to a deformed state, but the catheter is left intact. In this case, the high level module may find the catheter in the output image and match between the input catheter at its original position and the output catheter at its deformed position inside the anatomical structure output by the DeformNN module. In some embodiments, matching the catheter may be achieved by finding the tip of the catheter in both 3D images and climbing along the length of the catheter, or by any other suitable method. In some embodiments, the positions of the catheter before and after deformation may each be represented using the following curve functions:

number

[0139] In some embodiments, a set of deformation differences can then be calculated using: Δγ=γ0-γ1

[0140] In some embodiments, since the DeformNN module finds the catheter position inside the anatomical structure, it can be safely assumed that γ1(σ) is inside the luminal structure. In some embodiments, each 3D position γ1(σ) along the undeformed luminal structure can then be updated to its deformed position γ0(σ) using any suitable skeletal model for display or other computational algorithms. In some embodiments, the deformation of the luminal structure is indirectly revealed by matching the catheter positions before and after deformation (output by the DeformNN module).

[0141] In some embodiments, alternatively, in a more direct approach, the DeformNN module may be designed and trained to output a deformed luminal structure based on the catheter position, leaving the catheter intact. In this case, the output image is a deformed version of the input luminal structure, which can be used for display or other computational algorithms. For example, the output luminal structure can be matched to the input luminal structure using 3D image registration techniques or by using the respective skeletons of the input and output structures. In some embodiments, by matching between the input and output structures, a deformation vector can be calculated for each shared point inside the input and output structures. In some embodiments, the deformation vector can then be applied to a skeleton model of the luminal structure to track its deformation state, which is solved in real time by the DeformNN module.

[0142] 9a-d, schematic diagrams of typical operations performed by the DeformNN module are shown, according to some embodiments of the present invention. In some embodiments, when it is difficult to determine whether the catheter is inside one lumen 902 or the other lumen 902 due to high symmetry or significant pre-registration system errors, as shown in FIG. 9a, for example, the DeformNN module may choose to output two possible virtual catheters 904, 906 with similar or different intensities (probabilities), as shown in FIG. 9b, for example, if the DeformNN module is designed to render the catheter in its corrected position inside the anatomy. In some embodiments, this indicates that the DeformNN module is not sure of the correct deformation, and the intensity of each output catheter reflects the AI's confidence in the particular position. In this case, the high-level module may select one of the output catheter curves based on the output intensities or other high-level considerations. For example, the high-level module may choose to display the catheter that is closer to the catheter already presented by the system, which prevents "jumps" between different catheter hypotheses (especially if the output intensities are similar). In some embodiments, a segmented catheter may alternatively be displayed to the user to reflect that the system is not sure of the actual catheter location inside the anatomy. In this view, the operator is presented with two or more virtual catheters inside the luminal structure, each displayed with a different intensity or opacity corresponding to its output intensity by the AI. In some embodiments, the user may have this information for "informational" purposes only. In some embodiments, the user may use this information to tell the system which direction to take.In some embodiments, once the ambiguity is resolved, e.g., as shown in FIG. 9c, by advancing the catheter 908 further towards the target and after the curve has a more defined shape, the DeformNN module is taught about the actual catheter location inside the anatomy, and the output intensity of the segmented catheter naturally decreases 910, e.g., as shown in FIG. 9d, and the DeformNN module outputs a single strong catheter intensity 912 at its output. Thus, in some embodiments, the opacity of all other virtual catheters decreases as the ambiguity is resolved, so that the system view eventually shows a single strong catheter in a resolved location inside the anatomy. In some embodiments, in an alternative view, if the DeformNN module outputs multiple catheter hypotheses, the system may choose to show the catheter only up to the point where it begins to segment (as output by the DeformNN module). In some embodiments, the remainder of the catheter (i.e., the left and right segment) may then be rendered in “red” or transparent to indicate to the user that the system is unsure about the location of this portion of the catheter. In some embodiments, in another alternative view, catheter ambiguity may split the screen into, for example, left and right screens, each showing a different virtual location of the catheter inside the anatomy. In some embodiments, once the ambiguity is resolved, the "winning" half grows into the full screen view, pushing the other half out of view. In some embodiments, the NavNN module may be presented with a localization image that includes multiple catheter hypotheses (possibly with different strengths) and trained to still continue navigation even under these ambiguous conditions. For example, if the NavNN module is using memory, it may attempt certain driving maneuvers that lead to a final catheter position. In some embodiments, the NavNN module may then "check" whether the final catheter position is progressing towards the target.If not, it may choose to pull back the catheter and attempt a different drive motion (since it has already attempted the first drive motion, as encoded in its memory or state vector) so that the final final catheter position advances toward the target.

[0143] In some embodiments, training of the DeformNN module is performed by presenting pairs of undeformed input localization images and deformed output localization images. In some embodiments, these images can be collected by using a realistic simulator module, as described above for the training process of the NavNN module. In this scenario, the exact simulated position of the catheter is known to the simulation. In some embodiments, the true position of the catheter in the simulation inside the luminal structure is used to generate an output localization image for the DeformNN module. In this image, the catheter is placed exactly in its true position inside the anatomical structure, which would be output by the DeformNN module. In some embodiments, some deformation model is applied to the luminal structure to create the input image. For example, the structure can be randomly deformed based on standard polynomial or spline techniques, or using more complex techniques that mimic the anatomical deformation of real organs (e.g., using finite element methods and / or finite volume physical simulations, which may be based on physical measurements of various tissues and structures). In some embodiments, deformations are applied only to the luminal structure and not to the catheter, resulting in an "undeformed" localization image (one with no deformation compensation) where the catheter may appear to intersect with the luminal boundary. In some embodiments, this creates an image pair that can be used to train the DeformNN module. In some embodiments, similar to the NavNN module, collecting data from simulations can form a large set of training samples for many patients, targets inside luminal structures, and different catheter poses, which is critical for successful training of the AI ​​model.

[0144] In some embodiments, recording of actual procedures may be used to collect accurate deformation data of a living organ, e.g., a lung. In some embodiments, a catheter may be introduced into a specific known airway inside the lung, and the complete position of the catheter may be recorded under a specific forced or natural deformation, thereby teaching the deformation of that airway. In some embodiments, multiple catheters may also be introduced into multiple known airways, and their complete positions may be recorded to teach the deformation of multiple airways in parallel under a specific applied force. In some embodiments, training samples may also be collected from a mechanical simulated model, as in the NavNN module. In some embodiments, multiple tracked sensors may be placed inside the organ (e.g., on the pleura of the lung) to record real-time data of the deformation. In some embodiments, multiple CBCT (Cone Beam CT) scans may be performed while deforming the organ, and the different scans may be aligned using deformable registration to reveal the deformation vector between the scans under a specific applied force. In some embodiments, the deformation may be learned and measured by other means as well, e.g., by using contrast agents, markers, external sensors, using ultrasound probes, fluoroscopic imaging, among other suitable means.

[0145] In some embodiments, the DeformNN module can be further trained in a pre-procedure phase for a particular patient using deformation augmentation methods as described above to further adapt the model to the luminal structure of the particular patient, thus improving the performance of the AI ​​model during the procedure. For example, the luminal structure of the patient can be loaded into an offline simulator module prior to the procedure. In some embodiments, a simulated catheter can then be placed at different random locations inside the simulated luminal structure. In some embodiments, deformations of the luminal structure can be simulated by the simulator module to form pairs of undeformed and deformed localized images. In some embodiments, the trained DeformNN module can be presented with newly created image pairs and further trained with a small learning rate based on these pairs so as to still retain its weights from the original training, but these weights are then fine-tuned towards fitting the deformations of the current patient. In some embodiments, these operations potentially fine-tune and bias the deformation model for the current patient, slightly losing its generality in favor of performance for solving the deformations on the anatomical structure of the current patient.

[0146] A typical Stress Neural Network (StressNN) module In some embodiments, the system includes a catheter stress detection algorithm that utilizes the tracked catheter position and its shape at its anatomical location to estimate catheter stress inside the patient's airway. In some embodiments, the algorithm examines the catheter's shape and issues warnings if the catheter is about to break or starts applying too much force to the airway, etc. In some embodiments, these warnings can be used, for example, to monitor the robot's drive operation, as well as to issue warnings in the handheld case to ensure patient safety and system stability. In some embodiments, the algorithm is based on pure geometric considerations as well as a dedicated Stress Neural Network (StressNN) module that analyzes the catheter's shape.

[0147] In some embodiments, a force sensor may be integrated inside the drive mechanism to predict the force applied by the catheter to the airway (performed by the physician using a handheld catheter), but another option is to utilize catheter tracking information for the distance the robotic catheter advances to estimate the stress of the catheter inside the airway. In some embodiments, as described elsewhere herein, the fully tracked curve of the catheter is analyzed in its localized state inside the anatomical structure to accurately predict the level of stress of the catheter inside the airway. In general, if the catheter follows a smooth path, it is most likely to be relaxed and not damage the tissue. If the catheter starts to build a curved shape inside a straight airway and a loop of the catheter shape starts to form, the stress level of the catheter is considered high and when using a robotic drive mechanism, the robotic drive mechanism is stopped. In some embodiments, the catheter is then pulled and released. In some embodiments, a potential advantage of combining the proposed stress detection mechanism with an external or internal force sensor is that it potentially provides better protection for the robotic drive catheter.

[0148] In some embodiments, for example, when the catheter is driven forward a known distance, the tip of the catheter is expected to advance accordingly. In the extreme case where the tip of the catheter does not move, it is concluded that tension was created along the length of the catheter and did not translate into forward tip motion. In some embodiments, the instantaneous shape of the catheter can be further analyzed and stress levels in the length of the catheter can be inferred based on its shape inside the luminal structure.

[0149] In some embodiments, for example, physical finite element simulations that realistically simulate the physical properties of the catheter and luminal structures can be used to estimate the forces exerted by the catheter on the luminal structures for a given shape and location inside the anatomical structure. In this case, the catheter is placed inside the simulated luminal structures exactly as it would be placed inside the real structures tracked in the procedure. In some embodiments, these are performed in real time during the intervention. In some embodiments, these are only performed in the simulation, meaning not during the procedure, for example to teach the NN and / or other software. In some embodiments, a simulation can then be performed to calculate physical simulated forces based on the simulated structure of the catheter and the simulated behavior of the lumen. In some embodiments, once the contact forces, as well as the internal catheter forces, have been calculated, a binary or smooth threshold may be used to calculate a risk estimate of the forces (e.g., a scalar between 0 and 1).

[0150] In some embodiments, the 3D localization image as described above can be used to visualize the shape of the catheter inside the luminal structure in 3D. In some embodiments, the localization image can be input into a dedicated Stress Neural Network (StressNN) module, which outputs a force risk estimate based on the shape of the catheter inside the luminal structure as visualized by the localization image. For example, the StressNN module may output a value close to 0 if the catheter is released inside the luminal structure, and a value close to 1 if the shape of the catheter inside the luminal structure indicates risk (e.g., if the shape of the catheter is very curved or loops are starting to form). In this case, the high-level module may pull the catheter back until the StressNN module outputs a value closer to 0 and the catheter is released. In some embodiments, a localization image of greater support (e.g., one in which a traceable length of the complete catheter is visible) is provided to provide a higher level of confidence to the StressNN module. In some embodiments, this allows the StressNN module to also consider the proximal portion of the catheter where curves and loops may be built up during the procedure.

[0151] In some embodiments, the StressNN module is trained using a simulator module, where a simulated catheter is introduced into a luminal structure and moved to random positions inside an organ. In some embodiments, the labeled samples are collected by calculating contact forces and internal catheter forces by physical simulation, and pairing localization images with their corresponding force risk estimates based on the calculated forces. In some embodiments, training of the StressNN module is performed by providing records of previous medical procedures (e.g., by using sensors in the catheter and recording forces during the procedure), e.g., by analyzing the records and deriving actions performed by the user along with the catheter status at that moment.

[0152] In some embodiments, the NavNN is also used to detect catheter stress inside the luminal structure. In some embodiments, the NavNN is trained to pull back the catheter whenever the operator (or simulator) detects a high level of catheter stress. In some embodiments, this teaches the NavNN to perform catheter stress detection as shown in the 3D localization image and to pull back the catheter if stress is building up. In some embodiments, the final catheter stress detection is performed by a physical simulation module, a dedicated StressNN module, a NavNN module, or any combination of the above.

[0153] Overview of a typical endoluminal device with tracking and navigation system Referring now to FIG. 10, a schematic diagram of an exemplary endoluminal device with a tracking and navigation system is shown, according to some embodiments of the present invention. The endoluminal device shown in FIG. 10 is a modified version of the endoluminal device shown in FIG. 1, with the addition of components responsible for providing input regarding navigation, deformation, and stress. In some embodiments, the endoluminal system 1000 comprises an endoluminal device 1002 configured for endoluminal intervention, such as an endoscope or bronchoscope or vascular catheter, or a vascular guidewire. In some embodiments, the endoluminal device 1002 includes one or more cameras and / or one or more sensors 1014 at a distal end of the endoluminal device 1002. In some embodiments, the endoluminal device 1002 is connected to a computer 1004 configured to monitor and control operations performed by the endoluminal device 1002 (including, in some embodiments, self-steering operations of the endoluminal device 1002, as further described below). In some embodiments, the endoluminal system 1000 further includes a transmitter 1006 configured to generate an electromagnetic field used by the endoluminal system 1000 to monitor the position of the endoluminal device 1002 inside the patient 1008. In some embodiments, the endoluminal system 1000 further includes a display unit 1010 configured to show a dedicated image to the operator, thereby potentially assisting the operator in navigation of the endoluminal device 1002 during the endoluminal intervention. In some embodiments, the endoluminal system 1000 optionally further includes one or more sensors 1012 configured to monitor the movement of the patient 1008 during the endoluminal intervention. In some embodiments, the patient movement is used to assist in navigation of the endoluminal device 1002 inside the patient 1008. In some embodiments, the computer 1004 includes a NavNN module 1016 configured to receive accurate real-time localization images from, for example, one or more cameras and / or one or more sensors 1014, as described above. In some embodiments, as previously described, the NavNN module 1016 then generates driving directions for the intraluminal device 1002 inside the patient 1008 to reach a desired location therein.In some embodiments, the computer 1004 includes a DeformNN module 1018 configured to calculate deformation information and provide it to the system 2D / 3D view to generate a more accurate image of the catheter location inside the anatomical structure, as well as to the NavNN module. The NavNN module then utilizes the deformation information to potentially increase the accuracy of navigation and driving direction. In some embodiments, the computer 1004 includes a StressNN module 1020 configured to calculate and / or estimate the stress exerted by the catheter on the tissue in which the endoluminal device 1002 is being manipulated. In some embodiments, the StressNN module 1020 performs the calculations / estimations, optionally in real-time, based on the position and location of the catheter inside the patient's body 1008. In some embodiments, the computer 1004 includes a high-level module 1022. The high level module 1022 receives all information from the localization system (transmitters and sensors), the NavNN module, the DeformNN module, and the StressNN module and uses this information to drive one or more mechanisms within the endoluminal system 1000, such as a robotic mechanism that drives the distal end of the endoluminal device 1002 (steering - see below), and a robotic mechanism that drives the advancement and / or withdrawal of the endoluminal device 1002 into and out of the patient.

[0154] Exemplary self-steering endoluminal device and expanded view In some embodiments, the intraluminal device 1002 includes a mechanical working distal tip configured to be driven either manually or automatically to direct and propel the intraluminal device 1002 towards a desired location inside the patient's body 1008. In some embodiments, the instrument (the intraluminal device 1002) is configured to be capable of autonomously orienting its working tip towards a particular target, having suitable spatial recognition algorithms (e.g., based on information received from the NavNN and / or DeformNN modules) and sensing capabilities. For example, in the context of intraluminal device navigation, the system allows for a self-steering device, where the tip of the device self-steering depending on its position relative to the target while the operator moves the device distally or proximally. In some embodiments, such a target may be, for example, a point on a path to which the tip of the device is configured to be directed. In this example, to follow the path to the target, the operator only needs to carefully push the device distally while the tip self-steering through the branching points of the luminal tree, until the device reaches its target. Further to this example, in some embodiments, a pre-operative plan is created on an external computing device (e.g., a laptop or tablet or any other suitable device), the luminal structure is segmented, and targets and pathways are identified. In some embodiments, the plan may then be transferred to the device via a physical connection, wireless, WiFi, Bluetooth, NFC (near field communication), or other transfer methods and protocols.

[0155] In some embodiments, the point in space of the self-steering tip may be a target within the moving volume, e.g., the breathing lungs, or a target, e.g., in the liver, or a target, e.g., in a soft blood vessel, or a target, e.g., in the digestive system, while the tip of the catheter is configured to orient toward this target without operator intervention.

[0156] In some embodiments, the intraluminal device 1002 may include a handle containing the necessary electronic processor and control components, such as the necessary algorithms, power source, and the necessary electromechanical drive components, In some embodiments, the intraluminal device 1002 may be a disposable device or a non-disposable device.

[0157] In some embodiments, the endoluminal device 1002 may be connected to an external screen on which a representation of the luminal structure is displayed along with an updated display of the instrument's position inside the lumen. In some embodiments, in addition to or instead of a display of the instrument's position, other feedback means are provided to inform the operator of the status of the system. In some embodiments, such an indication may be, for example, a flashing green light as long as the instrument is on track to reach the target (e.g., following the path); or a steady green light display once the target is reached. In some embodiments, it may be a steady red light display or vibration feedback using a vibration motor in the handle if the target may not be reached at the current location and the catheter needs to be pulled back (e.g., the tip is past the target or the tip is in the wrong branch). In some embodiments, in addition to the displays mentioned above, audio displays may be played by a small speaker inside the handle of the catheter to guide the operator through the procedure. In some embodiments, further instruction and warning methods are not mentioned here but are within the scope of the present invention.

[0158] In some embodiments, the electromechanical drive component can consist of a small motor inside the handle of the catheter. In some embodiments, there can be a single small motor that controls the roll angle of the passive "J" catheter. In some embodiments, the NavNN module 1016 may output two drive actions: PUSH / PULL, ROLL. In some embodiments, if a ROLL action is required, the high level module 1022 automatically activates a roll motor inside the catheter to perform the catheter rotation, so that the catheter is always automatically aligned with the next branch point to the target. In some embodiments, if a PUSH action is required, a green LED on the handle of the catheter may flash to indicate to the operator that the catheter is on track to the target and needs to be manually pushed. In some embodiments, if a PULL action is required, vibration feedback may be activated in the handle, for example using a vibration motor inside the handle, or a red LED may be turned on or flashed to indicate to the operator that the catheter is off track and needs to be retracted. In some embodiments, when a PUSH or PULL motion is required, the high level module 1022 activates an internal forward / reverse motor in the catheter to perform a limited forward or reverse movement of the catheter, so that the catheter automatically advances toward the target (or is pulled back when the catheter enters the wrong lumen). In some embodiments, the dimension (size or length) of the movement (either forward or backward) performed by the catheter is limited by the mechanical properties of the motor (optionally located within the handle of the catheter). In some embodiments, the dimension (size or length) is fixed and known. In some embodiments, the dimension (size or length) is actively adjustable and known, for example, by either replacing the motor or adjusting the force provided by the motor. In some embodiments, the system is configured to use the "known dimension of movement" to provide fine adjustments to the navigation toward the target.In some embodiments, instead or in addition, the system is configured to use "known dimensions" to maintain internal stability of the anatomy, for example, by actuating the device (forward actuation, backward actuation, and de-actuation) when reaching a moving target. The system can maintain a specific position even as the target moves, thereby maintaining internal stability of the anatomy relative to the target.

[0159] To understand the complexity of the task, the following example is given: Breathing causes natural deformation of tissue. For example, in the lung, the lower lobe of the lung can move / deform about 2cm to about 3cm during breathing. To provide the most accurate "picture" for navigation, the system uses the DeformNN module to update the lumen map in real time according to the detected movement of the patient (e.g., caused by breathing). In some embodiments, the movement is monitored, for example, using one or more sensors placed on the patient and / or on the bed and / or on the operating table. In some embodiments, once the movement is accurately incorporated into the dynamic lumen map, other deformations (e.g., deformations caused by actually driving the device to the location) are also taken into account and actively incorporated into the dynamic lumen map. Thus, at this point, there is a 3D lumen map that is constantly being updated for deformations caused by organic or induced movements. Once this is achieved, the user can command the system to maintain the selected position. For example, to maintain a given distance to the target (e.g., stay 15mm from the target) or to maintain a position on a particular point on the lumen map selected by the user so that the device is kept on the selected point. In some embodiments, the system drives the propulsion device to achieve fine adjustment of the navigation and positioning of the device using the system's "knowledge" of the "known dimensions of movement" caused by the drive. In some embodiments, the potential advantage of fixing the device to an anatomical location inside the luminal structure relative to a moving target is that it is superior to the alternative of stabilizing the device in free space or stabilizing the device relative to the luminal structure, which does not take into account the actual target motion, which may have other motion characteristics from the lumen. 3D tracking systems typically track the device in tracking coordinates relative to a transmitter (e.g., in the EM), which is typically fixed to the bed. Thus, the device is tracked in "free 3D space", i.e., for example, in bed coordinates.Thus, the device location may vibrate significantly (e.g., 2-3 cm) in its tracked x, y, z location due to the patient's breathing or other organic or non-organic deformations, but the anatomical location of the device inside the body does not actually change (e.g., the device is in the same location inside the lumen). The target does move, however, due to the deformations caused by breathing. Known techniques typically fix the robotic catheter in free space. This is done by fixing the robotic device to the same x, y, z location "in free space" relative to the tracking source by applying some control mechanism to the catheter location. In some cases, this method has a major drawback, since the fixed x, y, z location relative to the tracking source does not reflect a fixed location relative to the anatomy.

[0160] In some embodiments, the disposable catheter is completely wireless and includes a power source such as a battery, a microprocessor, a dedicated ASIC / FPGA, NFC communication support, red / green indicator LEDs, a vibration motor, and a small rotation and / or forward motor for the catheter. A typical system flow chart is shown in FIG. 11. In some embodiments, the pre-operative plan is executed on a tablet device for a particular patient and communicated to the wireless catheter using NFC by attaching the catheter to the tablet in a catheter-patient pairing step 1102. In some embodiments, optionally, upon pairing, i.e., successful transmission of the patient's plan onto the catheter, an audio indication may be played or an LED may be turned on (1104). In some embodiments, optionally, the plan may consist of a segmented lumen structure, a path plan to the target, and target markings. In some embodiments, the segmented lumen structure is optionally of sparse nature and therefore can be compressed to fit the memory limitations of most microprocessors (e.g., only a few kilobytes) using, for example, Huffman coding or other suitable methods. In some embodiments, the electromagnetic calibration may also be optionally transferred to the wireless catheter upon pairing. In some embodiments, the electromagnetic transmitter identifier or the complete configuration and calibration may be optionally transferred to the wireless catheter to enable the catheter to perform fully calibrated electromagnetic tracking during the procedure. In some embodiments, a sampling of the camera sensor is performed (1108). If the catheter consists of digital electromagnetic sensors, no external amplifiers and DSP are required to obtain full 6DOF tracking, only a software algorithm 1110 that can be implemented on most microprocessors (depending on the electromagnetic configuration and calibration transferred). In some embodiments, during the procedure, the catheter can then use the 6DOF tracking algorithm 1110 to resolve the complete catheter position by processing the measured magnetic fields from its multiple sensors as described above.In some embodiments, the catheter position is then adapted to the luminal structure in one or more registration processes. In some embodiments, a multi-channel 3D localization image may be rendered (1114) using the methods described above, optionally in real time, for example, using a special GPU block in a dedicated ASIC / FPGA chip. In some embodiments, the localization image may include a dedicated camera channel by rendering a 2D camera frame onto the 3D localization image using the methods described above, optionally accelerated by a dedicated GPU. In some embodiments, the 2D camera frame may be captured from a camera sensor at the tip of the catheter. In some embodiments, the raw camera image may be processed by an image signal processor (ISP) block 1112 in the ASIC / FPGA. In some embodiments, a DeformNN module may be used to track the organ distortion in real time using the rendered localization image (1116). In some embodiments, the deformed localization image is used to update the system view (1118). In some embodiments, the DeformNN module process is computed on a dedicated ASIC / FPGA chip, and the DeformNN data is sent to the NavNN module 1016 for further use (1120). In some embodiments, following the DeformNN operation, NavNN can be executed to compute the optimal drive motion towards the target (1124) or to stabilize the catheter on a moving target, also hardware accelerated by a dedicated ASIC / FPGA. In some embodiments, the output from the NavNN module is used to provide feedback 1122 to the operator, as previously described. In some embodiments, once the target is reached (as achieved by the high level module 1022), feedback can optionally be provided to the operator, and biopsy and treatment tools can be inserted through a special working channel in the catheter.In some embodiments, the StressNN module may also be used to estimate a risk estimate of force of the catheter inside the luminal structure before or after passing the localization image to the NavNN module. In some embodiments, if the force risk estimate is close to 1, indicating that the catheter is exerting too much force inside the luminal structure, then the system may be stopped or the catheter may be automatically pulled back until it is released (indicated by the force risk estimate being close to 0 again). In some embodiments, the system flow is regulated by a microprocessor, which may be a dedicated chip or may be incorporated as a block in a dedicated ASIC / FPGA chip.

[0161] In some embodiments, the wireless self-steering catheter can also be equipped with WiFi equipment to transmit compressed (e.g., using H.265) or uncompressed 2D / 3D system views to an external monitor. In this case, the views may be generated in real time using the catheter's dedicated GPU and can optionally be encoded using, for example, a hardware-accelerated H.265 encoder inside the ASIC / FPGA. In some embodiments, the system views can be displayed by any WiFi-enabled device through web services, RTSP protocol web browsers, or any other video streaming software. In some embodiments, the views are displayed on an external monitor, providing important 2D / 3D navigation information to the operator's physician or on a tablet or smartphone. In some embodiments, the endoscopic video can be displayed on a small portable display screen attached to the handle of the catheter, similar to a periscope or magnified view. In some embodiments, the operator can then "look into the patient" through the small display as if the catheter were a periscope. In some embodiments, the displayed endoscopic video may be augmented with additional 3D navigation data, such as a route to the target, targets or other navigation instructions (e.g., physician orders), additional anatomical features from a CT (or MRI scan, or angiogram, etc.) In some embodiments, in addition or instead of displaying an augmented endoscopic video view, a purely virtual 3D view of the fully tracked catheter in its anatomical location inside its luminal structure, for example, as would be displayed on an external monitor during a typical navigation procedure, may be displayed.

[0162] In some embodiments, the intraluminal device includes one or more steering mechanisms configured to orient and steer the intraluminal device in one or more directions. In some embodiments, the one or more steering mechanisms include one or more of the following:

[0163] 1. One or More Pull Wires: In some embodiments, one or more wires are connected to one or more joints or points along the shaft.

[0164] 2. One or more pre-curved shafts. In some embodiments, one or more pre-curves are placed one inside the other, and rotating one of the pre-curved shafts relative to the other results in deflection of the shaft, e.g., when both curves of the shaft are aligned, maximum deflection is achieved, but when the curves of the shaft are opposite each other, minimum deflection is achieved.

[0165] 3. One or more shafts with different mechanical properties, one inside the other. In some embodiments, the deflection of the shaft is achieved by using two shafts; one is a pre-curved shaft and the other is not a pre-curved shaft and has variable stiffness. In some embodiments, the deflection is performed by axially translating the shafts relative to each other. Translating the pre-curved section into the softer section of the variable stiffness shaft results in the greatest deflection, and translating the pre-curved section into the stiffer section of the variable stiffness shaft results in the least deflection.

[0166] 4. A combination of any of the above to generate deflection, e.g., both shafts are pre-curved, have variable stiffness, or both, with either rotation, axial translation, or both.

[0167] 5. Two or more coaxial tubes. In some embodiments, deflection of the shaft is accomplished by using two coaxial tubes. The stiffness of one tube is not uniform along the circumference of the cross section of the tube. In some embodiments, varying the stiffness along the circumference of the cross section of the tube can be achieved by changing the material composition and / or structure of the cross section, by selectively removing material around the circumference, or a combination thereof. In some embodiments, deflection is achieved by axially translating one of the tubes relative to the other, such that the shaft deflects toward the softer side of the variable stiffness tube when under compression and toward the stiffer side of the tube when under tension.

[0168] In some embodiments, deflecting the shaft is accomplished by using one or more of the methods described above when both tubes have variable stiffness along the circumference and the tubes are assembled with the stiff sides misaligned.

[0169] In some embodiments, deflecting the shaft is accomplished by imparting a pre-curve to the shaft or by varying stiffness around the circumference in multiple sections, using one or more of the methods described above (pre-curved shaft or variable stiffness around the circumference). In some embodiments, the pre-curve and the various stiffnesses or different sections can be aligned or oriented in different directions.

[0170] In some embodiments, the steering action is one or more of the following:

[0171] 1. Clockwise and counterclockwise rotation of the shaft.

[0172] 2. Advancement of the shaft forward or backward.

[0173] 3. Deflecting the tip, for example, by using one or more pull wires.

[0174] 4. Unidirectional deflection, for example using a single pull wire.

[0175] 5. Bidirectional deflection, for example by using two pull wires.

[0176] 6. Multi-directional deflection. For example, i. By using more than two pull wires, e.g., four wires in two perpendicular planes, allowing for deflection and straightening in two planes, in two directions within each plane, when pulling one wire per plane at a time while releasing the opposite wire.

[0177] ii. The use of more than two pull wires, for example three or four pull wires, distributed around the shaft axis, allowing deflection in any direction by a combination of pulling on one or more wires.

[0178] 7. Deflect the tip using one or more pull wires to achieve out-of-plane, three-dimensional deflection.

[0179] i.Dual pull wires in one plane, which is offset from the plane of symmetry of the shaft, allowing out-of-plane deflection in two directions, which are out-of-plane and not opposite each other.

[0180] ii. one or more pull wires are connected to a shaft having a non-uniform stiffness around the circumference of the cross section of the shaft. In some embodiments, varying stiffness around the circumference of the cross section of the tube can be achieved by varying the material composition and / or structure of the cross section, by selectively removing material around the circumference, or by a combination of these methods. In some embodiments, the deflection direction is determined by the circumferential position of the pull wire compared to the stiffness distribution around the circumference.

[0181] iii. Deflection in the manner described above can be achieved by varying the cross-section of the shaft along its axis by changing either the directionality of stiffness within the cross-section and / or the overall stiffness of the cross-section and / or the position of the pull wire within the cross-section, creating deflections in various directions along the axis of the catheter and allowing for three-dimensional out-of-plane deflection.

[0182] A typical tap-to-drive interface In some embodiments, the system includes a user interface configured to allow a user to control the electromechanically driven intraluminal device by indicating a destination. In some embodiments, the intraluminal device is advanced using other drive methods, such as by applying a magnetic field to a magnetic device, or by using pneumatic or hydraulic pressure to drive the device. In some embodiments, the operator drives the system to navigate the instrument tip to a location within the organ by indicating to the system the desired final location and orientation of the instrument tip. In some embodiments, once the operator indicates the desired destination to the system, the system is triggered to manipulate and drive the instrument using AI or other methods such that the resulting location is the requested location and orientation within the body. In some embodiments, safety mechanisms are in place to prevent undesired motion.

[0183] In some embodiments, the operator marks the desired end location and direction of the device, for example by tapping on a point in a 3D map representing the endoluminal structure displayed on a touch screen. In some embodiments, this causes the system to steer the tip of the device to the appropriate destination location within the organ. In some embodiments, the same is accomplished, for example, by clicking a mouse pointer on a location on a computer screen displaying a representation of the anatomical structure, for example a CT slice (or an MRI scan, or an angiogram, etc.). In some embodiments, for example, the operator indicates the location to the system by selecting a predefined location, for example a pulmonary bronchial bifurcation, or for example a vessel bifurcation, for example an anatomical landmark, or for example a predefined target or tagged location, from a menu or other UI element. In some embodiments, optionally, a destination location is automatically suggested by the system, such as a location that is automatically identified as a suspicious lesion. In some embodiments, optionally, the operator indicates the destination by issuing a voice command. It is understood that these embodiments are provided by way of example, and that further embodiments of the invention are possible within the scope of the invention.

[0184] In some embodiments, the system displays curved planar reconstruction type views generated by multiple segments of CT planes (or other imaging modalities) "stitched" together to form a continuous 2D view, e.g., from the trachea to the target in the case of the lungs, or from an entry port in the femoral artery to a target in the cerebral vasculature. In some embodiments, such views, e.g., following a pre-planned path, allow the user to see the anatomical details encoded in the imaging while focusing on the path to the target. In some embodiments, at each branch point, the view displays only the "right" option to the target. In some embodiments, taking a "wrong turn" is intuitively detectable when the tip of the navigation device leaves the displayed imaging plane. In some embodiments, optionally, a warning to the user may also be displayed in such cases. In some embodiments, this view may be used to indicate to the system the destination of the next segment of navigation; e.g., directly to the target, by pointing at the target, or, e.g., by having multiple waypoints at different points along the path (e.g., each lumen branch point). In some embodiments, this potentially allows the operator to easily select a "progress bar" style point to advance the device. In some embodiments, waypoints may be reached incrementally; the user simply commands the system to proceed to the next waypoint until the target is reached. In some embodiments, the view is compact and encodes all information relevant to the physician to monitor the semi-autonomous navigation process. This includes all surrounding anatomical features (as seen in the displayed CT strip or other imaging modality used) as well as the final target. In some embodiments, when the user indicates a destination, such indication may be to an intraluminal location, or an extraluminal location, or other unsafe or dangerous location, and the system warns, limits, and / or prevents navigation according to safety limits or other considerations.In some embodiments, such limits may be fixed by the manufacturer and / or pre-operatively determined by the operator and / or may be set ad-hoc by the operator (e.g., by a confirmation message invoked in response to an operator action). In some embodiments, such safety mechanisms are optionally configured or overridden when appropriate operator permission is given. In some embodiments, for example, the system may interpret any point indicated on the graphical user interface as being within the lumen, and thus may match a point indicated outside the lumen with the closest point inside the lumen on the lumen tree. In this example, the system may then position the catheter tip such that it is pointed precisely towards the point indicated by the user outside the lumen. In this example, the system may indicate a corrected position compared to the originally indicated position. In some embodiments, other indications may be made to inform the user that an alternative location has been selected. In some embodiments, the system may display a magnification of the area of ​​interest to allow the user to precisely point to the tip destination and alignment location. For example, this may be performed using a "magnifying glass" style view that is invoked once the user specifies the target destination. In some embodiments, this magnified view then allows fine-tuning of the required position, or this may be accomplished by a "first-person" style view that helps the operator select the correct tip orientation, for example on a 3D render of the lesion.

[0185] In some embodiments, the system is triggered to stop forward movement according to a predefined maximum travel distance. For example, the driven device is only allowed to travel a limited distance before waiting for further operator commands. In some embodiments, a final destination may be shown, but performed one distance at a time, thus providing greater control. In some embodiments, a safety zone may be shown on the 3D map, where automatic movement is permitted, but outward movement must be manually controlled.

[0186] In some embodiments, the interface is limited by a safety mechanism in the form of a deadman switch type control, which allows movement of the device tip only while a trigger switch is engaged and is overridden by a spring-loaded action. Another embodiment of such a switch may be a foot paddle, which allows movement only while it is depressed. Other embodiments use other methods of push-to-actuate mechanisms.

[0187] Typical Use of the System in Vascular Clinical Applications In some embodiments, the system is used in neurovascular cases, for example, acute ischemic stroke caused by large vessel occlusion (LVO), or in other cases, for example, peripheral arterial occlusion. In some embodiments, a revascularization device is introduced to perform a thrombectomy, for example, a stent-assisted thrombectomy, or for example, a direct aspiration thrombectomy, using one or more devices (for example, a guidewire, or a microcatheter, or a reperfusion catheter, or a stent retrieval device, or others). In some embodiments, shape and location sensors are attached to each distal section, each connected to a tracking device, allowing simultaneous tracking of shape, location, and forces exerted on each other and the vessel, and also allowing the display of real-time deformation of anatomical structures, such as, for example, arteries, clots, surrounding tissues, etc. In another embodiment, for example, used in intravascular cases, the same is achieved by reconstructing the 3D shape of the device in near real time from one or more fluoroscopic projections, tracking the device and its shape, location, forces exerted on each other, and anatomical lumen, allowing the display of real-time deformation of anatomical structures, such as, for example, arteries, clots, surrounding tissues, etc. In some embodiments, reconstructing the 3D shape of the device from the fluoroscopic projections is performed by identifying the tip or full curve of the device in multiple fluoroscopic 2D projections, identifying the location of the fluoroscope in some reference coordinate system (e.g., using optical fiducials), and finding the 3D location and / or shape of the device by optimization such that the backprojected 2D device curve fits the 2D curve observed from the fluoroscopic projections.

[0188] As used herein in reference to an amount or value, the term "about" means "within 20% of."

[0189] The terms "comprises," "compring," "includes," "including," "has," "having," and their cognates mean "including but not limited to."

[0190] The term "consisting of" means "including and limited to."

[0191] The term "consisting essentially of" means that a composition, method, or structure may include additional ingredients, steps, and / or moieties, but only if the additional ingredients, steps, and / or moieties do not materially alter the basic and novel characteristics of the claimed composition, method, or structure.

[0192] As used herein, the singular forms "a," "an," and "the" include plural references unless the context clearly dictates otherwise. For example, the term "a compound" or "at least one compound" may include a plurality of compounds, including mixtures thereof.

[0193] Throughout this application, embodiments of the invention may be described with reference to a range format. It should be understood that the description in range format is merely for convenience and brevity and should not be construed as an inflexible limitation on the scope of the invention. Thus, the description of a range should be considered to specifically disclose all possible subranges as well as individual numerical values ​​within that range. For example, a description of a range such as "1-6" should be considered to specifically disclose subranges such as "1-3", "1-4", "1-5", "2-4", "2-6", "3-6", etc., as well as individual numerical values ​​within that range, e.g., 1, 2, 3, 4, 5, and 6. This applies regardless of the breadth of the range.

[0194] Whenever a range of values ​​is given herein (e.g., any pair of values ​​joined by "10-15," "10 to 15," or any other such range designation), it is intended to include any number (fractional or integer) within the limits of the stated range, including the limits of the range, unless the context clearly indicates otherwise. The terms "range / ranging / ranges" between a first denoted value and a second denoted value, and "to," "up to," "until," or "through" a first denoted value to a second denoted value (or other such range designation terms) are used interchangeably herein and are intended to include the first and second denoted values ​​and all fractional and integer values ​​therebetween.

[0195] Unless otherwise specified, the numerical values ​​used herein and any numerical ranges based thereon are approximations within the reasonable precision of measurement and rounding errors that one of ordinary skill in the art would understand.

[0196] It is understood that certain features of the invention that are, for clarity, described in the context of separate embodiments, may also be provided in combination in a single embodiment. Conversely, various features of the invention that are, for brevity, described in the context of a single embodiment, may also be provided separately, or in any suitable subset, or as preferred in any other described embodiment of the invention. Particular features described in the context of various embodiments are not to be construed as essential features of those embodiments, unless the embodiment is inoperable without those elements.

[0197] While the invention has been described in conjunction with specific embodiments thereof, it is evident that many alternatives, modifications, and variations will be apparent to those skilled in the art. Accordingly, it is intended to embrace all such alternatives, modifications, and variations that fall within the spirit and broad scope of the appended claims.

[0198] It is the intention of the applicant(s) that all publications, patents, and patent applications referenced herein are incorporated by reference in their entirety into the specification as if each individual publication, patent, or patent application was specifically and individually set forth as being incorporated herein by reference. In addition, citation or identification of any reference in this application should not be construed as an admission that such reference is available as prior art to the present invention. To the extent that a section heading is used, it should not be construed as necessarily limiting. In addition, any priority document(s) of this application are hereby incorporated by reference in their entirety into the present specification.

Claims

1. 1. A method for generating a steering plan for a self-steering endoluminal system, comprising: a. selecting accessible locations through one or more lumens in a digital endoluminal map where a self-steering endoluminal device needs to reach, the digital endoluminal map being based on a pre-operative volumetric image; b. generating a navigational movement for the intraluminal device to reach the location; c. assessing deformation to one or more lumens from the one or more lumens in the digital endoluminal map; d. updating the digital endoluminal map according to the deformation; e. updating the steering plan according to the results of the updating of the digital endoluminal map while the self-steering endoluminal system is reaching the location; 1. A method for generating a steering plan for a self-steering endoluminal system, comprising:

2. performing the navigation operation until the location is reached. The method of claim 1.

3. said updating said maneuver plan being performed in real time; The method of claim 1.

4. The method further includes assessing a level of stress applied to the lumen by the navigation operation performed by the endoluminal device; The method is performed until the stress level falls below a predetermined threshold. The method of claim 1.

5. providing the plan to the self-steering endoluminal system. The method of claim 1.

6. generating the digital endoluminal map including the one or more lumens based on the images; the image is one or more of a CT scan, an MRI scan, an angiogram; The method of claim 1.

7. generating a navigation behavior includes performing a first simulation of the navigation behavior; evaluating the deformation includes performing a second simulation of the deformation, and further including updating the digital endoluminal map according to the deformation simulated in the second simulation; assessing the stress level includes performing a simulation of the stress level, and further including updating the navigation behavior to reduce the stress level; further comprising one or more of: The method of claim 1.

8. assessing the deformation further includes assessing deformation caused by breathing, heartbeat, and other causes unrelated to the self-steering intraluminal system. The method of claim 1.

9. and assessing the deformation further comprises assessing deformation caused by the navigation operation performed by the endoluminal device. The method of claim 1.

10. further comprising tracing at least a partial curve of the endoluminal device. The method of claim 1.

11. further comprising incorporating data from said tracking into said digital endoluminal map. The method of claim 10.

12. evaluating the deformation from the one or more lumens to the one or more lumens according to a result of the tracing of the at least partial curve of the intraluminal device. The method of claim 10.

13. updating the steering plan according to a result of the tracing of the at least partial curve of the endoluminal device. The method of claim 10.

14. updating the digital endoluminal map according to a result of said tracing the at least partial curve of the endoluminal device. The method of claim 10.

15. further comprising assessing the stress level by tracing at least a partial curve of the endoluminal device. The method of claim 4.

16. further comprising actuating the steerable elongate body of the intraluminal device to reduce the stress level. The method of claim 4.

17. 1. A self-steering intraluminal system, comprising: a. an intraluminal device comprising a self-steerable elongate body; b. A computer memory storage medium comprising: i. receiving a selection of accessible locations through one or more lumens within a digital endoluminal map that the self-steering endoluminal device needs to reach, the digital endoluminal map receiving the selection of accessible locations based on a pre-operative volumetric image; ii. generating a navigation operation for the intraluminal device to reach the location; iii. Evaluating the deformation from one or more lumens to one or more lumens in the digital endoluminal map; iv. updating the digital endoluminal map according to the deformation; v. updating a steering plan according to the results of the updating of the digital endoluminal map while the self-steering endoluminal system is reaching the location; a computer memory storage medium containing instructions for: A self-steering intraluminal system comprising:

18. The computer memory storage medium comprises: a. a navigation module including instructions for generating navigation operations to be performed by the steerable elongate body of the endoluminal device to reach a selected desired location within a digital endoluminal map; b. a deformation module including instructions for evaluating deformations to one or more lumens; c. a high-level module containing instructions for receiving information from one or more of the navigation module and the deformation module and driving the steerable elongate body of the intraluminal device accordingly; d. a stress module comprising instructions for assessing a stress level exerted on the lumen by the steerable elongate body of the intraluminal device; including one or more of:

20. The system of claim 17.

19. the high-level module further includes instructions to receive information from the stress module and actuate the steerable elongate body of the intraluminal device accordingly.

20. The system of claim 18.

20. the intraluminal device including one or more sensors and at least one external transmitter for monitoring the location of the intraluminal device during the navigation operation; 20. The system of claim 17.

21. The navigation module includes: a. generating navigational movements to be performed by the steerable elongate body of the endoluminal device to assist in reaching a selected desired location within a digital endoluminal map; b. performing a first simulation of the navigation operation; [0033] comprising instructions to perform one or more of:

20. The system of claim 18.

22. The high level module comprises: generating a maneuver plan based on the received information; b. generating the digital endoluminal map including the one or more lumens based on an image, wherein the image is one or more of a CT scan, an MRI scan, or an angiogram; [0033] Further comprising instructions to perform one or more of:

20. The system of claim 18.

23. The transformation module: a. performing a second simulation of said deformation; b. updating the digital endoluminal map according to the deformations simulated in the second simulation; [0033] Further comprising instructions to perform one or more of:

20. The system of claim 18.

24. The stress module is a. performing a third simulation of said stress level; b. updating the navigation behavior to reduce the stress level; [0033] Further comprising instructions to perform one or more of:

20. The system of claim 18.

25. assessing the deformation further includes assessing deformation caused by breathing, heartbeat, and other causes unrelated to the self-steering intraluminal system.

20. The system of claim 17.

26. the intraluminal device includes one or more steering mechanisms including one or more pull wires, one or more pre-curved shafts, one or more shafts having variable stiffness along a body of the one or more shafts, and one or more coaxial tubes; one or more of the one or more pre-curved shafts and one or more shafts having variable stiffness along a body of the one or more shafts are disposed within the other; the one or more steering mechanisms are configured to cause one or more steering actions including rotating the shaft, advancing / retracting the shaft, deflecting a tip of the device, and deflecting a portion of the shaft of the device.

20. The system of claim 17.

27. the computer memory storage medium further comprising instructions for tracing at least a partial curve of the endoluminal device; 20. The system of claim 17.

28. further comprising incorporating data from said tracking into said digital endoluminal map.

28. The system of claim 27.

29. evaluating the deformation from the one or more lumens to the one or more lumens according to a result of the tracing of the at least partial curve of the intraluminal device.

29. The system of claim 28.

30. updating the steering plan according to a result of the tracing of the at least partial curve of the endoluminal device.

29. The system of claim 28.

31. updating the digital endoluminal map according to a result of said tracing the at least partial curve of the endoluminal device.

29. The system of claim 28.

32. assessing a stress level according to a result of said tracing said at least partial curve of said endoluminal device.

29. The system of claim 28.

33. and actuating the steerable elongate body of the intraluminal device includes actuating the steerable elongate body of the intraluminal device to reduce the stress level.

20. The system of claim 18.

34. and assessing the deformation further comprises assessing deformation caused by the navigation operation performed by the endoluminal device.

20. The system of claim 17.

35. 1. A method for generating steering motions for a self-steering endoluminal system, comprising: assessing deformations to one or more lumens while the self-steering endoluminal system is reaching a selected location; b. updating a digital endoluminal map according to the deformation, wherein the digital endoluminal map is based on a pre-operative volumetric image; c. generating steering movements for the endoluminal device according to the updated digital endoluminal map; 12. A method for generating steering motions for a self-steering endoluminal system, comprising:

Images