System and method for robotically locking on an endoscopic deforming target

Abstract

The invention discloses systems and methods for locking a robotic endoluminal device on a breathing target, the method comprises receiving at least one chosen target to lock onto and actuating one or more controlling elements of said robotic endoluminal device to keep a predetermined position and / or orientation between said robotic endoluminal device said the target.

Description

[0001] SYSTEM AND METHOD FOR ROBOTICALLY LOCKING ON AN ENDOSCOPIC

[0002] DEFORMING TARGET

[0003] RELATED APPLICATION / S

[0004] This application claims the benefit of priority of U.S. Provisional Patent Application No. 63 / 649,606 filed on 20 May 2024, the contents of which are incorporated herein by reference in their entirety.

[0005] FIELD AND BACKGROUND OF THE INVENTION

[0006] The present invention, in some embodiments thereof, relates to navigating an endoluminal device and, more particularly, but not exclusively, to systems and methods for robotically locking on an endoscopic breathing and deforming target and / or pathway.

[0007] During endoluminal treatments, an endoluminal device is navigated inside the body of the patient until reaching a required location where the treatment is required. Usually, the target location is located at a location where the body continues to move, for example, due to respiration. It is extremely difficult to perform a localized treatment at the right location when the tissues constantly move while also moving the endoluminal device.

[0008] Additional background art includes International Patent Application No. WO2022 / 123577 disclosing systems and methods for tracking movement of a catheter within airways of a lung. A deformable model of the lung represents airways of the lung as airway segments joined at bifurcations. Deformation of the lung model uses modification of the angles and / or positions of the airway segments with respect to each other. An initial model may be generated, for example, based on segmentation of a CT image. A baseline deformable registration of the initial model to a lung shape of the patient at the beginning of the procedure may be established from position measurements of the catheter along plurality of different pathways. Optionally, the baseline registration is dynamic according to respiratory phase. Relative to the baseline registration, realtime changes in lung shape may be modeled by using further measurements obtained using the catheter as it navigates to a target, preferably using position sensors distributed along the catheter body.

[0009] SUMMARY OF THE INVENTION

[0010] Following is a non-exclusive list including some examples of embodiments of the invention. The invention also includes embodiments which include fewer than all the features in an example and embodiments using features from multiple examples, also if not expressly listed below.

[0011] Example 1. A method for locking a robotic endoluminal device on a breathing target, comprising: a. receiving at least one chosen target to lock onto; b. actuating one or more controlling elements of said robotic endoluminal device to keep a predetermined position and / or orientation between said robotic endoluminal device said the target.

[0012] Example 2. The method according to example 1, further comprising monitoring at least one deformation of said breathing target.

[0013] Example 3. The method according to example 1 or example 2, further comprising calculating at least one first actuation instruction of said one or more controlling elements in view of said deformation of said breathing target to keep said predetermined position and / or orientation.

[0014] Example 4. The method according to any one of examples 1-3, further comprising actuating said one or more controlling elements according to said at least one first actuation instruction.

[0015] Example 5. The method according to any one of examples 1-4, further comprising monitoring an introduction of an instrument within said robotic endoluminal device.

[0016] Example 6. The method according to any one of examples 1-5, further comprising calculating at least one second actuation instruction of said one or more controlling elements in view of said introduction of said instrument to keep said predetermined position and / or orientation.

[0017] Example 7. The method according to any one of examples 1-6, further comprising actuating said one or more controlling elements according to said at least one second actuation instruction.

[0018] Example 8. The method according to any one of examples 1-7, wherein one or more controlling elements comprises are one or more pulling wires.

[0019] Example 9. The method according to any one of examples 1-8, wherein said actuating comprises moving a part of said robotic endoluminal device in a direction selected from the group consisting of: up, down, left, right, forwards, backwards and any combination thereof.

[0020] Example 10. The method according to any one of examples 1-9, wherein said actuating comprises stiffening a part of said robotic endoluminal device.

[0021] Example 11. The method according to any one of examples 1-10, wherein said breathing target is one or more of target lesions, a special tissue, a point along an anatomical pathway, any point in a lumen, any point in the tissue. Example 12. The method according to any one of examples 1-11, further comprising monitoring said robotic endoluminal device utilizing one or more additional endoluminal devices.

[0022] Example 13. The method according to any one of examples 1-12, wherein said monitoring comprises monitoring while said robotic endoluminal device interacts with a target.

[0023] Example 14. The method according to any one of examples 1-13, further comprising actuating said one or more controlling elements of said robotic endoluminal device in view of a result of said monitoring.

[0024] Example 15. An endoscopic robotic system, comprising: a. an endoscopic robotic device; b. circuitry comprising instructions and / or configured to receive instructions to perform a method comprising: i. receiving at least one chosen target to lock onto; ii. actuating one or more controlling elements of said robotic endoluminal device to keep a predetermined position and / or orientation between said robotic endoluminal device said the target.

[0025] Example 16. The system according to example 15, wherein said circuitry comprises further instructions for monitoring at least one deformation of said breathing target.

[0026] Example 17. The system according to example 15 or example 16, wherein said circuitry comprises further instructions for calculating at least one first actuation instruction of said one or more controlling elements in view of said deformation of said breathing target to keep said predetermined position and / or orientation.

[0027] Example 18. The system according to any one of examples 15-17, wherein said circuitry comprises further instructions for actuating said one or more controlling elements according to said at least one first actuation instruction.

[0028] Example 19. The system according to any one of examples 15-18, wherein said circuitry comprises further instructions for monitoring an introduction of an instrument within said robotic endoluminal device.

[0029] Example 20. The system according to any one of examples 15-19, wherein said circuitry comprises further instructions for calculating at least one second actuation instruction of said one or more controlling elements in view of said introduction of said instrument to keep said predetermined position and / or orientation.

[0030] Example 21. The system according to any one of examples 15-20, wherein said circuitry comprises further instructions for actuating said one or more controlling elements according to said at least one second actuation instruction. Example 22. The system according to any one of examples 15-21, wherein one or more controlling elements comprises are one or more pulling wires.

[0031] Example 23. The system according to any one of examples 15-22, wherein said actuating comprises moving a part of said robotic endoluminal device in a direction selected from the group consisting of: up, down, left, right, forwards, backwards and any combination thereof.

[0032] Example 24. The system according to any one of examples 15-23, wherein said actuating comprises stiffening a part of said robotic endoluminal device.

[0033] Example 25. The system according to any one of examples 15-24, wherein said breathing target is one or more of target lesions, a special tissue, a point along an anatomical pathway, any point in a lumen, any point in the tissue.

[0034] Example 26. The system according to any one of examples 15-25, wherein said circuitry comprises further instructions for monitoring said robotic endoluminal device utilizing one or more additional endoluminal devices.

[0035] Example 27. The system according to any one of examples 15-26, wherein said monitoring comprises monitoring while said robotic endoluminal device interacts with a target.

[0036] Example 28. The system according to any one of examples 15-27, wherein said circuitry comprises further instructions for actuating said one or more controlling elements of said robotic endoluminal device in view of a result of said monitoring.

[0037] Unless otherwise defined, all technical and / or scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the invention pertains. Although methods and materials similar or equivalent to those described herein can be used in the practice or testing of embodiments of the invention, exemplary methods and / or materials are described below. In case of conflict, the patent specification, including definitions, will control. In addition, the materials, methods, and examples are illustrative only and are not intended to be necessarily limiting.

[0038] As will be appreciated by one skilled in the art, some embodiments of the present invention may be embodied as a system, method or computer program product. Accordingly, some embodiments of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a “circuit,” “module” or “system.” Furthermore, some embodiments of the present invention may take the form of a computer program product embodied in one or more computer readable medium(s) having computer readable program code embodied thereon. Implementation of the method and / or system of some embodiments of the invention can involve performing and / or completing selected tasks manually, automatically, or a combination thereof. Moreover, according to actual instrumentation and equipment of some embodiments of the method and / or system of the invention, several selected tasks could be implemented by hardware, by software or by firmware and / or by a combination thereof, e.g., using an operating system.

[0039] For example, hardware for performing selected tasks according to some embodiments of the invention could be implemented as a chip or a circuit. As software, selected tasks according to some embodiments of the invention could be implemented as a plurality of software instructions being executed by a computer using any suitable operating system. In an exemplary embodiment of the invention, one or more tasks according to some exemplary embodiments of method and / or system as described herein are performed by a data processor, such as a computing platform for executing a plurality of instructions. Optionally, the data processor includes a volatile memory for storing instructions and / or data and / or a non-volatile storage, for example, a magnetic hard-disk and / or removable media, for storing instructions and / or data. Optionally, a network connection is provided as well. A display and / or a user input device such as a keyboard or mouse are optionally provided as well.

[0040] Any combination of one or more computer readable medium(s) may be utilized for some embodiments of the invention. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.

[0041] A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including, but not limited to, electromagnetic, optical, or any suitable combination thereof. A computer readable signal medium may be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

[0042] Program code embodied on a computer readable medium and / or data used thereby may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0043] Computer program code for carrying out operations for some embodiments of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0044] Some embodiments of the present invention may be described below with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0045] These computer program instructions may also be stored in a computer readable medium that can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function / act specified in the flowchart and / or block diagram block or blocks.

[0046] The computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0047] Some of the methods described herein are generally designed only for use by a computer, and may not be feasible or practical for performing purely manually, by a human expert. A human expert who wanted to manually perform similar tasks might be expected to use completely different methods, e.g., making use of expert knowledge and / or the pattern recognition capabilities of the human brain, which would be vastly more efficient than manually going through the steps of the methods described herein.

[0048] BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

[0049] Some embodiments of the invention are herein described, by way of example only, with reference to the accompanying drawings. With specific reference now to the drawings in detail, it is stressed that the particulars shown are by way of example and for purposes of illustrative discussion of embodiments of the invention. In this regard, the description taken with the drawings makes apparent to those skilled in the art how embodiments of the invention may be practiced.

[0050] In the drawings:

[0051] Figure 1 is a schematic representation of an exemplary robotic endoluminal device, according to some embodiments of the invention;

[0052] Figure 2 is a schematic representation of an endoluminal device not pointing towards a target, according to some embodiment of the invention;

[0053] Figure 3 is a schematic representation of an endoluminal device pointing towards a target, according to some embodiment of the invention; and

[0054] Figure 4 is a flowchart of an exemplary method, according to some embodiments of the present invention.

[0055] DESCRIPTION OF SPECIFIC EMBODIMENTS OF THE INVENTION

[0056] The present invention, in some embodiments thereof, relates to navigating an endoluminal device and, more particularly, but not exclusively, to systems and methods for robotically locking on an endoscopic breathing and deforming target and / or pathway.

[0057] Overview

[0058] An aspect of some embodiments of the invention relates to robotically controlling a robotic endoluminal device to cause, at least a distal end of the device, to keep a constant direction towards a chosen target, when the device is in a non-stable, optionally constantly moving, environment inside the patient.

[0059] An aspect of some embodiments of the invention relates to a method for robotically locking an endoscopic elongated device (robotic endoluminal device) on a breathing and deforming target and / or pathway. In some embodiments, the method comprises one or more of: identifying and selecting a target in a preoperative CT scan, navigating an endoscopic elongated device through lumen structure to the target using endoscopic robotic navigation system, instructing the robotic endoluminal device to lock on the target and / or pathway, tracking the breathing and deforming target and / or pathway position in 3D, for example, using deformable registration and / or deformation tracking algorithms, instructing the robotic endoluminal device to maintain its position and orientation relative to the real-time deforming target and / or pathway, optionally, while dynamically controlling the stiffness of its bendable section, and interacting with the target using endoscopic working channel tools (instruments) while maintaining the robotic endoluminal device’s position and orientation relative to the target.

[0060] An aspect of some embodiments of the invention relates to a method for robotically locking an endoscopic elongated device (robotic endoluminal device) on a target in a breathing and deforming target and / or pathway. In some embodiments, the method comprises one or more of: identifying and selecting a target in a preoperative inner body scan (such as a radiographic scan); navigating a robotic endoluminal device through lumen structure to the target using endoscopic robotic navigation system; instructing the robotic endoluminal device to lock on the target and / or pathway; tracking a position of a target and / or pathway in a breathing and / or deforming three- dimensional environment; and maintaining position and orientation of the robotic endoluminal device relative to the real-time position-deforming target and / or pathway.

[0061] In some embodiments, the method comprises interacting with the target using endoscopic working channel tools (instruments) while maintaining the robotic endoluminal device’s position and orientation relative to the target.

[0062] In some embodiments, the method comprises using deformable registration and / or deformation tracking algorithms.

[0063] In some embodiments, the method comprises dynamically controlling the stiffness of the bendable section of the robotic endoluminal device.

[0064] An aspect of some embodiments of the invention relates to locking a robotic endoluminal device’s position and orientation on a target in a deforming position and / or in a breathing environment, which is potentially advantageous in order to potentially increase the accuracy of the position of an inserted tool (instrument) relative to the target center, as well as to reduce the forces which are applied to the surrounding tissues. In some embodiments, making the robotic endoluminal device’s bending section stiffer, without correcting its shape and position relative to the deforming surrounding anatomy, may exert forces on the surrounding tissues which are potentially harmful for a patient. In some embodiments, the system is configured to dynamically control the robotic endoluminal device’s stiffness based on tool-tracking algorithms, and to dynamically control the robotic endoluminal device’s position and orientation relative to the target based on real-time deformable registration and / or deformation-tracking algorithms. In some embodiments, by doing so, the system is then able to “flow” with the anatomy (e.g., not to resist to anatomical breathing or other types of anatomical motion), while maintaining position and orientation relative to a moving target, thus providing seamless interaction with the moving target to the physician, while reducing the forces which are applied to surrounding tissues.

[0065] In some embodiments, a target may be generalized to be any anatomical feature of interest, not necessarily a target lesion or a special tissue. In some embodiments, for example, a target may be a waypoint along an anatomical pathway. In some embodiments, a preoperative plan may be generated prior to the procedure, either automatically or by a user physician, which contains an anatomical pathway which leads to a target (for example, a lesion, a stroke, aneurysm or other anatomical target of interest). For example, in the setting of lung navigation, an anatomical pathway to a target lesion may be created automatically or manually prior to a procedure, based on a patient’ s preoperative CT or MRI scan. In the setting of endovascular navigation, an anatomical pathway to a target lesion or other anatomical target may be created automatically or manually prior to or during the procedure, based on a patient’ s preoperative CT or MRI scan, or can be created during procedure using angiograms. In some embodiments, the pathway to target may be a path through a complex lumen structure which leads from an entry (for example, from the patient’s trachea or a patient’ s artery or vein) to a potentially peripheral target (for example, to a target lesion in the lung or a stroke or aneurysm in the brain). In some embodiments, during a navigational procedure, the physician may be presented with a pathway to the target and may steer a robotic endoscope (or robotic endoluminal device, such as an endovascular device or a neurovascular device) through a lumen structure to follow a preplanned preoperative pathway until reaching a potentially peripheral target. In some embodiments, as described herein, when reaching the target, the robotic system may lock on the target such that the robotic endoluminal device always points to the target. In some embodiments, additionally or alternatively to locking on the marked target, a virtual target can be used such as a waypoint along a pathway to the target, and the robotic endoluminal device may be instructed to lock on a virtual target rather than or in addition to just locking on the endpoint target. In some embodiments, the robotic endoluminal device may be instructed to lock on a target waypoint which is in front and close (for example, 1 mm close, 2 mm close, 3 mm close, 5 mm close) to the robotic endoluminal device’s current tip position along a pathway to the target. In some embodiments, the robotic endoluminal device’s tip position along a pathway to the target can be computed by finding the nearest pathway point relative to the robotic endoluminal device’s tracked tip position and / or orientation, or by using K-nearest neighbor methods or by any other suitable method. In some embodiments, the virtual pathway to the target may be virtually deformed by deformation tracking algorithms, similarly to tracking the deformation of the endpoint target. In some embodiments, by instructing the robotic endoluminal device to lock on a close (or closest) waypoint along the pathway to target, the robotic endoluminal device then maintains its position and / or orientation relative to the pathway to the target. This potentially aligns the robotic endoluminal device relative to the pathway to the target, such that the user physician no longer has to steer the robotic endoluminal device manually (using a trackball, a joystick device, a tablet or by any other user interface device). In some embodiments, by aligning the robotic endoluminal device relative to a pathway to the target, the user then only needs to instruct the robotic endoluminal device to move forward or backwards, or the robotic endoluminal device may move forward autonomously, which may potentially reduce procedure times and increase procedure yields. In some embodiments, locking on a pathway point is done using same target locking methods as described herein, with the only exception that the target position is now taken as a waypoint along a pathway to the target, and the waypoint along the pathway to the target updates as the robotic endoluminal device progresses in its way along the pathway and towards the target. In some embodiments, enabling lock on target methods to an updating waypoint along a pathway (which is for example, 1 mm, 2 mm, 3 mm, 5 mm, 10 mm ahead of the robotic endoluminal device’ s tip along the pathway) provides a lock on pathway method for automatically steering a robotic endoluminal device while navigating to a target. In some embodiments, enabling deformation tracking algorithms to the entire pathway to the target, along with the endpoint target, provides a lock on deforming pathway method which is beneficial while navigating in deforming organs.

[0066] In some embodiments, when the robotic endoluminal device reaches the endpoint target (which is the endpoint of the pathway) the lock on pathway then may serve as a lock on target method, as explained herein, by just considering the closest waypoint along the pathway as the endpoint target.

[0067] In some embodiments, by locking on the pathway to target, the robotic endoluminal device may then just be instructed to move forward or backwards to reach a potentially peripheral and deforming (or breathing) target. In some embodiments, the robotic endoluminal device may be automatically instructed to move forward or backwards towards a potentially peripheral and deforming (or breathing) target, such that the navigation becomes partially or fully autonomous.

[0068] In some embodiments, by keeping the robotic endoluminal device aligned with a deforming pathway, and by instructing the robotic endoluminal device to move forward towards a target, the navigation to a potentially peripheral and deforming (or breathing) target can be autonomous, and the robotic endoluminal device may lock on the deforming (or breathing) upon reaching the endpoint target. In some embodiments, the user physician can then interact with the target using working channel tools while the robotic endoluminal device keeps its position and orientation relative to the deforming (or breathing) target.

[0069] In some embodiments, interaction with the target can be done robotically, for example, using robotic working channel tools (instruments), such that the physician does not necessarily need to manually interact with the target.

[0070] In some embodiments, working channel tools (instruments) can be inserted robotically through the robotic endoluminal device’s working channel and to a target lesion, while the robotic endoluminal device keeps its position relative to the target, such that the biopsy and / or treatment procedure can be autonomous.

[0071] In some embodiments the system described herein is capable of autonomous robotic navigation. One possible advantage of such system performing autonomous navigation by locking on a deforming or breathing target or pathway is that autonomous navigation is performed using a fully deterministic analytical computation.

[0072] In some embodiments, while locking on a target or on a pathway, the user physician may still control the steering of the robotic endoluminal device through a user interface device such as a joystick device, trackball device, touchscreen or any other suitable device. In some embodiments, while locking on a target or on a pathway the user may steer the robotic endoluminal device relative to its locked position and / or orientation, such that the user just adds delta positions and / or orientations to the locked position and / or orientation of the robotic endoluminal device relative to the target or to the pathway. In some embodiments, this can be used to point the robotic endoluminal device to a slightly or completely different position, for example, a nearby position, relative to the locked target or pathway. In some embodiments, this enables manual steering control while locking on a potentially deforming (or breathing) target or pathway, and thus provides greater flexibility for the user physician, who may wish to correct the robotic endoluminal device’ s position and / or orientation or to approach the target and / or pathway from a slightly different angle while being locked on a target and / or pathway. In some embodiments, the user may override the lock on target and / or pathway mechanism by manually steering the robotic endoluminal device, for example using a trackball or joystick device controller. In this case, the user is able to take full manual control over the robotic endoluminal device by performing manual steering using a user interface controller. In some embodiments, the user may navigate to a target using a 1-axis controller, such as a scroll wheel, which instructs the robotic endoluminal device to go forward or backward, while the robotic endoluminal device is instructed to always lock on the pathway and / or target, and automatically performs the necessary steering actions to do so. In this case, the user only controls the linear movement of the robotic endoluminal device using a scroll wheel (or any other suitable device) and does not need to control the steering of the robotic endoluminal device. In another embodiment, the robotic endoluminal device does not perform any steering action without an explicit user action. In that case, the robotic endoluminal device holds its position while until the user instructs it to move forward, for example, using a scroll wheel or any other suitable user interface controller. In some embodiments, when the user instructs the robotic endoluminal device to move forward, the robotic endoluminal device will perform the suitable action which is one of: align with the target and / or pathway, move linearly forward, move linearly backward. In this case, the robotic endoluminal device performs the next action necessary to lock on the target and / or pathway as a result of a user action through a user interface controller, and does not deflect or move otherwise. In some embodiments, when the user scrolls a wheel forward, the robotic endoluminal device aligns with the pathway and moves linearly forward only when the robotic endoluminal device is sufficiently aligned with the pathway to follow the pathway with great certainty. This can help avoid situations where the robotic endoluminal device is not deflected enough and may end up taking a wrong turn near a bifurcation along the pathway. In another embodiment, a combined motion is used, where the robotic endoluminal device both deflects and moves linearly along the pathway while the user scrolls a scroll wheel, potentially shortening the time needed to reach a target. In some embodiments, the robotic endoluminal device deflects and moves only when the user performs a user interface action, such as scrolling a scroll wheel, and moves in accordance with the magnitude of the user interface action. In some embodiments, in the case of a 1-axis scroll wheel or 2-axes trackball device, the robotic endoluminal device performs a deflection or movement action which is correlative to the amount of scrolling or rolling of the scroll wheel and / or trackball device. In some embodiments, when the user rolls a scroll wheel backwards, the robotic endoluminal device moves backwards along the pathway, potentially avoiding applying unnecessary forces on the patient’s tissue. In some embodiments, the robotic endoluminal device may deflect and move faster along the pathway as the user performs faster user actions, such as rolling a scroll wheel faster. In some embodiments, the robotic endoluminal device holds still if the user does not perform any user interface action. In some embodiments, the robotic endoluminal device may perform deflection actions autonomously, while holding its linear position still. In that case the user actions only control the linear movement of the robotic endoluminal device while the robotic endoluminal device deflects automatically to lock on the target and / or pathway. In some embodiments, the robotic endoluminal device performs deflection actions automatically only when reaching the target, to lock on the target, but deflections actions along the pathway are performed only as a result of explicit user interface actions. This provides flexibility and safety for the user, by not moving the robotic endoluminal device automatically during navigation but only as a result of explicit user interface action, while performing automatic deflection actions while reaching the target, to allow for accurate target interaction as working channel instruments are being inserted. In some embodiments the robotic endoluminal device performs micro linear motions to lock on a target, for example, on a breathing target. For example, lung lesions may move linearly forward and backward relative to the robotic endoluminal device’s tip. In this case, the system may track the target movement (for example, using deformation and / or breathing tracking algorithms) and perform micro linear movement actions such that the robotic endoluminal device not only locks its orientation relative to the moving target, but also keeps its linear position relative to the linearly moving target. In another example, a stroke or aneurysm targets or other endovascular or endoscopic targets may deform and move relative to the robotic endoluminal device’s tip, not necessarily due to patient’s breathing which may be less dominant in organs other than the lung. In this case, the system may still track the target movement and perform micro linear and / or deflection movement actions such that the robotic endoluminal device keeps its position relative to the moving target, with or without deformation due to breathing.

[0073] Any combination of the above manual and / or automatic, explicit and / or implicit, deflection and / or linear, lock on target and / or pathway mechanisms can be used, according to some embodiments of the present disclosure.

[0074] Before explaining at least one embodiment of the invention in detail, it is to be understood that the invention is not necessarily limited in its application to the details of construction and the arrangement of the components and / or methods set forth in the following description and / or illustrated in the drawings and / or the Examples. The invention is capable of other embodiments or of being practiced or carried out in various ways. Introduction

[0075] Without being bound to theory, robotic endoluminal systems are navigated towards an anatomical target which was identified, for example, in a preoperative CT scan, in order to perform one or more medical interventions. The target may be a peripheral target, for example, a peripheral lung lesion or nodule, or a stoke or aneurysm in the brain. For example, in the setting of the lung, a robotic bronchoscope (referred just as robotic endoluminal device) is navigated to a peripheral lung lesion in order to sample localized biopsy of that target lesion, or in order to deliver localized therapy to that lesion. Upon reaching the target, the physician uses a working channel which exists inside the robotic endoluminal device to introduce working channel instruments (just referred to as “instruments”) from an orifice in the robotic endoluminal device’s proximal end (outside the patient) directly into the target, at the distal end of the robotic endoluminal device. In order to provide treatment to the target (i.e., accurately interact with the target tissue instead of mistakenly applying the instrument to a non-target tissue) it is beneficial that the robotic endoluminal device will point directly to the target while the instrument is being introduced, such that the instrument will interact with the target as it extends out of the robotic endoluminal device’s tip. In the setting of an endovascular procedure, such as a neurovascular procedure or other endovascular procedures, a robotic catheter (robotic endoluminal device) such as a catheter or a microcatheter or a guidewire is navigated to a peripheral target in the brain or in other organs, for example, to a stroke or an aneurysm or a lesion or other suitable targets, to provide localized therapy to that target. In order to provide accurate treatment to the target it is beneficial that the robotic endoluminal device will point directly to the target while treatment is being delivered.

[0076] General description of an exemplary solution

[0077] The solution provided by the present invention comprises a system and method for robotically locking a robotic endoluminal device on a breathing and deforming target and / or pathway.

[0078] In some embodiments, the system is configured to allow the physician to manipulate the robotic endoluminal device while inserting the instrument such that the robotic endoluminal device keeps pointing at the target. In some embodiments, the system is configured to allow the physician to apply corrections to the robotic endoluminal device’s position and orientation either manually or robotically based on feedback received from a localization system (which may show both the robotic endoluminal device’s tip position and / or shape as well as the target location), and / or from an imaging device (such as REBUS, Fluoroscopy, Cone-beam CT or any other suitable imaging device), and / or using the robotic endoluminal device’s camera or using any other suitable feedback information.

[0079] In some embodiments, the system provides the physician with proper feedback to enable correction of the robotic endoluminal device’s position and orientation such that the robotic endoluminal device always points to the target, to make sure that the inserted instrument interacts directly with the target. In some embodiments the system may display the robotic endoluminal device video image to the physician, and the target may be visible in the endoscopic image, and the physician may then manipulate the robotic endoluminal device such that the target is centered on the endoscopic image. In some embodiments, the robotic endoluminal device may be tracked by a localization system. The robotic endoluminal device’s tip position, and / or shape can be presented to the user on a system display, for example, in 3D, while also displaying the 3D position of the target and / or pathway, such that the physician understands the robotic endoluminal device’s position and orientation relative to the target. In some embodiments, the target and / or pathway’s deformed position may be tracked using deformation tracking algorithms and displayed to the physician in their deformed position inside the deforming anatomy, in 2D and / or in 3D, alongside with the robotic endoluminal device’s position and / or shape, such that the physician is then able to understand the device’s position and / or shape relative to the deforming target and / or pathway and perform necessary manipulations such that the robotic endoluminal device points to the target and / or aligns with the pathway. In some embodiments, the localization system may be used to track the 3D position and / or shape of the robotic endoluminal device, and a vector can be displayed on the screen, which shows the direction of the robotic endoluminal device’ s tip to predict the direction in which an inserted working channel instrument will follow relative to the target. In some embodiments, the vector may be displayed alongside with a deformation tracked target, which may be deformed and breathing as being tracked by the system, such that the user understands its relative position to the deforming and / or breathing target. The physician can use this information to understand if the robotic endoluminal device points towards the target center, and correct the robotic endoluminal device’s position and / or orientation such that the displayed vector points to the target. In some embodiments, the system may display the offset between the displayed vector and the target center, quantitively, such that the physician is aware of the expected error while interacting with the target. In some embodiments the user may correct the robotic endoluminal device’s position and / or orientation by using robotic user interface controllers to deflect the robotic endoluminal device and / or move it linearly forward / backward such that it points to the target. In some embodiments the system corrects the robotic endoluminal device’s position automatically by using feedback information from a localization and / or deformation tracking system and performs steering and / or linear motion actions to keep the robotic endoluminal device locked on the target (pointing to the target).

[0080] Exemplary robotic endoluminal device

[0081] Referring now to Figure 1, showing a schematic representation of an exemplary robotic endoluminal device, according to some embodiments of the invention. In some embodiments, an exemplary robotic endoluminal device 100 comprises an elongate body 102 configured to be steered in order to reach a certain location within the body. In some embodiments, the robotic endoluminal device 100 comprises one or more sensors 104, for example at the most distal end of the endoscopic robotic device 100 and along the elongated body 102. In some embodiments, the one or more sensors 104 allow for the tracking of the elongated body 102, including the tridimensional configuration (bending and / or twisting of the elongated body) of the robotic endoluminal device 100 inside the body of the patient. In some embodiments, the sensors 104 are a plurality of digital DC magnetometers. In some embodiments, one or more transmitters 106 are positioned outside the patient and are used to generate magnetic fields, optionally low frequency magnetic fields (for example, between 30Hz and 100Hz, or 100Hz and 200Hz, or 200Hz and 500Hz or 500Hz and 1kHz). In some embodiments, the robotic endoluminal device 100 comprises one or more cameras 108 at the distal end, configured to allow a user to see during the navigation and actuation of the robotic endoluminal device 100. In some embodiments, the distal end of the robotic endoluminal device 100 comprises a bending section, which allows the steering of the robotic endoluminal device 100. In some embodiments, the robotic endoluminal device 100 comprises one or more steering mechanisms 114 which allow a user to direct, for example, the distal end of the endoscopic robotic device 100 (and / or the bending section at the distal end) towards a certain direction in order to facilitate the navigation within the body of the patient. In some embodiments, exemplary steering mechanism can be one or more pull-wires. In some embodiments, the steering mechanisms 114 are actuated from a handle 110, by motors (since it is a robotic endoluminal device), or optionally manually by a user. In some embodiments, additionally, the robotic endoluminal device 100 comprises an optionally motorized advancement / retraction mechanism configured to move the robotic endoluminal device 100 forwards or backwards during the navigation towards the desired location inside the body of the patient. In some embodiments, the robotic endoluminal device 100 is connected to one or more computers 112 configured to control the overall operation of the system. In some embodiments, the one or more computers 112 can be or include one or more of a cellphone, a tablet, a touch screen, a regular computer operated with keyboard and mouse, a virtual reality system, a voice commanded system and any combination thereof. For the explanations below, these “one or more computers” will be referred just as “electronic devices” or where necessary a specific electronic device will be mentioned. In some embodiments, the robotic endoluminal device 100 allows for the insertion of an instrument 116 into the robotic endoluminal device 100, for example, from outside the body of the patient. In some embodiments, the instrument 116 extends within the elongated body 102 until reaching the distal end of the robotic endoluminal device 100. In some embodiments, the robotic endoluminal device 100 comprises a dedicated opening at the distal end (not shown in Figure 1) from which the instrument exits and the user is allowed to perform treatment actions using the instrument 116.

[0082] Exemplary control of the robotic endoluminal device for treatment with instruments

[0083] In some embodiments, the system is configured to automatically maintain a position and orientation of the distal end of the robotic endoluminal device relative to an anatomical target (just referred to “target locking mode”). In some embodiments, optionally, the system automatically detects that the distal end of the robotic endoluminal device has reached the target and automatically enters into “target locking mode” mode.

[0084] In some embodiments, where the system is a completely automated robotic endoluminal system, the system automatically directs the robotic endoluminal device towards a desired location within the body in the proximity of the target.

[0085] In some embodiments, “target locking mode” is a mode defined as performing one or more actions which will ensure that the distal end of the robotic endoluminal device is directed (is pointed) towards a chosen target and / or location and / or path.

[0086] In some embodiments, the one or more actions are one or more of: actuating the steering mechanism (for example, up, down, right, left), advancing / retracting the robotic endoluminal device and stiffening / loosening the elongated body of the robotic endoluminal device.

[0087] For example, when the robotic endoluminal device reaches the target, the distal bending section is automatically steered by the system, such that the distal end of the robotic endoluminal device will point directly to the target and will stay pointed towards the target. In some embodiments, the system allows for semi-automated actions, for example, a user can navigate the robotic endoluminal device towards the location manually, and once at the location, the system can automatically direct the distal end of the robotic endoluminal device towards the target. Then, the physician can insert the instrument and interact with the target, without the need to keep actuating the robotic endoluminal device to ensure correct direction of the robotic endoluminal device towards the target. In some embodiments, as part of the automated controlling of the robotic endoluminal device to stay locked relative to the target, the robotic endoluminal device is configured to move in any 3D direction relative to the target while an instrument is being inserted. For example, an instrument may have a certain stiffness, and may apply forces on a bent robotic endoluminal device while the tool is being inserted into the robotic endoluminal device’s working channel. For example, an inserted instrument may cause the robotic endoluminal device’s distal bending section to straighten, or to partially straighten, or to change its position or orientation or shape while being inserted, for example, due to the instrument’s stiffness or rigidity. This may cause the robotic endoluminal device to lose the locked position and orientation relative to the target. This will cause the inserted instrument to miss the target and interact with a non-target tissue, which is undesirable in a localized biopsy or treatment procedure.

[0088] In some embodiments, a robotic endoluminal device can maintain its direction to the target by increasing the stiffness (or rigidity) of its bending section while an instrument is being inserted. In some embodiments, for example, once the robotic endoluminal device is pointed towards the target, the robotic endoluminal device’s bending section is stiffened, either manually or automatically, before an instrument is inserted. Therefore, when the instrument is then inserted, the applied forces of the instrument on the bending section (due to the instrument’s stiffness) will be less significant, which will allow the robotic endoluminal device to keep the lock (bending, shape, direction, distance, position and / or orientation) on the target. In some embodiments, a potential advantage of stiffening the bending section of the robotic endoluminal device is that it allows to keep the locking on the target, which potentially increases the probability of hitting the target while introducing a working channel tool.

[0089] In some embodiments, the bending section of the robotic endoluminal device is stiffened by pulling the robotic endoluminal device’s steering mechanisms, for example, the pull- wires. For example, in a 4-pull-wire robotic endoluminal device, 4 pull wires allow the distal end of the robotic endoluminal device to bend in any direction in space, by pulling a one or more of the wires of the 4 pull wires with a certain mixture of pulling forces. For example, pulling one or two selected wires out of 4 existing pull wires can provide bending in any 3D direction in space of the endoscope’s bending section, by selecting the one or two pull- wires to pull based on the desired bending direction. In some embodiments, pulling 4 wires simultaneously with balanced pull forces or pull lengths (for example, 3 -millimeter length pulling of all wires) does not bend the scope, but rather increases its stiffness. In some embodiments, the increase in the stiffness makes the bending more difficult. In some embodiments, this is the mechanism that allows the insertion of an instrument while avoiding the instrument to change the bending configuration of the bending section. In some embodiments, for example, the opposite is also correct, by releasing 4 wires simultaneously, by same amount of pull length (for example, 3-millimeter length releasing of all wires) the robotic endoluminal device’s bending section becomes softer, and the pull wires do not resist bending in any direction. In some embodiments, the scope can be made stiffer by other stiffening / rigidizing methods, such as dynamic rigidizing methods (see for example, WO2019 / 018682). In some embodiments, the robotic endoluminal device can be made gradually more or gradually less stiff, depending on the need.

[0090] In some embodiments, the user may override the lock on target and / or pathway mechanism by manually steering the robotic endoluminal device, for example using a trackball or joystick device controller. In this case, the user is able to take full manual control over the robotic endoluminal device by performing manual steering using a user interface controller. In some embodiments, the user may navigate to a target using a 1-axis controller, such as a scroll wheel, which instructs the robotic endoluminal device to go forward or backward, while the robotic endoluminal device is instructed to always lock on the pathway and / or target, and automatically performs the necessary steering actions to do so. In this case, the user only controls the linear movement of the robotic endoluminal device using a scroll wheel (or any other suitable device) and does not need to control the steering of the robotic endoluminal device. In another embodiment, the robotic endoluminal device does not perform any steering action without an explicit user action. In that case, the robotic endoluminal device holds its position while until the user instructs it to move forward, for example, using a scroll wheel or any other suitable user interface controller. In some embodiments, when the user instructs the robotic endoluminal device to move forward, the robotic endoluminal device will perform the suitable action which is one of: align with the target and / or pathway, move linearly forward, move linearly backward. In this case, the robotic endoluminal device performs the next action necessary to lock on the target and / or pathway as a result of a user action through a user interface controller, and does not deflect or move otherwise. In some embodiments, when the user scrolls a wheel forward, the robotic endoluminal device aligns with the pathway and moves linearly forward only when the robotic endoluminal device is sufficiently aligned with the pathway to follow the pathway. This can help avoid situations where the robotic endoluminal device is not deflected enough and may end up taking a wrong turn near a bifurcation along the pathway. In another embodiment, a combined motion is used, where the robotic endoluminal device both deflects and moves linearly along the pathway while the user scrolls a scroll wheel, potentially shortening the time needed to reach a target. In some embodiments, the robotic endoluminal device deflects and moves only when the user performs a user interface action, such as scrolling a scroll wheel, and moves in accordance with the magnitude of the user interface action. In some embodiments, in the case of a 1-axis scroll wheel or 2-axes trackball device, the robotic endoluminal device performs a deflection or movement action which is correlative to the amount of scrolling or rolling of the scroll wheel and / or trackball device. In some embodiments, when the user rolls a scroll wheel backwards, the robotic endoluminal device moves backwards along the pathway, while keeping its alignment with the path, potentially avoiding applying unnecessary forces on the patient’s tissue. In some embodiments, the robotic endoluminal device may deflect and move faster along the pathway as the user performs faster user actions, such as rolling a scroll wheel faster. In some embodiments, the robotic endoluminal device holds still if the user does not perform any user interface action. In some embodiments, the robotic endoluminal device may perform deflection actions autonomously, while holding its linear position still. In that case the user actions only control the linear movement of the robotic endoluminal device while the robotic endoluminal device deflects automatically to lock on the target and / or pathway. In some embodiments, the robotic endoluminal device is configured to perform deflection actions automatically, and the user is responsible for actuating the controls that are responsible for pushing the robot forwards or pulling it backwards. In some embodiments, the robotic system is configured to non-respond to the “forwards-command” as long as the robotic endoluminal device (or at least the tip) is not sufficiently aligned with the pathway to the target, even if the user instructs the robotic endoluminal device to move forwards using the user controller interface. For example, if the robotic endoluminal device is not yet sufficiently aligned with the pathway to a target, for example, it is not within 5 degrees alignment, or 10 degrees alignment, or 30 degrees alignment, the robotic system will ignore user actions instructing it to move forwards as to avoid navigating to a wrong turn. In some embodiments, the system displays an indication while the robotic endoluminal device is not sufficiently aligned with the pathway and / or displays that forward linear movement is temporarily disabled while the robot deflects. In some embodiments, a user controller interface with force feedback can be used to notify the user that forward movement is currently disabled. In some embodiments, the robotic endoluminal device performs deflection actions automatically only when reaching the target, to lock on the target, but deflections actions along the pathway are performed only as a result of explicit user interface actions. This provides flexibility and safety for the user, by not moving the robotic endoluminal device automatically during navigation but only as a result of explicit user interface action, while performing automatic deflection actions while reaching the target, to allow for accurate target interaction as working channel instruments are being inserted. In some embodiments, the robotic system performs lock- on-target deflection or linear actions only when the user activates a lock-on-target mode, for example, by pressing a button or a foot switch or any other suitable user interface control when reaching the target. In this case, the system does not perform lock-on-target corrective actions only until the user explicitly activates the lock-on-target mode. In some embodiments the robotic endoluminal device performs micro linear motions to lock on a target, for example, on a breathing target. For example, lung lesions may move linearly forward and backward relative to the robotic endoluminal device tip. In this case, the system may track the target movement (for example, using deformation and / or breathing tracking algorithms) and perform micro linear movement actions such that the robotic endoluminal device not only locks its orientation relative to the moving target, but also keeps its linear position relative to the linearly moving target. In some embodiments, the robotic system combines force measurements from one or more force sensors such as a load-cell, or from force estimations, which are based on an EM or fiber-optic shape and / or position localization system, or from an image-based localization system such as Fluoroscopy imaging, CBCT (Cone-beam CT), Ultrasound or any other suitable image-based localization systems. In some embodiments, such force estimations, together with the device’s tracked curve, are used to estimate the force applied on the tissue, for example, by estimating the bending radii along the tracked device or the relative positions of neighboring sensing elements to estimate the pressure (force) which is applied on the tissue and back on the device. In some embodiments, the robotic system disables deflection and / or linear motion while excessive forces are sensed, so as to reduce the forces exerted on the patient’s tissue. In some embodiments, the robotic system is configured to automatically perform deflection actions to lock on a target and / or pathway while monitoring the forces which are applied on the tissue. In some embodiments, when an excessive force is detected, the robotic system is configured to disable further motion and only allow explicit user actions, or may automatically perform a calculated motion which reduces the force on the tissue. In some embodiments, upon detecting excessive forwards linear force, the robotic system may slightly retract the robotic endoluminal device until the excessive forwards linear force is reduced, or it may disable further forwards motion until the excessive linear force is reduced (for example, through an explicit user action). In some embodiments, while locking on a target, the robotic system automatically deflects and / or moves the robotic endoluminal device to maintain its position and / or orientation relative to a potentially deforming and breathing target, while optionally interacting with that target using a working channel instrument. In some embodiments, in addition, the robotic system does so while monitoring the forces applied on the tissue during the automatic deflection and / or movement of the robotic endoluminal device tip and ensures that the performed automatic motion actions do not exert excessive force on the surrounding tissue, as being measured by one or more force sensors. Any combination of the above manual and / or automatic, explicit and / or implicit, deflection and / or linear, lock on target and / or pathway mechanisms can be used, while monitoring one or more of force, location, shape, tool position or other features of the endoluminal robotic device according to some embodiments of the present disclosure.

[0091] Exemplary assessment of position during target locking mode

[0092] In some embodiments, as mentioned above, the user navigates to a peripheral target and, upon reaching the target, the robotic endoluminal device is locked to point directly towards a target, either manually or automatically. Also as mentioned above, in target locking mode the system maintains the position and orientation of the distal end of the robotic endoluminal device relative to the target, such that the robotic endoluminal device always points to the target.

[0093] In some embodiments, the target locking mode comprises keeping a certain distance, position and / or orientation between the distal end of the robotic endoluminal device with the target. In some embodiments, the distance, position and / or orientation is kept, for example, by comparing the location of the robotic endoluminal device with the target location during the target locking mode. In some embodiments, an exemplary distance can be for example, a distance of less than 20 mm or less than 10 mm or less than 5 mm. In some embodiments, as mentioned above, during the target locking mode, the system is configured to actuate the robotic endoluminal device in one or more ways necessary in order to keep the locking, including steering, stiffening and / or linearly moving the robotic endoluminal device.

[0094] In some embodiments, the system is configured to perform corrections of the location and / or orientation of the robotic endoluminal device based on data received from one or more of: a localization system; an imaging device; the robotic endoluminal device’s camera; and any other suitable feedback information. In some embodiments, the system comprises instructions to use a control loop to control the robotic endoluminal device’s position and orientation relative to the target and correct for movements and / or rotations of the robotic endoluminal device’s tip relative to the target, such that the distal end of the robotic endoluminal device always points to the target.

[0095] In some embodiments, for example, using feedback information and / or data from an electromagnetic (EM) shape and position localization system, the system compares the position and orientation of the robotic endoluminal device’s distal end, or the position and orientation of the bending section curve relative to the target position, and apply a control loop (by using its robotic motors) to steer the robotic endoluminal device such that it faces the target as being tracked and localized by the localization system, for example, using PID (Proportional-integral-derivative control) or using any other control method. In some embodiments, the performance of the target locking mode is based on the accuracy of the feedback information. For example, a localization system used to provide feedback information for the robotic endoluminal system, provides real-time position and / or orientation of the tip of the robotic endoluminal device, and / or the robotic endoluminal device’s bending section, and / or the target position, for example, in EM transmitter coordinates or in any other suitable frame of reference (or coordinate system). In some embodiments, the localization system provides accurate feedback information. In some embodiments, a potential advantage of utilizing a localization system that provides accurate feedback information is that it allows the robotic endoluminal system to be able to accurately steer the distal end so it is always facing the target. If the localization system provides a wrong target location, then the robotic endoluminal device will be robotically steered to point to a false target location. For example, if the localization system provides a target location which is 1 cm off the true anatomical target, then the scope will be robotically steered to point to a 1 cm off false target location. In some embodiments, this may happen not because the robotic control loop is unable to mechanically keep track of the target position, as being reported by the localization system, but rather because the localization system provides wrong target locations to the robotic system.

[0096] In some embodiments, deformable registration and deformation tracking algorithms are used to accurately track the true anatomical target position, for example, in EM transmitter coordinates or in other suitable frame of reference. In some embodiments, in the setting of an EM shape and position tracking system, the robotic endoluminal device’s tracked position and shape are used to provide information to deformation-tracking algorithms to track the deformation of the organ, such as the lung, the brain, the heart, the liver, the GI tract or other organs. In addition, deformable registration algorithms are used to learn the initial deformable registration of a preoperative CT scan to the current patient’s anatomical state, as well as optionally learning a deformable breathing pattern of the patient, during or prior to navigating to a target. In some embodiments, by applying deformation tracking and / or deformable registration techniques, the target’s true anatomical position is accurately tracked (for example, within an error smaller than 1 mm, or smaller than 2 mm, or smaller than 3 mm, or smaller than 1 cm) in real-time. In some embodiments, the target’s true anatomical position is then being used as feedback information to the robotic endoluminal system’s target locking mode control loop. In some embodiments, for example, a deformation tracking algorithm tracks a target’s anatomical real-time position, for example in EM transmitter (TX) coordinates based on EM curve-tracked (where a “curve” includes a shape and a position) robotic endoluminal device shape and position, and provides that information in real-time to the robotic endoluminal system’s target locking module, which is responsible for performing and maintaining the target locking mode. In some embodiments, the target locking module may also receive the robotic endoluminal device’s shape and position, for example in EM transmitter coordinates, for an EM shape and position tracking system. In some embodiments, the target locking module may then compare the robotic endo luminal device’s shape and position relative to the target’s true anatomical tracked position and robotically correct the robotic endoluminal device’s position and orientation in real-time so that it always points to the target.

[0097] In some embodiments, the system comprises a deformation-tracking algorithm configured to track the position of a breathing target. For example, in the setting of the lung, a target may be located near the diaphragm. In this case, the target may move significantly in accordance with patient’ s breathing (for example, it may move by more than 3 mm, or more than 1 cm, or more than 2 cm), as being observed in TX coordinates (which do not breathe with the patient). In some embodiments, a breathing-aware deformation-tracking algorithm is configured to track the breathing target location in real-time, for example, using an EM shape and position tracking system, or using fiber-optic shape sensing, for example in EM TX coordinates or in other suitable frame of reference.

[0098] In some embodiments, the target locking module receives in real-time breathing target locations which may move for example forward and backward in accordance with patient breathing. In some embodiments, in order to maintain the robotic endoluminal device’s position and orientation (and, as a result, distance) relative to the target, the robotic endoluminal system moves the robotic endoluminal device linearly in accordance with the patient’s breathing, thereby keeping the distance as well as the position and orientation from the target. In some embodiments, the system is therefore configured and enabled to lock on a breathing, deformable target. In some embodiments, tracking of patient’s breathing is done using external reference patient sensors, which may be attached for example to the patient’s chest. In some embodiments, certain deformable registration algorithms may use information from such external sensors or others to keep track of patient’ s breathing during the registration process. In some embodiments, by keeping track of breathing, the deformable registration is then able to learn the patient’s breathing pattern and provide a real-time breathing deformable registration of the patient during the procedure. In some embodiments, by doing so, a target can be tracked in real-time in its true anatomical position (for example, in EM TX coordinates), even during patient’s breathing. In some embodiments, by combining deformable registration algorithms with deformation-tracking algorithms, the system can then track the organ’s registration and deformation during the procedure, while keeping track of deformation caused by both initial state of patient, breathing pattern of patient, local forces applied by the robotic endoluminal device (and / or the instrument), or any other relevant cause of deformation. In some embodiments, by combining real-time information of deformable registration and / or deformation-tracking algorithms with a robotic control loop, the robotic endoluminal system is then able to maintain the robotic endoluminal device’s position and orientation, and / or shape, facing the deforming target, even in challenging cases of a highly breathing and / or deforming target.

[0099] In some embodiments, performing the control loop to maintain the robotic endoluminal device’s shape, and / or to maintain the endoscope’s position and orientation relative to a real-time tracked target is done using a PID or using other robotic control techniques. In some embodiments, the robotic endoluminal device’s orientation is compared with a robotic endoluminal device’s desired orientation, as schematically shown for example in Figures 2 and 3.

[0100] The endoscope’s desired orientation may be the computed direction between the real-time tracked target position (Po) and the real-time tracked robotic endoluminal device’s tip position (r0). The two directions (i.e., the robotic endoluminal device’s tracked tip direction z and the desired robotic endoluminal device’s direction —2— can be compared through a vector cross product E l O-roll

[0101] P _ * which is indicative of the error between the two: E = z X — - — HPo-roll

[0102] If the directions align, then the cross product is close to zero and the control loop does not need to correct the robotic endoluminal device’s tip position and / or orientation. If the directions differ, then the cross-product results in a non-zero vector E which indicates the rotation / movement needed to be performed to the robotic endoluminal device’ s tip so it would be directed to the target. Since E is orthogonal to z, it can also be viewed as a 2D vector in the X-Y plane in the robotic endoluminal device’s tip frame of reference. The X-Y components of vector E in the robotic endoluminal device’s tip frame of reference indicate the rotation / movement which needs to be applied to the robotic endoluminal device’s tip direction z in order to align it with — - — (so it I O-ro II would point to the target).

[0103] This rotation / movement can be applied by pulling a certain mixture with certain pulling force of the robotic endoluminal device’s pull wires. For example, control loop calibration can be used to transform between pull- wire forces and / or displacements and robotic endoluminal device’s bending directions. By correlating between the two (for example, through offline or online calibration) the system can then generate an action for each cross-product vector E, such that the robotic endoluminal device would improve its position and orientation relative to the target, making a step as to point more towards the target. In some embodiments, instead of using a vector cross product to compute the error between the current position and orientation of the tip and the desired position and orientation, a 3D target point can be represented in tip coordinates and / or in camera coordinates using a 3D transformation. In some embodiments, conversion between tip coordinates, which may be tracked by an EM localization system, and camera coordinates may be available for example by knowing the mechanical construction of the robotic endoluminal device, for example, knowing that a tip EM sensor is mechanically aligned with the camera, or by pre-calibrating the camera relative to the tip sensor, or by other suitable methods. Features can then be described in tip coordinates, and / or in camera coordinates, and / or in pull-wire coordinates interchangeably. In some embodiments, the robotic endoluminal device’s tip position and orientation may be described using a 3D transformation in space, which encodes both its position and orientation, for example, as a 4x4 3D transformation matrix which includes a rotation and translation, that describes the tip’s position and orientation in some known coordinate system, for example, in EM transmitter coordinates. In addition, the position, and potentially also orientation, of the 3D target (which may be a target lesion or a waypoint along a pathway or any other suitable target) may also be described as a 3D transformation. The target may then be described in tip coordinates, which describes how it is positioned and oriented relative to the robotic endoluminal device’s tip position and orientation. In some embodiments, by describing the target position and / or orientation in tip coordinates, the target position relative to the tip (in tip coordinates) then indicates the rotation / movement which needs to be applied to the robotic endoluminal device’s tip direction z in order to align it with the target. In some embodiments, the pull-wires are positioned in known or pre-calibrated locations relative to the tip, such that the robotic system knows which combination of pull-wires to pull in order to deflect the tip in a certain direction, as described above. The robotic system can then use the target location in tip coordinates to compute the amount of pulling for each pull-wire to generate a deflection and align the tip towards the target. In some embodiments, conversion can be made between tip coordinates, camera coordinates, pull-wire coordinates, EM tip coordinates, fiber-optic shape sensor tip coordinates to convert between tracked tip position and pull-wire actions, to generate a deflection based on tip position and / or orientation or based on camera position and / or orientation. In some embodiments, the conversion between those coordinate systems can be precalibrated before procedure, according to some embodiments of the present disclosure.

[0104] In some embodiments, other methods can be used to generate deflection actions based on feedback from a localization system, for example an EM shape and position localization system, or a fiber optic shape sensor localization system, combining information of the tip and / or bending section position and shape, with 3D tracking of the target relative to the tip, and performing corrective actions to deflect the tip such that it points towards the target.

[0105] In some embodiments, the process can be repeated iteratively until the size of the error between the current tip position and orientation and the desired position and orientation (which points at the target), which may be computed using the aforementioned vector cross-product as well as other error metrics decreases and it is substantially zero, which means that the robotic endoluminal device points at the target, as depicted in Figure 3.

[0106] In some embodiments, the lock-in adjustment process can be done in real-time, based on feedback information from the real-time localization system, so that the distal end of the robotic endoluminal device always corrects its position and orientation relative to a real-time tracked target (which may deform and / or breathe).

[0107] In some embodiments, the system is configured to sense the introduction of an instrument into the robotic endoluminal device and optionally automatically make the distal bending section stiffer. In some embodiments, a potential advantage of doing this is that this can potentially prevent the bending section from losing its shape and orientation relative to the target. In some embodiments, further potential advantage is that by doing this, further controlled corrections for the bending section’s position and shape may not be necessary in a control loop, or a control loop can converge faster to always point at the target.

[0108] In some embodiments, it may be potentially disadvantageous to make the bending section stiff if unnecessary. By making the bending section stiffer, forcing a certain shape, the bending section’s forced stiff shape may deform the surrounding tissue and may thus impact a deformationtracking algorithm performance. For a deformation-tracking algorithm to perform well, it is advisable to sense the shape and position of a lumen by using a soft inserted endoscope which assumes the shape of its surrounding lumen as is, without forcing a certain other shape or position. This is also advisable for registration algorithms which match the endoscope’s sensed shape to a known shape of a lumen, for example, as appears in a patient’ s preoperative CT scan, and by that are able to localize the sensed shape scope inside a lumen structure. It is thus advisable, according to some embodiments of the present disclosure, to reduce the level of stiffness for the robotic endoscope as much as possible, and make it stiffer only when necessary, for example, during tool insertion.

[0109] In some embodiments, the system uses tool-tracking and / or tool-sensing methods to automatically sense the insertion of instruments inside the robotic endoluminal device’s working channel. For example, as schematically shown in Figure 1, an EM shape and position sensor may comprise a sensor array, located along the elongated body of the robotic endoluminal device, for example, in the form of a plurality of magnetometer sensors, assembled on an FPC and wrapped around the elongated body of the robotic endoluminal device. In some embodiments, each sensor element may be used to sense the presence of an inserted ferromagnetic working channel tool (an instrument), for example, using EM distortion detection methods, since the sensed EM fields differ greatly (for example, by more than 1% or by more than 5% or by more than 10%) from theoretically generated fields emitted by a transmitter. In some embodiments, for example, an instrument positioned inside the robotic endoluminal device’s working channel can be tracked and / or estimated by modeling the EM distortion created by the instrument on nearby EM sensors. In some embodiments, an instrument position can be tracked and / or estimated by sensing the magnetic field bias which is created by the instrument on nearby EM sensors. By sensing the insertion of an instrument, and / or by optionally modeling the distortion caused by instrument insertion, the system is then able to track and / or estimate the position of an inserted instrument along the robotic endoluminal device’s working channel. In some embodiments, when an instrument is inserted into the robotic endoluminal device’s working channel, the system tracks its position along the robotic endoluminal device’s working channel. In some embodiments, when the instrument approaches the distal end’s bending section, the system can then gradually increase the stiffness of the bending section (for example, by gradually pulling the robotic endoluminal device’s pull- wires), for example, in accordance with the instrument’s proximity to the robotic endoluminal device’s tip. In some embodiments, the system may gradually increase the bending section’s stiffness from 0% (when the instrument is a few millimeters before the beginning of the robotic endoluminal device’s bending section) to 100% (when the instrument approaches the tip of the robotic endoluminal device). In some embodiments, by doing so, the system automatically makes the distal end of the robotic endoluminal device stiffer only when needed - that is, when an instrument is inserted. In some embodiments, the distal end of the robotic endoluminal device remains soft until an instrument is inserted, in which moment making the distal end stiffer is advantageous (while engaging with the target), to resist the forces which are applied by the inserted instrument itself and through the robotic endoluminal device’s working channel, and to maintain the robotic endoluminal device’s shape during target interaction such that an optional corrective control loop will need less actions to maintain the distal end position and orientation relative to the engaged target.

[0110] In some embodiments, making the distal end’s bending section gradually stiffer as a working channel instrument is being inserted is potentially advantageous to allow the tool to easily enter the working channel and aligning it with the target only after it was inserted. If the bending section was stiff to begin with, the instrument insertion may not have been possible due to a potential curved and stiff working channel. After or during the insertion of the instrument, the bending section would become stiff gradually, allowing the tool to pass smoothly while gradually aligning it with the target, according to some embodiments of the present disclosure.

[0111] In some embodiments, additionally or alternatively to making the bending section gradually stiffer, the robotic system may gradually align the robotic endoluminal device with the target as a working channel tool is being introduced and tracked. The robotic endoluminal device may then be soft as long as an instrument is not being inserted, and as the instrument is being inserted the robotic endoluminal device may then gradually start bending to point at the target, until the instrument is fully inserted and the robotic endoluminal device fully points directly to the target.

[0112] In some embodiments, the system comprises and / or is connected to a dedicated database of known distortions of specific instruments. In some embodiments, when an unknown instrument is inserted into the robotic endoluminal device, the unknown instrument is identified, either by the user providing this information, or for example by recognizing the EM distortion created by the instrument and matching it with the database of known distortions, then the mechanical properties of the instrument inserted are known and used. In this case, the mechanical deformation created by the instrument when inserted through the robotic endoluminal device can be projected and automatically compensated in real time during the insertion of the instrument. In some embodiments, additional corrections to maintain target lock may still be required and can still be performed in the methods described herein however minimizing the required corrections can help increase the overall accuracy.

[0113] Exemplary additional devices

[0114] In some embodiments, additional robotic endoluminal devices or probe-like devices (referred hereinafter just as “additional devices”) can be inserted into the body of the patient, for example, to reach the areas surrounding and / or in the vicinity of a target. In some embodiments, such additional devices can be blunt probes, catheters, J-catheters or other suitable devices which contain position and / or shape tracking sensors, such as EM localization sensors or fiber-optic shape sensors. In some embodiments, such additional devices may be passive devices without localization sensors, which are visible under fluoroscopy. Their position and / or shape can then be tracked using real-time X-ray imaging. Each of the inserted additional devices can be EM shape and position tracked, according to some embodiments of the present disclosure, such that it is tracked and localized by the EM shape and position localization system. Each additionally inserted localized additional device adds more information to deformable registration and deformation tracking algorithms, especially in the proximity of its shape and position inside the anatomy. By deploying one or more additional devices to the proximity of the engaged target, deformation-tracking can be enhanced for that target and the deforming target can be tracked more accurately, specifically during the tool engagement stage. For example, when an instrument is engaging a target, it may deformably push the anatomical target such that it deforms locally, along with its surrounding tissue. An additional localized additional device, which may be inserted to a different or same lumen in the proximity of the target, may sense the movement caused by the interacting instrument and may provide information to the deformation-tracking algorithm which is indicative of this local deformation. The deformation-tracking algorithm may then compute and display the local deformation which is caused by the inserted working channel instrument and its manipulation of the target, which may potentially increase the accuracy of the target’s real-time tracked position in space. With the increased accuracy, the robotic system then receives as feedback a more accurate real-time target location (for example, in EM transmitter coordinates) and can use it for correcting the robotic endoluminal device’s pose relative to the target in its corrective control loop. For example, if the working channel instrument pushes the target away from the scope, the robotic system may sense that and apply one or more of a linear motion and a steering action to bring the robotic endoluminal device closer to the target, regardless of it’s being pushed back.

[0115] In some embodiments, a second or third tracked robotic endoluminal device may be inserted into an organ to a same or different lumen to provide more information to a deformation tracking algorithm while a working channel instrument interacts with the target via a first robotic endoluminal device’s working channel. In some embodiments, the second or third robotic endoluminal device is EM shape and / or position tracked, and its tracked curve provides information to a deformation tracking algorithm which improves the position accuracy of the deforming and breathing target during and before target interaction, as being tracked by the deformation tracking algorithm.

[0116] In some embodiments, the inserted working channel instrument is tracked using tool tracking methods, as mentioned above. During the interaction with the target, the tool tracked position and / or orientation may be used by a deformation tracking algorithm to improve the deformation tracking of the organ and specifically of the target. By enhancing the accuracy of the real-time tracked deformed and breathing target, the robotic system can then better compensate for the target deformation and correct the robotic endoluminal device position and orientation relative to the target in its corrective control loop. In some embodiments, the inserted working channel instrument contains a localization sensor, for example, an EM position and / or shape sensor or a fiber-optic shape sensor, which provides real-time position and / or shape tracking of the working channel instrument as it’s being inserted into the working channel and pushed towards the target. Tool position and / or shape information can then be used to track the instrument position inside the working channel and apply dynamic stiffening (rigidization) and / or steering of the robotic endoluminal device’s bending section, as well as providing information for a deformation tracking algorithm to enhance the tracking of the target’s deforming and breathing position while the tool interacts with the target.

[0117] Exemplary methods

[0118] Referring now to Figure 4, showing a flowchart of an exemplary method, according to some embodiments of the present invention. In some embodiments, an exemplary method of locking a robotic endoluminal device on a breathing target comprises one or more of the following actions:

[0119] 1. Receiving at least one chosen target to lock onto (402).

[0120] 2. Actuating one or more controlling elements of the robotic endoluminal device to keep a predetermined position and / or orientation between the robotic endoluminal device and the target (404).

[0121] In some embodiments, the method further comprises monitoring one or more of a deformation of the target, a shape, a position of device, a curve of the device, forces, direction of the tip of the device, distance, position and / or orientation from the target (406). In some embodiments, the method further comprises calculating at least one actuation instruction (408) of said one or more controlling elements to keep said predetermined distance, position and / or orientation. In some embodiments, the method further comprises performing said actuating in order to compensate for said deformation.

[0122] As used herein with reference to quantity or value, the term “about” means “within ± 10 % of’.

[0123] The terms “comprises”, “comprising”, “includes”, “including”, “has”, “having” and their conjugates mean “including but not limited to”.

[0124] The term “consisting of’ means “including and limited to”.

[0125] The term “consisting essentially of’ means that the composition, method or structure may include additional ingredients, steps and / or parts, but only if the additional ingredients, steps and / or parts do not materially alter the basic and novel characteristics of the claimed composition, method or structure.

[0126] As used herein, the singular forms “a”, “an” and “the” include plural references unless the context clearly dictates otherwise. For example, the term “a compound” or “at least one compound” may include a plurality of compounds, including mixtures thereof. Throughout this application, embodiments of this invention may be presented with reference to a range format. It should be understood that the description in range format is merely for convenience and brevity and should not be construed as an inflexible limitation on the scope of the invention. Accordingly, the description of a range should be considered to have specifically disclosed all the possible subranges as well as individual numerical values within that range. For example, description of a range such as “from 1 to 6” should be considered to have specifically disclosed subranges such as “from 1 to 3”, “from 1 to 4”, “from 1 to 5”, “from 2 to 4”, “from 2 to 6”, “from 3 to 6”, etc.; as well as individual numbers within that range, for example, 1, 2, 3, 4, 5, and 6. This applies regardless of the breadth of the range.

[0127] Whenever a numerical range is indicated herein (for example “10-15”, “10 to 15”, or any pair of numbers linked by these another such range indication), it is meant to include any number (fractional or integral) within the indicated range limits, including the range limits, unless the context clearly dictates otherwise. The phrases “range / ranging / ranges between” a first indicate number and a second indicate number and “range / ranging / ranges from” a first indicate number “to”, “up to”, “until” or “through” (or another such range-indicating term) a second indicate number are used herein interchangeably and are meant to include the first and second indicated numbers and all the fractional and integral numbers therebetween.

[0128] Unless otherwise indicated, numbers used herein and any number ranges based thereon are approximations within the accuracy of reasonable measurement and rounding errors as understood by persons skilled in the art

[0129] It is appreciated that certain features of the invention, which are, for clarity, described in the context of separate embodiments, may also be provided in combination in a single embodiment. Conversely, various features of the invention, which are, for brevity, described in the context of a single embodiment, may also be provided separately or in any suitable subcombination or as suitable in any other described embodiment of the invention. Certain features described in the context of various embodiments are not to be considered essential features of those embodiments, unless the embodiment is inoperative without those elements.

[0130] Although the invention has been described in conjunction with specific embodiments thereof, it is evident that many alternatives, modifications and variations will be apparent to those skilled in the art. Accordingly, it is intended to embrace all such alternatives, modifications and variations that fall within the spirit and broad scope of the appended claims.

[0131] It is the intent of the applicant(s) that all publications, patents and patent applications referred to in this specification are to be incorporated in their entirety by reference into the specification, as if each individual publication, patent or patent application was specifically and individually noted when referenced that it is to be incorporated herein by reference. In addition, citation or identification of any reference in this application shall not be construed as an admission that such reference is available as prior art to the present invention. To the extent that section headings are used, they should not be construed as necessarily limiting. In addition, any priority document(s) of this application is / are hereby incorporated herein by reference in its / their entirety.

Claims

WHA IS CLAIMED IS:

1. A method for locking a robotic endoluminal device on a breathing target, comprising: a. receiving at least one chosen target to lock onto; b. actuating one or more controlling elements of said robotic endoluminal device to keep a predetermined position and / or orientation between said robotic endoluminal device said the target.

2. The method according to claim 1, further comprising monitoring at least one deformation of said breathing target.

3. The method according to claim 2, further comprising calculating at least one first actuation instruction of said one or more controlling elements in view of said deformation of said breathing target to keep said predetermined position and / or orientation.

4. The method according to claim 3, further comprising actuating said one or more controlling elements according to said at least one first actuation instruction.

5. The method according to claim 1, further comprising monitoring an introduction of an instrument within said robotic endoluminal device.

6. The method according to claim 5, further comprising calculating at least one second actuation instruction of said one or more controlling elements in view of said introduction of said instrument to keep said predetermined position and / or orientation.

7. The method according to claim 6, further comprising actuating said one or more controlling elements according to said at least one second actuation instruction.

8. The method according to claim 1, wherein one or more controlling elements comprises are one or more pulling wires.

9. The method according to claim 1, wherein said actuating comprises moving a part of said robotic endoluminal device in a direction selected from the group consisting of: up, down, left, right, forwards, backwards and any combination thereof.

10. The method according to claim 1, wherein said actuating comprises stiffening a part of said robotic endoluminal device.

11. The method according to claim 1, wherein said breathing target is one or more of target lesions, a special tissue, a point along an anatomical pathway, any point in a lumen, any point in the tissue.

12. The method according to claim 1, further comprising monitoring said robotic endoluminal device utilizing one or more additional endoluminal devices.

13. The method according to claim 12, wherein said monitoring comprises monitoring while said robotic endoluminal device interacts with a target.

14. The method according to claim 12, further comprising actuating said one or more controlling elements of said robotic endoluminal device in view of a result of said monitoring.

15. An endoscopic robotic system, comprising: a. an endoscopic robotic device; b. circuitry comprising instructions and / or configured to receive instructions to perform a method comprising: i. receiving at least one chosen target to lock onto; ii. actuating one or more controlling elements of said robotic endoluminal device to keep a predetermined position and / or orientation between said robotic endoluminal device said the target.

16. The system according to claim 15, wherein said circuitry comprises further instructions for monitoring at least one deformation of said breathing target.

17. The system according to claim 16, wherein said circuitry comprises further instructions for calculating at least one first actuation instruction of said one or more controlling elements in view of said deformation of said breathing target to keep said predetermined position and / or orientation.

18. The system according to claim 17, wherein said circuitry comprises further instructions for actuating said one or more controlling elements according to said at least one first actuation instruction.

19. The system according to claim 15, wherein said circuitry comprises further instructions for monitoring an introduction of an instrument within said robotic endoluminal device.

20. The system according to claim 19, wherein said circuitry comprises further instructions for calculating at least one second actuation instruction of said one or more controlling elements in view of said introduction of said instrument to keep said predetermined position and / or orientation.

21. The system according to claim 20, wherein said circuitry comprises further instructions for actuating said one or more controlling elements according to said at least one second actuation instruction.

22. The system according to claim 15, wherein one or more controlling elements comprises are one or more pulling wires.

23. The system according to claim 15, wherein said actuating comprises moving a part of said robotic endoluminal device in a direction selected from the group consisting of: up, down, left, right, forwards, backwards and any combination thereof.

24. The system according to claim 15, wherein said actuating comprises stiffening a part of said robotic endoluminal device.

25. The system according to claim 15, wherein said breathing target is one or more of target lesions, a special tissue, a point along an anatomical pathway, any point in a lumen, any point in the tissue.

26. The system according to claim 15, wherein said circuitry comprises further instructions for monitoring said robotic endoluminal device utilizing one or more additional endoluminal devices.

27. The system according to claim 26, wherein said monitoring comprises monitoring while said robotic endoluminal device interacts with a target.

28. The system according to claim 26, wherein said circuitry comprises further instructions for actuating said one or more controlling elements of said robotic endoluminal device in view of a result of said monitoring.

Images