Autonomous navigation of a steerable catheter

Abstract

Examples of autonomous navigation, movement detection, and / or control include, but are not limited to, autonomous navigation of one or more portions of a continuum robot towards a particular target, movement detection of the continuum robot, Follow-The-Leader smoothing, and / or state change(s) for a continuum robot. Examples of applications include imaging, evaluating, and diagnosing biological objects, such as, but not limited to, for Gastro-intestinal, cardio, bronchial, and / or ophthalmic applications, and being obtained via one or more optical instruments, such as, but not limited to, optical probes, catheters, endoscopes, and bronchoscopes. Techniques provided herein also improve processing and imaging efficiency while achieving images that are more precise, and also achieve devices, systems, methods, and storage mediums that reduce mental and physical burden and improve ease of use.

Description

Autonomous Navigation of a Steerable CatheterBACKGROUNDReference to Related Applications

[0001] This application claims priority from U.S. Provisional Patent Application Serial No. 63 / 560575 filed March 1, 2024, the contents of which are hereby incorporated herein by reference.Field of the Disclosure

[0002] The present disclosure generally relates to a robotic catheter system and more particularly to a steerable catheter than can be navigated autonomously as well as methods and mediums for autonomous navigation.Description of the Related Art

[0003] Endoscopy, bronchoscopy, catheterization, and other medical procedures facilitate the ability to look inside a body. During such a procedure, a flexible medical tool maybe inserted into a patient’s body, and an instrument maybe passed through the tool to examine or treat an area inside the body. For example, a bronchoscope is an endoscopic instrument to view inside the airways of a patient. Catheters and other medical tools may be inserted through a tool channel in the bronchoscope to provide a pathway to a target area in the patient for diagnosis, planning, medical procedure(s), treatment, etc.

[0004] Robotic bronchoscopes, robotic endoscopes, or other robotic imaging devices may be equipped with a tool channel or a camera and biopsy tools, and such devices (or users of such devices) may insert / retract the camera and biopsy tools to exchange such components. The robotic bronchoscopes, endoscopes, or other imaging devices may be used in association with a display system and a control system.

[0005] An imaging device, such as a camera, may be placed in the bronchoscope, the endoscope, or other imaging device / system to capture imagesinside the patient and to help control and move the bronchoscope, the endoscope, or the other type of imaging device, and a display or monitor maybe used to view the captured images. An endoscopic camera that may be used for control may be positioned at a distal part of a catheter or probe (e.g., at a tip section).

[0006] The display system may display, on the monitor, an image or images captured by the camera, and the display system may have a display coordinate used for displaying the captured image or images. In addition, the control system may control a moving direction of the tool channel or the camera. For example, the tool channel or the camera may be bent according to a control by the control system. The control system may have an operational controller (such as, but not limited to, a joystick, a gamepad, a controller, an input device, etc.), and physicians may rotate or otherwise move the camera, probe, catheter, etc. to control same. However, such control methods or systems are limited in effectiveness. Indeed, while information obtained from an endoscopic camera at a distal end or tip section may help decide which way to move the distal end or tip section, such information does not provide details on how the other bending sections or portions of the bronchoscope, endoscope, or other type of imaging device may move to best assist the navigation.

[0007] At least one application is looking inside the body relates to lung cancer, which is the most common cause of cancer-related deaths in the United States. It is also a commonly diagnosed malignancy, second only to breast cancer in women and prostate cancer in men. Early diagnosis of lung cancer is shown to improve patient outcomes, particularly in peripheral pulmonary nodules (PPNs). During a procedure, such as a transbronchial biopsy, targeting lung lesions or nodules may be challenging. Lately, Electromagnetically Navigated Bronchoscopy (ENB) is increasingly applied in the transbronchial biopsy of PPNs due to its excellent safety profile, with fewer pneumothoraxes, chest tubes, significant hemorrhage episodes, and respiratory failure episodes than a CT-guided biopsy strategy (see e.g., as discussed in C. R. Dale, D. K. Madtes, V. S. Fan, J. A. Gorden, and D. L. Veenstra, “Navigational bronchoscopy with biopsy versus computedtomography-guided biopsy for the diagnosis of a solitary pulmonary nodule: a costconsequences analysis,” J Bronchology Interv Pulmonol, vol. 19, no. 4, pp. 294-303, Oct. 2012, doi: 10.1097 / LBR.0B013E318272157D, which is incorporated by reference herein in its entirety). However, ENB has lower diagnostic accuracy or value due to dynamic deformation of the tracheobronchial tree by bronchoscope maneuvers (see e.g., as discussed in T. Whelan, R. F. Salas-Moreno, B. Glocker, A. J. Davison, and S. Leutenegger, “ElasticFusion,” International Journal of Robotics Research, vol. 35, no. 14, pp. 1697-1716, Dec. 2016, doi: 10.1177 / 0278364916669237, which is incorporated by reference herein in its entirety) and module motion due to the breathing motion of the lung (see e.g., as discussed in A. Chen, N. Pastis, B.Furukawa, and G. A. Silvestri, “The effect of respiratory motion on pulmonary nodule location during electromagnetic navigation bronchoscopy,” Chest, vol. 147, no. 5, pp. 1275-1281, May 2015, doi: 10.1378 / CHEST.14-1425, which is incorporated by reference herein in its entirety).

[0008] Vision-based tracking (VNB), as opposed to ENB, has been proposed to address the aforementioned issue of CT-to-body divergence (see e.g., as discussed in D. J. Mirota, M. Ishii, and G. D. Hager, “Vision-Based Navigation in Image-Guided Interventions,” https: / / doi.org / 10.1146 / annurev-bioeng-071910-124757, vol. 13, pp. 297-319, Jul. 2011, doi: IO.1146 / ANNUREV-BIOENG-O7191O-124757, which is incorporated by reference herein in its entirety). Vision-based tracking in VNB does not require an electromagnetic tracking sensor to localize the bronchoscope in CT; rather, VNB directly localizes the bronchoscope using the camera view, conceptually removing the chance of CT-to-body divergence.

[0009] Depth estimation was proposed as an alternative method of VNB to further reduce the CT-to-body divergence and overcome the intensity-based image registration drawbacks (see e.g., as discussed in M. Shen, S. Giannarou, and G. Z. Yang, “Robust camera localisation with depth reconstruction for bronchoscopic navigation,” Int J Comput Assist Radiol Surg, vol. 10, no. 6, pp. 801-813, Jun. 2015,doi: 10.1007 / S11548-015-1197-Y, which is incorporated by reference herein in its entirety).

[0010] Alternatively, autonomous navigation in robotic guided bronchoscopy is a relative new concept. Sganga et al. (as discussed in J. Sganga, D. Eng, C. Graetzel, and D. B. Camarillo, “Autonomous Driving in the Lung using Deep Learning for Localization,” Jul. 2019, Accessed: Jun. 28, 2023. [Online]. Available: https: / / arxiv.0rg / abs / 1907.08136vi, which is incorporated by reference herein in its entirety) proposed the first attempt to autonomously navigate through the lung airways having as primary focus to improve the intraoperative registration between CT and live images and then attempt to autonomously navigate to the target. The Sganga, et al. method was limited to only 4 airways. However, the Sganga, et al. method requires a great co-registration between the live image and the pre-operative CT scan which can be detrimentally affected by the same drawbacks as VNB.

[0011] Other efforts have been made not to autonomously navigate through the airways but to automatically control the catheter tensioning system. Jaeger, et al. (as discussed in H. A. Jaeger et al., “Automated Catheter Navigation With Electromagnetic Image Guidance,” IEEE Trans Biomed Eng, vol. 64, no. 8, pp. 1972- 1979, Aug. 2017, doi: 10.1109 / TBME.2016.2623383, which is incorporated by reference herein in its entirety) proposed such a method where Jaeger, et al. incorporated a custom tendon-driven catheter design with Electro-magnetic (EM) sensors controlled with an electromechanical drive train. However, the system needed heavy user interaction as the clinician uses a computer interfaced joystick to manipulate the catheter. A semi-automatic navigation of the biopsy needle during bronchoscopy was proposed by Kuntz, et al. (as discussed in A. Kuntz et al., “Autonomous Medical Needle Steering In Vivo,” Nov. 2022, Accessed: Jun. 28, 2023. [Online]. Available: https: / / arxiv.org / abs / 2211.02597vi, which is incorporated by reference herein in its entirety). The method uses pre-operative CT scans (3D) and EM sensors to co-register cloud points of the nodule and live guidance.Nevertheless, the method cannot be used to navigate the bronchoscopic catheter intothe airways, which is a critical step for reaching the nodules. Moreover, similar methods (as discussed in S. Chen, Y. Lin, Z. Li, F. Wang, and Q. Cao, “Automatic and accurate needle detection in 2D ultrasound during robot-assisted needle insertion process,” Int J Comput Assist Radiol Surg, vol. 17, no. 2, pp. 295-303, Feb. 2022, doi: 10.1007 / S11548-021-02519-6 / FIGURES / 8, which is incorporated by reference herein in its entirety) allow automatic localization of the needle in real-time without the need of an automatic needle navigation.

[0012] As such, there is a need for devices, systems, methods, and / or storage mediums that provide the feature(s) or details on how the other bending sections or portions of such imaging devices, imaging systems, etc. (e.g., endoscopic devices, bronchoscopes, other types of imaging devices / systems, etc.) may move to best assist navigation and / or state or state(s) for same, to keep track of a path of a tip of the imaging devices, imaging systems, etc., and there is a need for a more appropriate navigation of a device (such as, but not limited to, a bronchoscopic catheter being navigated to reach a nodule).

[0013] Accordingly, it would be desirable to provide at least one imaging, optical, or control device, system, method, and storage medium for controlling one or more endoscopic or imaging devices or systems, for example, by implementing automatic (e.g., robotic) or manual control of each portion or section of the at least one imaging, optical, or control device, system, method, and storage medium to keep track of and to match the state or state(s) of a first portion or section in a case where each portion or section reaches or approaches a same or similar, or approximately same or similar, state or state(s) and to provide a more appropriate navigation of a device (such as, but not limited to, a bronchoscopic catheter being navigated to reach a nodule).

[0014] To solve the problems discussed above, an autonomous navigation robot including 1) perception step, 2) planning step, 3) control step is described. For the planning step, a method and system is provided user-interface for user to instruct commands, and reflects the commands to a plan. Since problems also existin the automatic navigation of the steerable catheter since it can be counterintuitive fore users to instruct the system for the intended autonomously navigated route and to make any changes or modifications within the autonomous navigation.

[0015] U.S. Pat. Pub 2022 / 0160433 provides an automatic tool presence and workflow recognition, and states that the UI data can include a broad range of inputs made by the user and captured by the input devices, including buttons, menus, gestures, and voice commands. However, this simply provides voice input and does not enhance the autonomous workflow or overcome the issue of inappropriate user input.

[0016] For the control step, combining the information from the perception step and the planning step and defining the criteria to decide when to move forward into the lumen and when to continue to bend or optimize tip direction for future movement is described.SUMMARY

[0017] Accordingly, it is a broad object of the present disclosure to teach an autonomous navigation robot system having a steerable catheter; one or more actuators to steer and drive the steerable catheter; a user input device, a display; and a processor with one or more memory storing instructions configured to execute the stored instructions through the actuators, wherein the stored instructions include at least two driving modes for the robot system, including an autonomous driving mode and a manual driving mode, wherein the at least two driving modes are selected based on a force applied to the actuator of the robotic system during steering and driving of the robotic system through a certain path, or wherein the at least two driving modes are selected based on a time spent to complete steering and driving of the robotic system through a certain path.

[0018] Additional embodiments include the driving mode of the robotic system being changed from the autonomous driving mode to the manual drivingmode in a case where a time spent to complete steering and driving of the robotic system through a certain path is more than a predetermined time period.

[0019] In further embodiments, the driving mode of the robotic system is changed from the autonomous driving mode to manual driving mode in a case where the detected force during steering and of the robotic system driving through a certain path is more than a predetermined force.

[0020] In yet another contemplated embodiment, the manual mode includes at least one of a stop mode, a manual driving mode or another autonomous driving mode.

[0021] Furthermore, the robotic system is inserted into an object by an advance of a stage attached to the robotic system, and wherein the certain path is a path that the robotic system passes in accordance with an advance of the stage for a predetermined distance.

[0022] In other embodiments, the certain path is determined based on a position of at least three consecutive branching points. Furthermore, a start point of the certain path is between a first branching point and a second branching point and an end point of the certain path is between the second branching point and the third branching point. Alternatively, a start point of the certain path is a midpoint between a first branching point and a second branching point and an end point of the certain path is a midpoint between the second branching point and the third branching point.

[0023] In yet another derivative embodiment, the driving mode of the robotic tool is changed from the autonomous driving mode to a manual driving mode in a case where the time spent to complete bending of the robotic tool during the driving through the certain path is more than the predetermined period.

[0024] It is further contemplated that the driving mode of the robotic tool is changed from a first autonomous driving mode to a second autonomous driving mode which is different from the first autonomous driving mode in a case where thetime spent to complete bending of the robotic tool during the driving through the certain path is more than the predetermined period.

[0025] In additional embodiments, the predetermined period is equal to or more than 2.3 seconds and equal to or less than 6.03 seconds.

[0026] In yet another contemplated embodiment, the driving mode of the robotic tool is changed from the autonomous driving mode to the other driving mode in a case where the force or the time exceeds a threshold, wherein the threshold is defined as: (75th percentile)+i.5* interquartile range of a box-and-whisker plot for a dataset representing the force applied to the driving wire of the robotic tool during the driving through the certain path or the time spent to complete bending of the robotic tool during the driving through the certain path.

[0027] In another contemplated embodiment, the driving mode of the robotic tool is changed from the autonomous driving mode to the other driving mode in a case where the force or the time exceeds a threshold, wherein the threshold is defined as: the slope of regression line + 3*average of all measured data points of a dataset representing the force applied to the driving wire of the robotic tool during the driving through the certain path or the time spent to complete bending of the robotic tool during the driving through the certain path.

[0028] The subject innovation may also comprise a storage that stores the force applied to a driving wire of the robotic tool during a driving through a certain path or the time spent to complete bending of the robotic tool during the driving through the certain path; wherein the driving mode of the robotic tool is changed from the autonomous driving mode to the other driving mode in a case where the force or the time exceeds a threshold, and wherein the threshold is updated based on the stored force or the stored time.

[0029] In yet another embodiment, the one or more processors further perform: obtaining, based on the at least one model, an index indicating a difficulty of an operation of a robotic tool in the object or a probability of success of an operation of the robotic tool in the object; outputting, based on the index,information about a selection of one of tools, the tools including a manual tool steered manually or the robotic tool steered robotically.

[0030] It is further contemplated that the one or more processors further configured to execute the stored instructions to perform: obtaining an index indicating a difficulty of an operation of a robotic tool in the object or a probability of success of an operation of the robotic tool in the object for at least a part of the path of the robotic tool; and determining, based on the index corresponding to a part of the path, a steering mode of the robotic tool for the path, from among steering modes including a manual steering mode and one or more autonomous steering mode. It is also contemplated that the one or more processors further configured to execute the stored instructions to perform: switching, based on the index corresponding to a part of the path, a steering mode of the robotic tool for the path, from a first autonomous mode to a second autonomous mode which is different from the first autonomous mode.

[0031] In additional embodiments, the robotic tool is a robotic catheter.

[0032] The subject innovation also teaches a non-transitory computer readable storage medium storing instructions executed by one or more processors to perform: obtaining at least one model of an object; controlling a robotic system to move through the object based on the at least one model of the object; wherein a driving mode of the robotic system is changed from an autonomous driving mode to a manual driving mode based on a force applied to a driving wire of the robotic system during a driving through a certain path or a time spent to complete bending of the robotic system during the driving through the certain path.

[0033] It is also contemplated that the one or more memories storing instructions; and one or more processors configured to execute the stored instructions to perform: obtaining at least one model of an object; obtaining, based on the at least one model, an index indicating a difficulty of an operation of a robotic system in the object or a probability of success of an operation of the robotic system in the object; outputting, based on the index, information about a selection of one oftools, the tools including a manual tool steered manually or the robotic tool steered robotically.

[0034] Further features of the present disclosure will become apparent from the following description of exemplary embodiments with reference to the attached drawings, where like structure is indicated with like reference numerals.BRIEF DESCRIPTION OF THE DRAWINGS

[0035] For the purposes of illustrating various aspects of the disclosure, wherein like numerals indicate like elements, there are shown in the drawings simplified forms that maybe employed, it being understood, however, that the disclosure is not limited by or to the precise arrangements and instrumentalities shown. To assist those of ordinary skill in the relevant art in making and using the subject matter hereof, reference is made to the appended drawings and figures.

[0036] FIG. 1 illustrates at least one embodiment of an imaging, continuum robot, or endoscopic apparatus or system in accordance with one or more aspects of the present disclosure.

[0037] FIG. 2 is a schematic diagram showing at least one embodiment of an imaging, steerable catheter, or continuum robot apparatus or system in accordance with one or more aspects of the present disclosure;

[0038] FIG. 3A illustrate at least one embodiment example of a continuum robot and / or medical device that may be used with one or more technique(s), including autonomous navigation technique(s), in accordance with one or more aspects of the present disclosure. Detail A illustrates one guide ring of the steerable catheter.

[0039] FIGS. 3B-3C illustrate one or more principles of catheter or continuum robot tip manipulation by actuating one or more bending segments of a continuum robot or steerable catheter 104 of FIGS. 3A-3B in accordance with one or more aspects of the present disclosure.

[0040] FIG. 4 is a schematic diagram showing at least one embodiment of an imaging, continuum robot, steerable catheter, or endoscopic apparatus or system in accordance with one or more aspects of the present disclosure.

[0041] FIG. 5 is a schematic diagram showing at least one embodiment of a console or computer that may be used with one or more autonomous navigation technique(s) in accordance with one or more aspects of the present disclosure.

[0042] FIG. 6 is a flowchart of at least one embodiment of a method for planning an operation of at least one embodiment of a continuum robot or steerable catheter apparatus or system in accordance with one or more aspects of the present disclosure.

[0043] FIG. 7 is a flowchart of at least one embodiment of a method for performing autonomous navigation, movement detection, and / or control for a continuum robot or steerable catheter in accordance with one or more aspects of the present disclosure.

[0044] FIG. 8(a) shows images of at least one embodiment of an application example of autonomous navigation technique(s) and movement detection for a camera view (left), a depth map (center), and a thresholded image (right) in accordance with one or more aspects of the present disclosure.

[0045] FIG. 8(b) shows images of one embodiment showing a camera view (left), a semi-transparent color coded depth map overlaid onto a camera view (center) and a thresholded image (right).

[0046] FIG. 9 is an exemplary image having two airways and indicators of circle fit and target path in accordance with one or more aspects of the present disclosure.

[0047] FIG. 10 is an exemplary image having two airways and indicators of circle fit and target path in accordance with one or more aspects of the present disclosure.

[0048] FIG. 11 is a diagram showing two lumens and the threshold.[00491 FIG. 12 is a flowchart of at least one embodiment of a method for controlling the steerable catheter in accordance with one or more aspects of the present disclosure.

[0050] FIG. 13 is a flowchart of at least one embodiment of a method for controlling the steerable catheter, including speed setting, in accordance with one or more aspects of the present disclosure.

[0051] FIG. 14 is a diagram indicating bending speed and moving speed in accordance with one or more aspects of the present disclosure.

[0052] FIG. 15 is a flowchart of at least one embodiment of a method for controlling the steerable catheter including setting the threshold, in accordance with one or more aspects of the present disclosure.

[0053] FIG. 16 is a diagram of an airway with an indication of two thresholds, in accordance with one or more aspects of the present disclosure.

[0054] FIG. 17 is a flowchart of at least one embodiment of a method for controlling the steerable catheter, including blood detection, in accordance with one or more aspects of the present disclosure.

[0055] FIG. 18 provides a graph detailing the results of autonomous vs. human advancement of the steerable catheter, in accordance with one or more aspects of the present disclosure.

[0056] FIG. 19 depicts a shows a logistic regression curve used to predict the probability of the successful robotic bronchoscopy with autonomous navigation of the steerable catheter, in accordance with one or more aspects of the present disclosure.

[0057] FIG. 20 shows an image of a model lung, with bifurcation points, in accordance with one or more aspects of the present disclosure.

[0058] FIG. 21 is a graph comparing the median times for a bending command in a human operator vs. autonomous navigation, in accordance with one or more aspects of the present disclosure.

[0059] FIG. 22 is also a graph detailing medians of the maximum force at a bifurcation point by the human operators vs. autonomous navigation, in accordance with one or more aspects of the present disclosure.

[0060] FIG. 23 shows us the dependency of the time and the force on the airway generation of the lung comparing a human operator vs. autonomous navigation, in accordance with one or more aspects of the present disclosure.

[0061] FIG. 24 provides the dependency of the time and the force on the airway generation of the lung comparing a human operator vs. autonomous navigation, with regression lines and 95% confidential intervals, in accordance with one or more aspects of the present disclosure.

[0062] FIG. 25 depicts a flowchart for overall workflow, in accordance with one or more aspects of the present disclosure.

[0063] FIG. 26 is a graph detailing threshold for time to be set, in accordance with one or more aspects of the present disclosure.

[0064] FIG. 27 is a graph showing a threshold to be set for maximum force, in accordance with one or more aspects of the present disclosure.

[0065] FIG. 28 is an exemplary graph for time threshold being set, in accordance with one or more aspects of the present disclosure.

[0066] FIG. 29 is a graph depicting the shifted slope for a force threshold set, in accordance with one or more aspects of the present disclosure.

[0067] FIG. 30 depicts an image of a model lung with a color-coded difficulty index shown, in accordance with one or more aspects of the present disclosure.

[0068] FIG. 31 provides two camera views within a lung, at inhalation and exhalation phase, in accordance with one or more aspects of the present disclosure.

[0069] FIG. 32 is a chart showing specific data corresponding to the graph in FIG. 21 and FIG. 23, in accordance with one or more aspects of the present disclosure.DETAILED DESCRIPTION

[0070] Various exemplary embodiments, features, and aspects of the disclosure will be described below with reference to the drawings.< Rob otic Catheter System >

[0071] An embodiment of a robotic catheter system 100 is described in reference to FIG. 1 through FIG. 4. FIG. 1 illustrates a simplified representation of a medical environment, such as an operating room, where a robotic catheter system too can be used. FIG. 2 illustrates a functional block diagram of the robotic catheter system 100. FIGS. 3A-C represents the catheter and bending. FIGS. 4 - 5 illustrates a logical block diagram of the robotic catheter system 100. In this example, the system 100 includes a system console 102 (computer cart) operatively connected to a steerable catheter 104 via a robotic platform 106. The robotic platform 106 includes one or more than one robotic arm 108 and a linear translation stage 110.

[0072] In FIG. 1, A user 112 (e.g., a physician) controls the robotic catheter system 100 via a user interface unit (operation unit) to perform an intraluminal procedure on a patient 114 positioned on an operating table 116. The user interface may include at least one of a main display 118 (a first user interface unit), a secondary display 120 (a second user interface unit), and a handheld controller 124 (a third user interface unit). The main display 118 may include, for example, a large display screen attached to the system console 102 or mounted on a wall of the operating room and may be, for example, designed as part of the robotic catheter system 100 or be part of the operating room equipment. Optionally, there is a secondary display 120 that is a compact (portable) display device configured to be removably attached to the robotic platform 106. Examples of the secondary display 120 include a portable tablet computer or a mobile communication device (a cellphone).

[0073] The steerable catheter 104 is actuated via an actuator unit 122. The actuator unit 103 is removably attached to the linear translation stage 110 of the robotic platform 106. The handheld controller 124 may include a gamepad-like controller with a joystick having shift levers and / or push buttons. It may be a one-handed controller or a two-handed controller. In one embodiment, the actuator unit 122 is enclosed in a housing having a shape of a catheter handle. One or more access ports 126 are provided in or around the catheter handle. The access port 126 is used for inserting and / or withdrawing end effector tools and / or fluids when performing an interventional procedure of the patient 114.

[0074] The system console 102 includes a system controller 128, a display controller 130, and the main display 118. The main display 118 may include a conventional display device such as a liquid crystal display (LCD), an OLED display, a QLED display or the like. The main display 118 provides a graphic interface unit (GUI) configured to display one or more views. These views include live view image 132, an intraoperative image 134, and a preoperative image 136, and other procedural information 138. Other views that may be displayed include a model view, a navigational information view, and / or a composite view. The live image view 132 may be an image from a camera at the tip of the catheter. This view may also include, for example, information about the perception and navigation of the catheter 104. The preoperative image 136 may include pre-acquired 3D or 2D medical images of the patient acquired by conventional imaging modalities such as computer tomography (CT), magnetic resonance imaging (MRI), or ultrasound imaging. The intraoperative image 134 may include images used for image guided procedure such images may be acquired by fluoroscopy or CT imaging modalities. Intraoperative image 134 may be augmented, combined, or correlated with information obtained from a sensor, camera image, or catheter data.

[0075] In the various embodiments where a catheter tip tracking sensor 140 is used, the sensor may be located at the distal end of the catheter. The catheter tip tracking sensor 140 may be, for example, an electromagnetic (EM) sensor. If an EM sensor is used, a catheter tip position detector 142 is included in the robotic catheter system 100; this catheter tip position detector would include an EM field generator operatively connected to the system controller 128. Suitable electromagnetic sensors for use with a steerable catheter are well-known and described, for example, in U.S.Pat. No.: 6,201,387 and international publication W02020194212A1.

[0076] Similar to FIG. 1, the diagram of FIG. 2 illustrates the robotic catheter system 100 includes the system controller 128 operatively connected to the display controller 130, which is connected to the display unit 118, and to the hand held control 124. The system controller 128 is also connected to the actuator unit 122 via the robotic platform 106, which includes the linear translation stage 110. The actuator unit 122 includes a plurality of motors 144 that control the plurality of drive wires 160. These drive wires travel through the steerable catheter 104. One or more access ports 126 may be located on the catheter. The catheter includes a proximal section 148 located between the actuator and the proximal bending section 152 where they actuate the proximal bending section. Three of the six drive wires 160 continue through the distal bending section 156 where they actuate this section and allow for a range of movement. This figure is shown with two bendable sections (152 and 156). Other embodiments as described herein can have three bendable sections (see FIG. 3). In some embodiments, a single bending section may be provided, or alternatively, four or more bendable sections may be present in the catheter.

[0077] FIG. 3A shows an exemplary embodiment of a steerable catheter 104. The steerable catheter 104 includes a non-steerable proximal section 148, a steerable distal section 150, and a catheter tip 158. The proximal section 148 and distal bendable section 150 (including 152, 154 and 156) are joined to each other by a plurality of drive wires 160 arranged along the wall of the catheter. The proximal section 148 is configured with thru-holes or grooves or conduits to pass drive wires 160 from the distal section 150 to the actuator unit 122. The distal section 150 is comprised of a plurality of bending segments including at least a distal segment 156, a middle segment 154, and a proximal segment 152. Each bending segment is bent by actuation of at least some of the plurality of drive wires 160 (driving members). The posture of the catheter may be supported by non-illustrated supporting wires (support members) also arranged along the wall of the catheter (see U.S. Pat. Pub. US2021 / 0308423). The proximal ends of drive wires 160 are connected toindividual actuators or motors 144 of the actuator unit 122, while the distal ends of the drive wires 160 are selectively anchored to anchor members in the different bending segments of the distal bendable section 150.

[0078] Each bending segment is formed by a plurality of ring-shaped components (rings) with thru-holes, grooves, or conduits along the wall of the rings. The ring-shaped components are defined as wire-guiding members 162 or anchor members 164 depending on their function within the catheter. Anchor members 164 are ring-shaped components onto which the distal end of one or more drive wires 160 are attached. Wire-guiding members 162 are ring-shaped components through which some drive wires 160 slide through (without being attached thereto).

[0079] Detail “A” in FIG. 3A illustrates an exemplary embodiment of a ringshaped component (a wire-guiding member 162 or an anchor member 164). Each ring-shaped component includes a central opening which forms the tool channel 168, and plural conduits 166 (grooves, sub-channels, or thru-holes) arranged lengthwise equidistant from the central opening along the annular wall of each ring-shaped component. Inside the ring-shaped component, an inner cover such as is described in U.S. Pat. Pub US2021 / 0369085 and US2022 / 0126060, maybe included to provide a smooth inner channel and provide protection. The non-steerable proximal section 148 is a flexible tubular shaft and can be made of extruded polymer material. The tubular shaft of the proximal section 148 also has a central opening or tool channel 168 and plural conduits 166 along the wall of the shaft surrounding the tool channel 168. An outer sheath may cover the tubular shaft and the steerable section 150. In this manner, at least one tool channel 168 formed inside the steerable catheter 104 provides passage for an imaging device and / or end effector tools from the insertion port 126 to the distal end of the steerable catheter 104.

[0080] The actuator unit 122 includes one or more servo motors or piezoelectric actuators. The actuator unit 122 bends one or more of the bending segments of the catheter by applying a pushing and / or pulling force to the drive wires 160. As shown in FIG. 3A, each of the three bendable segments of the steerablecatheter 104 has a plurality of drive wires 160. If each bendable segment is actuated by three drive wires 160, the steerable catheter 104 has nine driving wires arranged along the wall of the catheter. Each bendable segment of the catheter is bent by the actuator unit 122 by pushing or pulling at least one of these nine drive wires 160. Force is applied to each individual drive wire in order to manipulate / steer the catheter to a desired pose. The actuator unit 122 assembled with steerable catheter 104 is mounted on the linear translation stage 110. Linear translation stage 110 includes a slider and a linear motor. In other words, the linear translation stage 110 is motorized, and can be controlled by the system controller 128 to insert and remove the steerable catheter 104 to / from the patient’s bodily lumen.

[0081] An imaging device 170 that can be inserted through the tool channel 168 includes an endoscope camera (videoscope) along with illumination optics (e.g., optical fibers or LEDs). The illumination optics provides light to irradiate the lumen and / or a lesion target which is a region of interest within the patient. End effector tools refer endoscopic surgical tools including clamps, graspers, scissors, staplers, ablation or biopsy needles, and other similar tools, which serve to manipulate body parts (organs or tumorous tissue) during examination or surgery. The imaging device 170 maybe what is commonly known as a chip-on-tip camera and maybe color or black-and-white.

[0082] In some embodiments, a tracking sensor 140 (e.g., an EM tracking sensor) is attached to the catheter tip 158. In this embodiment, steerable catheter 104 and the tracking sensor 140 can be tracked by the tip position detector 142. Specifically, the tip position detector 142 detects a position of the tracking sensor 140, and outputs the detected positional information to the system controller 100. The system controller 128, receives the positional information from the tip position detector 142, and continuously records and displays the position of the steerable catheter 104 with respect to the patient’s coordinate system. The system controller 128 controls the actuator unit 122 and the linear translation stage 110 in accordance with the manipulation commands input by the user 112 via one or more of the userinterface units (the handheld controller 124, a GUI at the main display 118 or touchscreen buttons at the secondary display 120).

[0083] FIG. 3B and FIG. 3C show exemplary catheter tip manipulations by actuating one or more bending segments of the steerable catheter 104. As illustrated in FIG. 3B, manipulating only the most distal segment 156 of the steerable section changes the position and orientation of the catheter tip 158. On the other hand, manipulating one or more bending segments (152 or 154) other than the most distal segment affects only the position of catheter tip 158, but does not affect the orientation of the catheter tip. In FIG. 3B, actuation of distal segment 155 changes the catheter tip from a position Pi having orientation 01, to a position P2 having orientation O2, to position P3 having orientation O3, to position P4 having orientation O4, etc. In FIG. 3C, actuation of the middle segment 154 changes the position of catheter tip 158 from a position Pi having orientation 01 to a position P2 and position P3 having the same orientation 01. Here, it should be appreciated by those skilled in the art that exemplary catheter tip manipulations shown in FIG. 3B and FIG. 3C can be performed during catheter navigation (i.e., while inserting the catheter through tortuous anatomies). In the present disclosure, the exemplary catheter tip manipulations shown in FIG. 3B and FIG. 3C apply namely to the targeting mode applied after the catheter tip has been navigated to a predetermined distance (a targeting distance) from the target.

[0084] FIG. 4 illustrates the system controller 128 executes software programs and controls the display controller 130 to display a navigation screen (e.g., a live view image 132) on the main display 118 and / or the secondary display 120. The display controller 130 may include a graphics processing unit (GPU) or a video display controller (VDC).

[0085] FIG. 5 illustrates components of the system controller 128 and / or the display controller 130. The system controller 128 and the display controller 130 can be configured separately. Alternatively, the system controller 128 and the display controller 102 can be configured as one device. In either case, the system controller128 and the display controller 130 comprise substantially the same components. Specifically, the system controller 128 and display controller 130 may include a central processing unit (CPU 182) comprised of one or more processors (microprocessors), a random access memory (RAM 184) module, an input / output (I / O 186) interface, a read only memory (ROM 180), and data storage memory (e.g., a hard disk drive (HDD 188) or solid state drive (SSD)).

[0086] The ROM 180 and / or HDD 188 store the operating system (OS) software, and software programs necessary for executing the functions of the robotic catheter system 100 as a whole. The RAM 184 is used as a workspace memory. The CPU 182 executes the software programs developed in the RAM 184. The I / O 186 inputs, for example, positional information to the display controller 130, and outputs information for displaying the navigation screen to the one or more displays (main display 118 and / or secondary display 120). In the embodiments descried below, the navigation screen is a graphical user interface (GUI) generated by a software program but, it may also be generated by firmware, or a combination of software and firmware.

[0087] The system controller 128 may control the steerable catheter 104 based on any known kinematic algorithms applicable to continuum or snake-like catheter robots. For example, the system controller controls the steerable catheter 104 based on an algorithm known as follow the leader (FTL) algorithm. By applying the FTL algorithm, the most distal segment 156 of the steerable section 150 is actively controlled with forward kinematic values, while the middle segment 154 and the proximal segment 152 (following sections) of the steerable catheter 104 move at a first position in the same way as the distal section moved at the first position or a second position near the first position.

[0088] The display controller 130 acquires position information of the steerable catheter 104 from system controller 102. Alternatively, the display controller 130 may acquire the position information directly from the tip position detector 142. The steerable catheter 104 may be a single-use or limited-use catheterdevice. In other words, the steerable catheter 104 can be attachable to, and detachable from, the actuator unit 122 to be disposable.

[0089] During a procedure, the display controller 130 can generate and outputs a live-view image or other view(s) or a navigation screen to the main display 118 and / or the secondary display 120. This view can optionally be registered with a 3D model of a patient’s anatomy (a branching structure) and the position information of at least a portion of the catheter (e.g., position of the catheter tip 158) by executing pre-programmed software routines. Upon completing navigation to a desired target, one or more end effector tools can be inserted through the access port 126 at the proximal end of the catheter, and such tools can be guided through the tool channel 168 of the catheter body to perform an intraluminal procedure from the distal end of the catheter.

[0090] The tool may be a medical tool such as an endoscope camera, forceps, a needle or other biopsy or ablation tools. In one embodiment, the tool may be described as an operation tool or working tool. The working tool is inserted or removed through the working tool access port 126. In the embodiments below, an embodiment of using a steerable catheter to guide a tool to a target is explained. The tool may include an endoscope camera or an end effector tool, which can be guided through a steerable catheter under the same principles. In a procedure there is usually a planning procedure, a registration procedure, a targeting procedure, and an operation procedure.<Autonomous navigation fimction>

[0091] System controller 128 includes an autonomous navigation mode. During the autonomous navigation mode, the user does not need to control the bending and translational insertion position of steerable catheter 104. The autonomous navigation mode comprises 1) perception step, 2) planning step and 3) control step. In the perception step, system controller 128 receives endoscope view and analyses the endoscope view to find addressable airways from the currentposition / orientation of steerable catheter 104. At end of this analysis, the system controller 128 percepts these addressable airways as paths in the endoscope view.

[0092] The planning step is a step to determine a target path, which is the destination for the steerable catheter 104. While there are a couple of different approaches to select one of the paths as the target path, this invention uniquely include means to reflect user instructions concurrently for the decision of target path among the precepted paths. Once the system determines the target paths with this concurrent user instructions, the target path is sent to the next step, a control step.

[0093] The control step is a step to control the steerable catheter 104 and linear translation stage no to navigate the steerable catheter 104 to the target path. This step is also an automatic step. The system controller 128 uses an information relating to the real time endoscope view, the target path and an internal design & status information on the robotic catheter system 100.

[0094] Through these three steps, the robotic catheter system too can navigate steerable catheter 104 autonomously by reflecting the user’s intention efficiently.

[0095] FIG. 8(a) signifies one of the design examples of this invention. The real-time endoscope view 800 are displayed in main display 118 (as a user output device) in system console 102 (800). The user can see the airways in the real-time endoscope view 1 through main display 118. This real-time endoscope view 800 is also sent to system controller 128. In the perception step, system controller 128 processes real-time endoscope view 800 and identifies path candidates by using image processing algorithms. Among these path candidates, the system controller 128 select the paths 2 with the designed computation processes, then displays the paths 2 with a circle with the real-time endoscope view 800.

[0096] In planning step, the system controller 128 provides for interaction from the user, such as a cursor, so that the user can indicate the target path by moving the cursor with joystick 124. When the cursor locates within the area of the path to be selected (one of the two circles), the system controller 128 recognizes the path with the cursor as the target path (FIG. 8(a)).

[0097] In further design example, the system controller 128 can pause the motion of the actuator unit 122 and linear translation stage 110 during the user is moving the cursor 3 so that the user can select the target path with the minimal change of the real-time endoscope view 1 and paths 2 since the system does not move.< Perception >

[0098] One or more of the features discussed herein may be used for perception and planning procedures, including using one or more models for artificial intelligence applications. As an example of one or more embodiments, FIG. 6 is a flowchart showing steps of at least one planning procedure of an operation of the continuum robot / catheter device 104. One or more of the processors discussed herein may execute the steps shown in FIG. 6, and these steps may be performed by executing a software program read from a storage medium, including, but not limited to, the ROMno or HDD 150, by CPU 120 or by any other processor discussed herein. One or more methods of planning using the continuum robot / catheter device 104 may include one or more of the following steps: (i) In step s6oi, one or more images such, as CT or MRI images, may be acquired; (ii) In step S602, a three dimensional model of a branching structure (for example, an airway model of lungs or a model of an object, specimen or other portion of a body) may be generated based on the acquired one or more images; (iii) In step S603, a target on the branching structure may be determined (e.g., based on a user instruction, based on preset or stored information, etc.); (iv) In step S604, a route of the continuum robot / catheter device 104 to reach the target (e.g., on the branching structure) maybe determined (e.g., based on a user instruction, based on preset or stored information, based on a combination of user instruction and stored or preset information, etc.); (v) In step S605, the generated model (e.g., the generated two-dimensional or threedimensional model) and the decided route on the model may be stored (e.g., in theRAM 130 or HDD or data storage 150, in any other storage medium discussed herein,in any other storage medium known to those skilled in the art, etc.). In this way, a model (e.g., a 2D or 3D model) of a branching structure may be generated, and a target and a route on the model maybe determined and stored before the operation of the continuum robot 104 is started.

[0099] In one or more of the embodiments below, embodiments of using a catheter device / continuum robot 104 are explained, such as, but not limited to features for performing autonomous navigation, movement detection, and / or control technique(s).

[0100] In one or more embodiments, the system controller 102 (or any other controller, processor, computer, etc. discussed herein) may operate to perform an autonomous navigation mode. During the autonomous navigation mode, the user does not need to control the bending and translational insertion position of the steerable catheter 104. The autonomous navigation mode may include or comprise: (1) a perception step, (2) a planning step, and (3) a control step. In the perception step, the system controller 102 may receive an endoscope view (or imaging data) and may analyze the endoscope view (or imaging data) to find addressable airways from the current position / orientation of the steerable catheter 104. At an end of this analysis, the system controller 102 identifies or perceives these addressable airways as paths in the endoscope view (or imaging data).

[0101] The planning step is a step to determine a target path, which is the destination for the steerable catheter 104. While there are a couple of different approaches to select one of the paths as the target path, the present disclosure uniquely includes means to reflect user instructions concurrently for the decision of a target path among the identified or perceived paths. Once the system 1000 determines the target paths while considering concurrent user instructions, the target path is sent to the next step, i.e., the control step.

[0102] The control step is a step to control the steerable catheter 104 and the linear translation stage 122 (or any other portion of the robotic platform 108) to navigate the steerable catheter 104 to the target path, pose, state, etc. This step mayalso be performed as an automatic step. The system controller 102 operates to use information relating to the real time endoscope view (e.g., the view 134), the target path, and an internal design & status information on the robotic catheter system1000.

[0103] Through these three steps, the robotic catheter system 1000 may navigate the steerable catheter 104 autonomously, which achieves reflecting the user’s intention efficiently.

[0104] As shown in FIG. 1, the real-time endoscope view 134 may be displayed in a main display 101-1 (as a user input / output device) in the system 1000. The user may see the airways in the real-time endoscope view 134 through the main display 101-1. This real-time endoscope view 134 may also be sent to the system controller 102. In the perception step, the system controller 102 may process the real-time endoscope view 134 and may identify path candidates by using image processing algorithms. Among these path candidates, the system controller 102 may select the paths with the designed computation processes, and then may display the paths with a circle, octagon, or other geometric shape with the real-time endoscope view 134 as discussed further below for FIGS. 7-8.

[0105] In planning step, the system controller 102 may provide a cursor so that the user may indicate the target path by moving the cursor with the joystick 105. When the cursor is disposed or is located within the area of the path, the system controller 102 operates to recognize the path with the cursor as the target path.

[0106] In a further embodiment example, the system controller 102 may can pause the motion of the actuator unit 103 and the linear translation stage 122 while the user is moving the cursor so that the user may select the target path with a minimal change of the real-time endoscope view 134 and paths since the system 1000 would not move in such a scenario. Additionally or alternatively, the features of the present disclosure maybe performed using artificial intelligence, including the autonomous driving mode. For example, deep learning may be used for performing autonomous driving using deep learning for localization. Any features of the presentdisclosure may be used with artificial intelligence features discussed in J. Sganga, D. Eng, C. Graetzel, and D. B. Camarillo, “Autonomous Driving in the Lung using Deep Learning for Localization,” Jul. 2019, Accessed: Jun. 28, 2023. [Online]. Available: https: / / arxiv.0rg / abs / 1907.08136vi, the disclosure of which is incorporated by reference herein in its entirety.

[0107] In one or more embodiments, the system controller 102 (or any other controller, processor, computer, etc. discussed herein) may operate to perform a depth map mode. A depth map may be generated or obtained from one or more images (e.g., bronchoscopic images, CT images, images of another imaging modality, etc.). A depth of each image may be identified or evaluated to generate the depth map or maps. The generated depth map or maps maybe used to perform autonomous navigation, movement detection, and / or control of a continuum robot, a steerable catheter, an imaging device or system, etc. as discussed herein. In one or more embodiments, thresholding maybe applied to the generated depth map or maps, or to the depth map mode, to evaluate accuracy for navigation purposes. For example, while not limited to only this type of a threshold, a threshold may be set for an acceptable distance between the ground truth (and / or a target camera location, a predetermined camera location, an actual camera location, etc.) and an estimated camera location for a catheter or continuum robot (e.g., the catheter or continuum robot 104). By way of a further example, the threshold may defined such that the distance between the ground truth (and / or a target camera location, a predetermined camera location, an actual camera location, etc.) and an estimated camera location is equal to or less than, or less than, a set or predetermined distance of one or more of the following: 5 mm, 10 mm, about 5 mm, about 10 mm, any other distance set by a user of the device (depending on a particular application). In one or more embodiments, the predetermined distance may be less than 5 mm or less than about 5 mm. Any other type of thresholding may be applied to the depth mapping to improve and / or confirm the accuracy of the depth map(s).

[0108] Additionally or alternatively, thresholding may be applied to segmentthe one or more images to help identify or find one or more objects and to ultimately help define one or more targets used for the autonomous navigation, movement detection, and / or control features of the present disclosure. For example, a depth map or maps may be created or generated using one or more images (e.g., CT images, bronchoscopic images, images of another imaging modality, vessel images, etc.), and then, by applying a threshold to the depth map, the objects in the one or more images may be segmented (e.g., a lung may be segmented, one or more airways may be segmented, etc.). In one or more embodiments, the segmented portions of the one or more images (e.g., the one or more segmented airways, the segmented portions of a lung, etc.) may define one or more navigation targets for a next automatic robotic movement, navigation, and / or control. Examples of segmented airways are discussed further below with respect to FIG. 8(a). In one or more embodiments, one or more of the automated methods that may be used to apply thresholding may include one or more of the following: a watershed method (such as, but not limited to, watershed method(s) discussed in L. J. Belaid and W. Mourou, “IMAGE SEGMENTATION: A WATERSHED TRANSFORMATION ALGORITHM,” 2011, vol. 28, no. 2, p. 10, 2011, doi: io.5566 / ias.v28.p93-iO2, which is incorporated by reference herein in its entirety), a k-means method (such as, but not limited to, k- means method(s) discussed in T. Kanungo, D. M. Mount, N. S. Netanyahu, C. D. Piatko, R. Silverman, and A. Y. Wu, “An efficient k-means clustering algorithm: Analysis and implementation,” IEEE Trans Pattern Anal Mach Intell, vol. 24, no. 7, pp. 881-892, 2002, doi: Doi io.iiO9 / Tpami.2OO2.1017616, which is incorporated by reference herein in its entirety), an automatic threshold method (such as, but not limited to, automatic threshold method(s) discussed in N. Otsu, “Threshold Selection Method from Gray-Level Histograms,” IEEE Trans Syst Man Cybern, vol. 9, no. 1, pp. 62-66, 1979, which is incorporated by reference herein in its entirety) using a sharp slope method (such as, but not limited to, sharp slope method(s) discussed in U.S. Pat. Pub. No. 2023 / 0115191 Ai, published on April 13, 2023, which is incorporated by reference herein in its entirety) and / or any combination of the subject methods.In one or more embodiments, peak detection may include any of the techniques discussed herein, including, but not limited to, the techniques discussed in at least “8 Peak detection,” Data Handling in Science and Technology, vol. 21, no. C, pp. 183- 190, Jan. 1998, doi: 10.1016 / 80922-3487(98)80027-0, which is incorporated by reference herein in its entirety.

[0109] In one or more embodiments, the depth map(s) may be obtained, and / or the quality of the obtained depth map(s) may be evaluated, using artificial intelligence structure, such as, but not limited, convolutional neural networks, generative adversarial networks (GANs), neural networks, any other Al structure or feature(s) discussed herein, any other Al network structure(s) known to those skilled in the art, etc. For example, a generator of a generative adversarial network may operate to generate an image(s) that is / are so similar to ground truth image(s) that a discriminator of the generative adversarial network is not able to distinguish between the generated image(s) and the ground truth image(s). The generative adversarial network may include one or more generators and one or more discriminators. Each generator of the generative adversarial network may operate to estimate depth of each image (e.g., a CT image, a bronchoscopic image, etc.), and each discriminator of the generative adversarial network may operate to determine whether the estimated depth of each image (e.g., a CT image, a bronchoscopic image, etc.) is estimated (or fake) or ground truth (or real). In one or more embodiments, an Al network, such as, but not limited to, a GAN or a consistent GAN (cGAN), may receive an image or images as an input and may obtain or create a depth map for each image or images. In one or more embodiments, an Al network may evaluate obtained one or more images (e.g., a CT image, a bronchoscopic image, etc.), one or more virtual images, and one or more ground truth depth maps to generate depth map(s) for the one or more images and / or evaluate the generated depth map(s). A Three Cycle-Consistent Generative Adversarial Network (scGAN) may be used to obtain the depth map(s) and / or evaluate the quality of the depth map(s), and an unsupervised learning method (designed and trained in an unsupervised procedure) may be employed onthe depth map(s) and the one or more images (e.g., a CT image or images, a bronchoscopic image or images, any other obtained image or images, etc.). Any feature or features of obtaining a depth map or performing a depth map mode of the present disclosure may be used with any of the depth map or depth estimation features as discussed in A. Banach, F. King, F. Masaki, H. Tsukada, and N. Hata, “Visually Navigated Bronchoscopy using three cycle-Consistent generative adversarial network for depth estimation,” Med Image Anal, vol. 73, p. 102164, Oct. 2021, doi: 10.1016 / J.MEDIA.2021.102164, the disclosure of which is incorporated by reference herein in its entirety.

[0110] In one or more embodiments, the system controller 102 (or any other controller, processor, computer, etc. discussed herein) may operate to perform a computation of one or more lumen (e.g., a lumen computation mode) and / or one or more of the following: a circle fit / blob process, a peak detection, and / or a deepest point analysis.

[0111] The problem of fitting a circle to a binary object is equivalent to the problem of fitting a circle to a set of points. In our case the set of points is the boundaiy points of the binary object. Given a set of points(xl,yl), (x2,y2), (x3,y3), . . (xn,yn) a circle (x — a)2+ (y — b)2= c2can be fit to the points by summing the squares of the distances from the points to the circle: SS(a, b, c) = Sfc=i(c - ^(xi - a)2+ (yi — b)2) . However, there are several other variations that can be applied as described in D. Umbach and K. N. Jones, "A few methods for fitting circles to data," in IEEE Transactions on Instrumentation and Measurement, vol. 52, no. 6, pp. 1881-1885, Dec. 2003, doi:10.1109 / TIM.2003.820472. Blob fitting can be achieved on the binary objects by calculating their circularity as AnArea / (perimeter)2and then defining the circle radius.

[0112] Peak detection is performed in a i-D signal and is defined as the extreme value of the signal. Similarly, 2-D image peak detection is defined as the highest value of the 2-D matrix. Herein, depth map is the 2-D matrix, and its peak isthe highest value of the depth math which actually correspond to the deepest point. However, since there might be more than one airway which are represented by different depth value concentrations along the depth map image, more than one peaks exist. The depth map produces an image which predicts the depth of the airways, therefore for each airway there is a concentration of non-zero pixels around a deepest point that the GANs predicted. By applying peak detection to all the nonzero concentrations of the 2-D depth map the peak of each concentration is detected; each peak corresponds to an airway.

[0113] One or more features discussed herein may be used for performing autonomous navigation, movement detection, and / or control technique(s) for a steerable catheter, continuum robot, imaging device or system, etc. as discussed herein. FIG. 7 is a flowchart showing steps of at least one procedure for performing autonomous navigation, movement detection, and / or control technique(s) for a continuum robot / catheter device (e.g., such as continuum robot / catheter device 104). One or more of the processors discussed herein, one or more Al networks discussed herein, and / or a combination thereof may execute the steps shown in FIG. 7, and these steps may be performed by executing a software program read from a storage medium, including, but not limited to, the ROM 110 or HDD 150, by CPU 120 or by any other processor discussed herein. While not limited thereto, one or more methods of performing autonomous navigation, movement detection, and / or control technique(s) for a catheter or probe of a continuum robot device or system may include one or more of the following steps: (i) in step S700, one or more images (e.g., one or more camera images, one or more CT images (or images of another imaging modality), one or more bronchoscopic images, etc.) are obtained; (ii) in step S701, a target detection method is selected (automatically or manually) (e.g., target detection (td) = 1 for the peak detection method or mode, td = 2 for the thresholding method or mode, td =3 for the deepest point method or mode, etc. - the target detection methods shown in FIG. 7 are illustrative, are not limited thereto, and may be exchanged or substitute or used along with any combination of detection methodsdiscussed in the present disclosure or known to those skilled in the art); (iii) in step S703, based on the td value, the method continues to perform the selected target detection method and proceeds to step S704 for the peak detection method or mode, to step S706 for the thresholding method or mode, or to step S711 for the deepest point method or mode; (iv) in a case where td = 1, the peak detection method or mode is performed in step S704, a target or targets are set to be the detected peak or peaks in step S705 and a counter (cn) is set to 2 (cn=2), and a number of targets is evaluated in step S710 such that, in a case where no targets are found (# targets = o) and the counter = 2, then td is set to a value of 3 and the process returns to the depth map step S702 and proceeds to step S711 for the deepest point method or mode; (v) in a case where td = 2, the thresholding method or mode is performed in step S706 to identify one or more objects, the counter is set to be equal to 1 (cn = 1), binarization is performed in step S707 to process the image data (e.g., the image data may be converted from color to black and white images, the image data may be split into data sets, etc.), fitting a circle in or on each object is performed in step S708 (also referred to as a blob fit or blob detection method), a target or targets is / are set to be at a predetermined or set location (e.g., a center) of the circle for each object of the one or more objects in step S709, a number of targets is evaluated in step S710 such that, in a case where no targets are found (# targets = o) and the counter = 1, then td is set to a value of 1 and the process returns to the depth map step S702 and proceeds to step S704 for the peak detection method or mode in step S704, a target or targets is / are identified as the detected peak or peaks in step S705, and a number of targets is evaluated in step S710; and (vi) in a case where the number of targets evaluated in step S710 is 1 or more, then the process proceeds to step S712 where the continuum robot or steerable catheter (or other imaging device or system) (e.g., the continuum robot or steerable catheter 104) is moved to the target or targets. In one or more embodiments, the steps S701 through S712 of FIG. 7 may be performed again for an obtained or received next image or images to evaluate the next movement, pose, position, orientation, or state for the autonomous navigation, movement detection,and / or control of the continuum robot or steerable catheter (or imaging device or system) 104. In step S702, the method may estimate (automatically or manually) the depth map or maps (e.g., a 2D or 3D depth map or maps) of one or more images. The one or more depth maps may be estimated or determined using any technique discussed herein, including, but not limited to, artificial intelligence. For example, any Al network, including, but not limited to a neural network, a convolutional neural network, a generative adversarial network, any other Al network or structure discussed herein or known to those skilled in the art, etc., may be used to estimate or determine the depth map or maps (e.g., automatically). The use of a target detection method value (e.g., td = 1, td = 2, td = 3) and / or a counter (cn = 1 or cn = 2) is illustrative, and the autonomous navigation, movement detection, and / or control technique(s) of the present disclosure are not limited thereto. For example, in one or more embodiments, a counter may not be used and / or a target detection method value may not be used such that at least one embodiment may iteratively perform a target detection method of a plurality of target detection methods and move on and use the next target detection method of the plurality of the target detection methods until a target or targets is / are found. Alternatively or additionally, even in a case where a target or targets has / have been found already using a particular target detection method, one or more embodiments may continue to use one or more of the other target detection methods (or any combination of the plurality of target detection methods or modes) to confirm and / or evaluate the accuracy and / or results of the target detection method or mode used to find the already-identified one or more targets. In other words, the identified one or more targets may be double checked, triple checked, etc. In one or more embodiments, the deepest point method or mode of step S711 may be used as a backup to identify a target or targets in a case where other target detection methods do not find any targets (# targets = o).Additionally or alternatively, one or more steps of FIG. 7, such as, but not limited to step S707 for binarization, maybe omitted in one or more embodiments.

[0114] In one or more embodiments, a non-transitory computer-readablestorage medium may store at least one program for causing a computer to execute a method for performing autonomous navigation, movement detection, and / or control of a continuum robot or catheter, the method comprising one or more of the following steps: (i) in step S700, one or more images (e.g., one or more camera images, one or more CT images (or images of another imaging modality), one or more bronchoscopic images, etc.) are obtained; (ii) in step S701, a target detection method is selected (automatically or manually) (e.g., target detection (td) = 1 for the peak detection method or mode, td = 2 for the thresholding method or mode, td =3 for the deepest point method or mode, etc. - the target detection methods shown in FIG. 7 are illustrative, are not limited thereto, and may be exchanged or substitute or used along with any combination of detection methods discussed in the present disclosure or known to those skilled in the art); (iii) in step S703, based on the td value, the method continues to perform the selected target detection method and proceeds to step S704 for the peak detection method or mode, to step S706 for the thresholding method or mode, or to step S711 for the deepest point method or mode; (iv) in a case where td = 1, the peak detection method or mode is performed in step S704, a target or targets are set to be the detected peak or peaks in step S705 and a counter (cn) is set to 2 (cn=2), and a number of targets is evaluated in step S710 such that, in a case where no targets are found (# targets = o) and the counter = 2, then td is set to a value of 3 and the process returns to the depth map step S702 and proceeds to step S711 for the deepest point method or mode; (v) in a case where td = 2, the thresholding method or mode is performed in step S706 to identify one or more objects, the counter is set to be equal to 1 (cn = 1), binarization is performed in step S707 to process the image data (e.g., the image data may be converted from color to black and white images, the image data maybe split into data sets, etc.), fitting a circle in or on each object is performed in step S708 (also referred to as a blob fit or blob detection method), a target or targets is / are set to be at a predetermined or set location (e.g., a center) of the circle for each object of the one or more objects in step S709, a number of targets is evaluated in step S710 such that, ina case where no targets are found (# targets = o) and the counter = 1, then td is set to a value of i and the process returns to the depth map step S702 and proceeds to step S704 for the peak detection method or mode in step S704, a target or targets is / are identified as the detected peak or peaks in step S705, and a number of targets is evaluated in step S710; and (vi) in a case where the number of targets evaluated in step S710 is 1 or more, then the process proceeds to step S712 where the continuum robot or steerable catheter (or other imaging device or system) (e.g., the continuum robot or steerable catheter 104) is moved to the target or targets. In one or more embodiments, the steps S701 through S712 of FIG. 7 may be performed again for an obtained or received next image or images to evaluate the next movement, pose, position, orientation, or state for the autonomous navigation, movement detection, and / or control of the continuum robot or steerable catheter (or imaging device or system) 104. In step S702, the method may estimate (automatically or manually) the depth map or maps (e.g., a 2D or 3D depth map or maps) of one or more images. The one or more depth maps may be estimated or determined using any technique discussed herein, including, but not limited to, artificial intelligence. For example, any Al network, including, but not limited to a neural network, a convolutional neural network, a generative adversarial network, any other Al network or structure discussed herein or known to those skilled in the art, etc., may be used to estimate or determine the depth map or maps (e.g., automatically). The use of a target detection method value (e.g., td = 1, td = 2, td = 3) and / or a counter (cn = 1 or cn = 2) is illustrative, and the autonomous navigation, movement detection, and / or control technique(s) of the present disclosure are not limited thereto. For example, in one or more embodiments, a counter may not be used and / or a target detection method value may not be used such that at least one embodiment may iteratively perform a target detection method of a plurality of target detection methods and move on and use the next target detection method of the plurality of the target detection methods until a target or targets is / are found. Alternatively or additionally, even in a case where a target or targets has / have been found already using a particular targetdetection method, one or more embodiments may continue to use one or more of the other target detection methods (or any combination of the plurality of target detection methods or modes) to confirm and / or evaluate the accuracy and / or results of the target detection method or mode used to find the already-identified one or more targets. In other words, the identified one or more targets may be double checked, triple checked, etc. In one or more embodiments, the deepest point method or mode of step S711 may be used as a backup to identify a target or targets in a case where other target detection methods do not find any targets (# targets = o). Additionally or alternatively, one or more steps of FIG. 7, such as, but not limited to step S707 for binarization, maybe omitted in one or more embodiments. For example, if segmentation is done using three categories, such as airways, background and edges of the image, then instead of a binary image, the image has three colors.

[0115] FIG. 8(a) shows images of at least one embodiment of an application example of autonomous navigation and / or control technique(s) and movement detection for a camera view 800 (left), a depth map 801 (center), and a thresholded image 802 (right) in accordance with one or more aspects of the present disclosure. A depth map may be created using the bronchoscopic images and then, by applying a threshold to the depth map, the airways may be segmented. The segmented airways shown in thresholded image 802 may define the navigation targets (shown in the octagons of image 802) of the next automatic robotic movement.

[0116] Fig. 8B shows images of showing a camera view (left), a semitransparent color coded depth map overlaid onto a camera view (center) and a thresholded image (right).

[0117] In one or more embodiments, the continuum robot or steerable catheter 104 may follow the target(s) (which a user may change by dragging and dropping the target(s) (e.g., a user may drag and drop an identifier for the target, the user may drag and drop a cross or an x element representing the location for the target, etc.) in one or more embodiments), and the continuum robot or steerable catheter 104 may move forward and rotate on its own while targeting apredetermined location (e.g., a center) of the target(s) of the airway. In one or more embodiments, the depth map (see e.g., in image 801) may be processed with any combination of blob / circle fit, peak detection, and / or deepest point methods or modes to detect the airways that are segmented. As aforementioned, the detected airways may define the navigation targets of the next automatic robotic movement. In a case where a cross or identifier is used for the target(s), the continuum robot or steerable catheter 104 may move in a direction of the airway with its center closer to the cross or identifier. The continuum robot or steerable catheter 104 may move forward and may rotate in an autonomous fashion targeting the center of the airway (or any other designated or set point or area of the airway) in one or more embodiments.

[0118] Additionally, a study was conducted to introduce and evaluate new and non-obvious techniques for achieving autonomous advancement of a multi-section continuum robot within lung airways, driven by depth map perception. By harnessing depth maps as a fundamental perception modality, one or more embodiments of the studied system aims to enhance the robot’s ability to navigate and manipulate within the intricate and complex anatomical structure of the lungs (or any other targeted anatomy, object, or sample). The utilization of depth maps enables the robot to accurately perceive its environment, facilitating precise localization, mapping, and obstacle avoidance. This, in turn, helps safer and more effective robot-assisted interventions in pulmonary procedures. Experimental results highlight the feasibility and potential of the depth map-driven approach, showcasing its ability to advance the field of minimally invasive lung surgeries (or other minimally invasive surgical procedures, imaging procedures, etc.).

[0119] As aforementioned, continuum robots are flexible systems used in transbronchial biopsy, offering enhanced precision and dexterity. Training these robots is challenging due to their nonlinear behavior, necessitating advanced control algorithms and extensive data collection. Autonomous advancements are crucial for improving their maneuverability.

[0120] Sganga, et al. introduced deep learning approaches for localizing a bronchoscope using real-time bronchoscopic video as discussed in J. Sganga, D. Eng, C. Graetzel, and D. Camarillo, “Offsetnet: Deep learning for localization in the lung using rendered images,” in 2019 International Conference on Robotics and Automation (ICRA), 2019, pp. 5046-5052, the disclosure of which is incorporated by reference herein in its entirety. Zou, et al. proposed a method for accurately detecting the lumen center in bronchoscopy images as discussed in Y. Zou, B. Guan, J. Zhao, S. Wang, X. Sun, and J. Li, “Robotic-assisted automatic orientation and insertion for bronchoscopy based on image guidance,” IEEE Transactions on Medical Robotics and Bionics, vol. 4, no. 3, pp. 588-598, 2022, the disclosure of which is incorporated by reference herein in its entirety. However, there are drawbacks to the techniques discussed in the Sganga, et al. and Zou, et al. publications.

[0121] This study of the present disclosure aimed to develop and validate the autonomous advancement of a robotic bronchoscope using depth map perception. The approach involves generating depth maps and employing automated lumen detection to enhance the robot’s accuracy and efficiency. Additionally, an early feasibility study evaluated the performance of autonomous advancement in lung phantoms derived from CT scans of lung cancer subjects.

[0122] Bronchoscopic operations were conducted using a snake robot developed in the researchers’ lab (some of the features of which are discussed in F. Masaki, F. King, T. Kato, H. Tsukada, Y. Colson, and N. Hata, “Technical validation of multi-section robotic bronchoscope with first person view control for transbronchial biopsies of peripheral lung,” IEEE Transactions on Biomedical Engineering, vol. 68, no. 12, pp. 3534-3542, 2021, which is incorporated by reference herein in its entirety), equipped with a bronchoscopic camera (OVM6946 OmniVision, CA). The captured bronchoscopic images were transmitted to a control workstation, where depth maps were created using a method involving a Three Cycle-Consistent Generative Adversarial Network (3CGAN) (see e.g., a 3CGAN as discussed in A. Banach, F. King, F. Masaki, H. Tsukada, and N. Hata, “Visuallynavigated bronchoscopy using three cycle-consistent generative adversarial network for depth estimation,” Medical Image Analysis, vol. 73, p. 102164, 2021. [Online]. Available: https: / / www.sciencedirect.com / science / article / pii / S1361841521002103, the disclosure of which is incorporated by reference herein in its entirety). A combination of thresholding and blob detection algorithms, methods, or modes was used to detect the airway path, along with peak detection for missed airways.

[0123] A control vector was computed from the chosen point of advancement (identified centroid or deepest point) to the center of the depth map image. This control vector represents the direction of movement on the 2D plane of original RGB and depth map images. A software-emulated joystick / gamepad was used in place of the physical interface to control the snake robot (also referred to herein as a continuum robot, steerable catheter, imaging device or system, etc.). The magnitude of the control vector was calculated, and if magnitude fell below a threshold, the robot advanced. If the magnitude exceeded the threshold, the joystick was tilted to initiate bending. This process was repeated using a new image from the Snake Robot interface.

[0124] During each trial of the autonomous robotic advancement, the robotic bronchoscope was initially positioned in front of the carina within the trachea. The program was initiated by the operator or user, who possessed familiarity with the software, initiating the robot’s movement. The operator’s or user’s sole task was to drag and drop a green cross within the bronchoscopic image to indicate the desired direction. Visual assessment was used to determine whether autonomous advancement to the intended airway was successfully achieved at each branching point. Summary statistics were generated to evaluate the success rate based on the order of branching generations and lobe segments.<Voice Input and Visualization >

[0125] In some embodiments, the user input device is a voice input device.Since the autonomous system is driving the steerable catheter, full directional control of the catheter is unnecessary for autonomous driving. It can also be unwanted. Alimited library of commands that the user can provide gives full control to the user to select which lumen is the correct one for the next navigation step, but prevents the user from, for example, trying to keep the steerable catheter in the center of the lumen since this can be accomplished through the autonomous function. The limited library also simplifies the system.

[0126] (Effective commands for the system) Effective voice commands form a limited library in the system and are, for the system are (1) start, (2) stop, (3) center, (4) up, (5) down, (6) right, (7) left, and (8) back. Voice commands sent to the system are classified as one of the effective commands or ignored when the sent command is not recognized as the effective commands.

[0127] (Autonomous Navigation with voice command) When the user starts the autonomous navigation, the user sends a voice command, “start” (S1010) and “Autonomous navigation mode” is displayed on the main display 118. The autonomous navigation system detects airways in the camera view (S1020). The centers of detected airways are displayed as diamond mark (560) in the detected airways in the camera view.

[0128] In order for the user to select an airway for the steerable catheter to move in, the user sends the one of voice commands from the options of “center”, “up”, “down”, “right” and “left”. When the voice command is accepted by the system, the color of “x” mark on the selected location is changed from black to red and a triangle 570 is displayed on the selected mark.

[0129] The selected location stays at the same location until a different location is accepted to the system. As the default, the “x” mark on the center is set when the autonomous navigation mode is started.

[0130] The system sets the closest airway from the selected x mark as the airway to be aimed based on the distance between the selected x mark and each diamond mark in the detected airways (S1030).

[0131] By choosing the closest airway as an operator’s intended airway from the “+” mark, which the operator moves with the voice commands, the operator caninstruct the intended airway intuitively and accurately. The operator can always clearly confirm the distance between the “+” mark and the intended airway on the display and easily understand which airway option the autonomous system will choose on the display. Therefore, this transparency gives the operator predictable system behavior and operation confidence during autonomous operation.

[0132] Also, since the position options of “+” mark include a limited number, the operator can determine the next position option easily and quickly.

[0133] Moreover, with this method, the autonomous system always has at least one intended airway until there is at least one airway candidate. This feature avoids the situation without the intended airway and make the system behavior robust.

[0134] The system detects the target point in the airway to be aimed, and shows a “+” mark 540 as the target point and an arrow 580 connecting the “+” mark and the center of the camera view (S1040).

[0135] The autonomous navigation system compares the distance between the center of the camera view and the target point with the threshold (S1050). If the distance between the center of the camera view and the target point is longer than the threshold, the robotic platform bends the steerable catheter toward the target point (S1060) until the distance between the center of the camera view and the target point is smaller than the threshold. If the distance between the center of the camera view and the target point is smaller than the threshold, the robotic platform moves the linear translational stage forward (S1080).

[0136] (Stop the autonomous navigation) The user can stop the autonomous navigation any time by sending a voice command, “stop”, and can start the manual navigation using a handheld controller 124 to control the steerable catheter and the linear translational stage. When the user takes over the control, “Manual navigation mode” is displayed on the main display 118. The user can restart the autonomous navigation when needed by sending a voice command, “start”.

[0137] (Retraction) While the robotic platform is bending the steerablecatheter and moving the linear translational stage forward, all input signals to the robotic platform are recorded in the data storage memory HDD 188 regardless of navigation modes. When the user needs to retract the robotic platform, the user sends a voice command, “back”, then the robotic platform inversely applies the recorded input signals taken during insertion to the steerable catheter and the linear translational stage. During retraction, the system displays “back” on the display.

[0138] (Backup and alternative systems) Instead of sending voice commands, the user can send the commands using other input devices including a number pad or a general computer keyboard. The commands can be assigned as numbers on the number pad or other letters on the keyboard. The other input devices may be used along with or instead of voice commands. In some embodiments, the input device is has a limited number of keys / buttons that can be pressed by the user. For example, a numerical keypad is used where 2, 4, 6, and 8 are the four directions and 5 is center.

[0139] One or more of the aforementioned features may be used with a continuum robot and related features as disclosed in U.S. Provisional Pat. App. No. 63 / 150,859, filed on February 18, 2021, the disclosure of which is incorporated by reference herein in its entirety. For example, FIGS. 9 to 11 illustrate features of at least one embodiment of a continuum robot apparatus 10 configuration to implement automatic correction of a direction to which a tool channel or a camera moves or is bent in a case where a displayed image is rotated. The continuum robot apparatus 10 enables to keep a correspondence between a direction on a monitor (top, bottom, right or left of the monitor) and a direction the tool channel or the camera moves on the monitor according to a particular directional command (up, down, turn right or turn left) even if the displayed image is rotated. The continuum robot apparatus 10 also may be used with any of the autonomous navigation, movement detection, and / or control features of the present disclosure.< Difficulty Index >

[0140] Naito et al. showed that geometrical indices of the lung can be used as a predictor of the successful bronchoscopy when a bronchoscopist uses a manual conventional bronchoscope (*). The subject innovation has shown that geometrical indices can also predict the probability of the successful robotic bronchoscopy with autonomous navigation system through our ex-vivo study. As provided in Fig. 18, the boxplots show the local curvature (LC) defined in the literature for successful and failed autonomous navigation in an ex-vivo lung block of a swine. Fig. 19 shows the logistic regression curve to predict the probability of the successful robotic bronchoscopy with autonomous navigation system using LC as a predictor.(*Naito M, Masaki F, Lisk R, Tsukada H, Hata N. Predicting reachability to peripheral lesions in transb ronchial biopsies using CT-derived geometrical attributes of the bronchial route. Int J Comput Assist Radiol Surg. 2023 Feb;i8(2):247-255. doi: 10.1007 / 511548-022-02723^. Epub 2022 Aug 20. PMID: 35986830.)< Validation study >

[0141] To evaluate the performance of the autonomous navigation, a validation study using a living swine model was conducted. The study was designed as a comparison study to compare the autonomous navigation with navigation by a human operator using a handheld controller. The autonomous navigation and the human operator were tasked to navigate the robotic catheter toward the peripheral areas of the lung of the living swine model. In the study, time for bending command and force defined below were collected as metrics to compare the autonomous navigation with navigation by the human operator.

[0142] (Airway generation) The airway generation for the data analysis of this study was defined as follows (see Fig. 20). First the airway shape was segmented using a pre-procedural CT scan. Then bifurcation points were identified in the segmented airway. Middle points were defined as the middle of two consecutive bifurcation points. An area between two middle points was defined as one generation, starting the first generation at the carina.

[0143] (Time) The total time for the system to receive the bending commands at each airway generation was defined as “time for bending command”.

[0144] (Force) The force applied to each driving wire was recorded at 100 Hz using a calibrated load cell. The maximum force at each airway generation was extracted and defined as “Max force at each airway generation”.

[0145] (Results) The median times for bending command were 2.5 [sec] (IQR = 1.0-5.6) and 1.3 [sec] (IQR = 0.7-2.3) for human operator and autonomous navigation, respectively (Fig. 21). The medians of the maximum force at each bifurcation point were 2.8 (IQR = 1.1-3.8) [N] and 1.4 (IQR = 0.9-2.1) [N] for the human operators and autonomous navigation, respectively (Fig. 22). The dependency of the time and the force on the airway generation of the lung are shown in Fig. 23 and Fig. 24 with regression lines and 95% confidential intervals.

[0146] (Findings) For both metrics, the regression lines for the autonomous navigation are smaller than the regression lines for the human operator. At each airway generation, the dispersion of data points for autonomous navigation was smaller than the dispersion of data points for human operator. For example, the data points of time at 5thgeneration for autonomous navigation ranged between o and 5 [s], whereas the data points for the human operator dispersed between o and 20 [s].

[0147] Fig. 32 shows specific data corresponding to the graph Fig.21 and Fig.23. Inventors found that there is less dispersion with autonomous navigation with respect to time spent to drive a catheter through a certain path. The Inventors found that an abnormality of operation is predicable or detectable based on the data when autonomous navigation system is used, while it is difficult to predict or detect the abnormality when a manual insertion is performed because the time taken to navigate the catheter differs depends on skills or experiences of the operator. The autonomous navigation system determines whether any abnormality of operation is happening based on the time spent to drive the catheter through the certain path and the driving mode of the robotic tool is changed from an autonomous driving mode toother driving mode in a case where a time spent to complete bending of the robotic tool during a driving through a certain path is more than a predetermined period.< Overall workflow>

[0148] Fig. 25 shows the flowchart for overall workflow based on our findings described above.

[0149] (Selection of type of bronchoscopy) First, pre-procedural CT scan is taken, and difficulty indices were calculated. Then, logistic regression calculates probabilities for all types of candidates of bronchoscopy, including manual conventional bronchoscopy, robotic bronchoscopy with a handheld controller, and robotic bronchoscopy with autonomous navigation, using one of or a combination of the calculated difficulty indices. Noting that the methods to calculate each probability can be different. Even if a same regression model was selected, the coefficients for the regression model are different for each type of bronchoscopy. The color-coded probabilities are shown along the centerline of segmented airway on a monitor for a clinician to select a type of bronchoscopy.

[0150] (Estimation of time) Once robotic bronchoscopy is selected, the total time for navigating the robotic catheter is estimated using one of the regression lines in Fig. 23.

[0151] (Safety thresholds for autonomous navigation) Once robotic bronchoscopy with autonomous navigation is selected, two types of safety threshold, time and force, are set based on Fig. 23 to Fig. 24.

[0152] (Fixed threshold) One method to decide the thresholds is to set the threshold at the upper whisker of Fig. 26 and Fig. 27, which is defined as Q3 (75thpercentile)+i.5*IQR (interquartile range). The threshold for time is shown in Fig. 28 as an example.

[0153] (Shifted slope as threshold) Another method to decide the threshold is to set a shifted slope, which is defined as the slope of regression line + 3*average of all measured data points. The shifted slope as the threshold for force is shown in Fig. 29 as an example.

[0154] (Bronchoscopy) After robotic bronchoscopy with autonomous navigation starts, the catheter tip position detector 142 starts tracking the location of the tip of the robotic catheter and shows the current airway generation of the robotic catheter. Every time the robotic catheter enters into a new airway generation, the system controller 128 starts counting the total time for bending at the current airway generation. When the total time for bending reaches the threshold, the system controller 128 stops the autonomous navigation, and shows a message, “take over control” on a monitor, for a clinician to control the robotic catheter with a handheld controller. After the clinician passes the airway generation where the autonomous navigation failed, the clinician restarts the autonomous navigation. Once the bronchoscopy finished, the difficulty indices and the safety thresholds are updated using the data obtained from the bronchoscopy.

[0155] (Alternative method to define time) Instead of identifying the airway generation of the robotic catheter and counting the total time for bending command in the identified airway generation, the total time for bending is counted as the total time for the system controller to receive bending commands while the linear translational stage 110 moves a pre-determined length from any arbitral point. The pre-determined length can be determined as the shortest length of airway generations along the pathway to the tumor to be biopsied measured in the segmented airway or as the shortest length calculated by models such as Weibel’s model.(**Weibel ER (1963) Morphometry of the human lung, chapter geometric and dimensional airway models of conductive, transitory and respiratory zones of the human lung. Springer, New York, pp 136-142)

[0156] (Example of failure mode) Fig. 31 shows the camera views at inhalation and exhalation phase. At the inhalation phase, all three airways in front of the camera are detected as airway, and A-i is set as the airway for the robotic catheter to move in. However, due to the breathing motion, A-i collapses. The autonomous navigation loses the airway in the camera view. Then the autonomous navigation setsA-2 as the airway for the robotic catheter to move in and start bending toward A-2. When the A-i opens again at the inhalation phase, the autonomous navigation re-set A-i as the airway for the robotic catheter to move in. In this situation, the robotic catheter bends back and forth between A-i and A-2, and is stuck at this bifurcation point if there is no time threshold.<Overall workflow -2>

[0157] (Bronchoscopy) When robotic bronchoscopy with autonomous navigation starts, the color-coded difficulty index is shown on the monitor along the pathway on the segmented airway from the carina to the tumor to be biopsied (Fig. 30) and the position of the linear translational stage 110 was set o at the carina. By comparing the insertion depth of the linear translational stage with the path on the segmented airway, the system controller decides the location of the robotic catheter. When the robotic catheter gets close to an area where the difficulty index is high, the system controller 128 stops the normal autonomous navigation, and shows a message, “take over control” on a monitor, for a clinician to control the robotic catheter with a handheld controller.

[0158] Instead of the clinician directly taking over the control, the system controller 128 can automatically switches the mode of the autonomous navigation from the normal autonomous navigation mode to the difficult autonomous navigation mode.

[0159] In the difficult autonomous navigation mode, the system controller 128 changes the internal settings of autonomous navigation. The internal settings include the speed of bending, the speed of the linear translational stage, and the criteria to decide bending or moving forward.

[0160] When the difficult autonomous navigation mode is started, the system controller 128 moves the tip of the catheter to look around to show the blind spots for the normal autonomous navigation mode to the clinician. Then the system controller 128 shows a message, “take over control” on a monitor, for a clinician to control therobotic catheter with a handheld controller. After the clinician passes the difficult area, the clinician restarts the normal autonomous navigation.<Planning step for autonomous navigation >

[0161] FIGS. 9 and 10 signify a design example of the planning step. In this design example, the system controller 128 highlights the two paths found in the perception step as path 1 (902) and path 2 (904). The display also includes a cursor 906 shown as a crosshair. If FIG. 9, the cursor 906 has been moved into path 2 (904), having a white circular feature with the user instruction and is selected as the target path. For example, the user can use the handheld controller (e.g., a joystick) 124 to select or change the target path by clicking on or inside a lumen or on or inside an indication of the lumen / path candidate. During this maneuver, the system controller 128 change the path 1 or path 2 based on the user instruction. This change may be a color change (black and white circles or shown, but any color scheme or other indicator, such as red and green, black and green, bolded, highlighted, may be used). FIG. 9 is a selection of the right-most lumen. FIG. 10 is a selection of the leftmost lumen. In FIG. 10, the selection is indicated by a bolder circle for path 1 / 904 the target path (e.g., path 1, 904) and a dashed circle for the unselected path 902. Concurrent user instruction for the target path among the paths allows reflecting user’s intention to the robotic catheter system during autonomous navigation effectively. Another optional feature is also shown in FIG. 10, where the cursor 906 has changed from a crosshair to a circle. In the workflow leading up to the image shown in FIG. 10, the user moved the cursor 906, which is optimized for the user to view both the cursor and the underlying image, until it was touching or inside the selected path 904, at which time, the cursor 906 is changed to a less obvious indicator, shown here as a small circle.

[0162] While the user may select or change the target path at any point in the planning step, as the autonomous system move into the control step, the user can optionally adjust the target path as described herein, or the prior target path can be used until the predetermined insertion depth is reached. The need for additionalselection of the target path can depend on the branching of the lumen network, where, for example, the user provides target path information as each branch within the airway becomes visible in the camera image.

[0163] While color is used in this example to indicate the selection of the target path, other colors or other indicators may be used as well, such as a bolder indicator, a flashing indicator, removal of the indicator around the non-selected path lumen, etc.

[0164] By having the user output device and displaying symbols for the paths and user instruction GUI with endoscope view in the user output device, user can form their intention intuitively and accurately with the visual information in one place. Especially, the dedicated user instruction GUI allows the user to select the target path immediately even when the paths are more than two.

[0165] By having the user output device and displaying symbols for the paths and differentiating the symbol for the target path from the other paths with endoscope view in the user output device, the user can form intention intuitively and accurately with the visual information in one place. Particularly, the differentiating the symbol for the target path achieve the user instruction with the minimal symbols without the dedicated user instruction GUI and allows the user to learn / understand how to read symbols with the minimal effort.

[0166] As shown in embodiments herein, circles (or ovals) are used as the symbol of the paths on the 2D interface, and the cursor provides a symbol for input of user instruction to the GUI. These GUIs have minimal obstacles for the endoscope view in the user output device. Also, selecting object with the cursor are very familiar maneuver from common computer operation, the user can easily learn how to use it.

[0167] By pausing the actuator and the linear translation stage during user’s instruction, the system can reduce the risk where the user miss the target path in their interaction. Also, this gives users to think and judge the target path among the paths without pressurizing the user to make decisions in the short time.

[0168] By allowing the user to add a new target path if the user cannot find thetarget path among the existing paths and find the target path in the endoscope view, the plan generated by the system becomes more accurate with minimal effort of the user.< Control >

[0169] A typical procedure of bronchoscopy to diagnose tumorous tissue in a region of interest is described below. FIG. 11 shows the typical camera image during the autonomous navigation and FIG. 12 shows the flowchart of the autonomous navigation.

[0170] In FIG. 12, the system detects the airways in the camera view (S1020). This method and system are particularly described above, and include using a depth map produced by processing one or more images obtained from the camera, fitting the lumen or lumens (e.g., airways) using an algorithm such as a circle fit, peak detection algorithm or similar, where, in instances of the camera image not able to perform the algorithm to find one or more lumen, using the deepest point. Next, the system sets the airway to be aimed (S1030). This provides a target path. This can be done by user interaction as discussed hereinabove or through an automated process. The system then detects the target point in the lumen (e.g., airway) for the target path (S1040). The target point may be the center of the circle that was fit to the airway.

[0171] Then, the autonomous system must aim the distal end of the steerable catheter towards the target point and move the steerable catheter forward, towards the target point (S1050). For this step, a threshold is set. If the target point is inside of the threshold, the steerable catheter is advanced, increasing the insertion depth (S1070). If the target point is not inside of the threshold, then the distal end of the steerable catheter is bent towards the target point (S1060). After the steerable catheter is bent further, the controller must re-assess whether the target point is inside of the threshold (S1050). If the steerable catheter has not yet reached the region of interest, or as at the predetermined insertion depth, the steerable catheter will move forward S1080). If the steerable catheter has reached the region ofinterest, or as at the predetermined insertion depth, the automated procedure will end (S1090).

[0172] Of note, in this process, after a particular iteration of moving forward (S1080) or bending toward the target point (S1060), the system then returns to detecting the lumen (e.g., airway) in the camera view. Thus, each iteration can be performed with a new and separate image taken by the camera. Knowledge of locations within the last image as well as knowledge obtained from prior data, such as from a pre-operative CT image, are not required. Therefore, robust registration algorithm(s) are not required to perform this automated driving function. This can be particularly advantageous in multiple situations and will not be effected by breathing as much as many attempts at autonomous navigation as seen in the literature.

[0173] For a bronchoscopic procedure, various parameters of the robotic platform 106 including the frame rate of the camera, the bending speed of the steerable catheter, the speed of linear translational stage 110, and the predetermined insertion depth of the linear translational stage are set. They each independently may be preset (e.g., a standard base value) or set by the user for the procedure. If set by the user, it may be based on the user’s preference, or based on specifics of the patient. The parameters maybe calculated or be obtained from a look-up table. The predetermined insertion depth may be estimated as the distance from the carina to the region of interest based on, for example, one or more preprocedural CT images. These parameters may be set before the start of the automated motion portion of the procedure, such as when the steerable catheter 104 reaches the carina.

[0174] A threshold is used during autonomous function to decide whether to bend the steerable catheter to optimize the direction and / or angle of the tip or to move forward using the linear translational stage. The threshold 510 may be a constant and can be defined based on the dimensions of the camera view. The threshold relates to the distance from the center of the camera view, (e.g., the center of the dotted circle 510 in a camera view 520, which is the center of the distal end ofthe steerable catheter) to the center of the airway 530 that has been selected as the target path (the target point 540). The threshold value is visualized in FIG. 11 as a dotted circle 510, however, it can alternatively be configured as a vector. The vector represents the direction of movement on the image plane of the image from the camera and depth map images, where the magnitude of the vector is the threshold value. The distance to be set as the threshold may be decided based on, for example, data from a lung phantom model. Alternatively, the threshold may be based on a library of threshold data. In some embodiments, the threshold set to 10%, 15%, 20%, 25%, 30%, 35%, or 40% or a value therebetween of the camera view dimension (i. e. , the distance of a diagonal line across the camera view), in some embodiments, the threshold set to between 25% and 35%, or around 30%.

[0175] In some embodiments, when the user starts the autonomous navigation, an indicator of the navigation mode being used, such as displaying “Autonomous navigation mode” on the main display 118. The autonomous navigation system detects airways in the camera view (S1020). The user places a mouse pointer 550 on the airway to be aimed 530 in the camera view for the autonomous navigation system to set the airway to be aimed (S1030). The autonomous navigation system detects the target point 540 in the detected airway as described above (S1040), then the autonomous navigation system compares the distance between the center of the camera view and the target point with the threshold (S1050). If the distance between the center of the camera view and the target point is longer than the threshold, the robotic platform bends the steerable catheter toward the target point (S1060) until the distance between the center of the camera view and the target point is smaller than the threshold. If the distance between the center of the camera view and the target point is smaller than the threshold, the robotic platform moves the linear translational stage forward (S1080).

[0176] In some embodiments, the user has the ability to stop the autonomous navigation any time by pushing a button on the handheld controller 124 and can start the manual navigation to control the steerable catheter and the linear translationalstage by the handheld controller. When the user takes over the control, an indicator of this control is provided, such as a display of “Manual navigation mode” on the main display 118. The user can restart the autonomous navigation when needed by, for example, pushing a button on the handheld controller 124.

[0177] When the steerable catheter reaches a position close to the region of interest (e.g., tumorous tissue), the user has the option to switch the navigation mode to the manual navigation and, for example, deploy a biopsy tool toward the region of interest to take a sample through the working tool access port.

[0178] A Return to the Carina function may be included with the autonomous driving catheter and system. While the robotic platform is bending the steerable catheter and moving the linear translational stage forward, all input signals to the robotic platform may be recorded in the data storage memory HDD 188 regardless of navigation modes. When the user indicates a start of the Return to Carina function (e.g., hitting the appropriate button on the handheld controller 124), the robotic platform inversely applies the recorded input signal taken during insertion to the steerable catheter and the linear translational stage.

[0179] In some embodiments, an insertion depth may be set before driving the steerable catheter. When the linear translational stage reaches the predetermined insertion depth, the autonomous navigation system can be instructed to stop bending the steerable catheter and moving the linear translational stage forward (S1070). The robotic platform then switches the mode from the autonomous navigation to the manual navigation. This allows the user to start interacting with the region of interest (e.g., take a biopsy) or to provide additional adjustments to the location or orientation of the steerable catheter.

[0180] In some embodiments, the frame rate is set for safe movement. The steps from S1020 to S1080 in FIG. 12 can be conducted at every single frame of camera image. Thus, depending on the capability of the Central Processing Unit 182, the maximum frame rate of the camera image that can be handled for the autonomous navigation is decided. Then based on the frame rate and the acceptablerisk during bronchoscopy, the speed of bending the steerable catheter and the speed of moving the linear translational stage are decided. If it is important to move the steerable catheter based on the images when the steerable catheter is moving faster than can be ‘seen’ by the images from the camera. In an example where the maximum frame rate is to frames per second (fps) and the acceptable amount of the airway pushed by the steerable catheter is 0.5mm, the speed of the linear translational stage maybe set less than 5 [mm / sec]. Other frame rates and risk factors will suggest different speeds.<Variable speed>

[0181] FIG. 13 shows an exemplary flowchart to set the speed of bending the steerable catheter and the speed of moving the linear translational stage based on the target point in the detected airway to be aimed.

[0182] FIG. 14 shows an exemplary display at the parameter settings. The user can set the bending speed of the steerable catheter at two points in the camera view, Bending speed 1 and Bending speed 2. During autonomous navigation, the autonomous navigation system sets the bending speed of the steerable catheter by linearly interpolating Bending speed 1 and Bending speed 2 based on the target point in the detected airway to be aimed (S1055). In general, Bending speed 2 is slower than Bending speed 1 so that the steerable catheter doesn’t overbend.

[0183] As shown in the exemplary display of FIG. 14, the user can set the bending speed of the linear translational stage at two points in the camera view, Moving speed 1 and Moving speed 2. During autonomous navigation, the autonomous navigation system sets the moving speed of the linear translational stage by linearly interpolating Moving speed 1 and Moving speed 2 based on the target point in the detected airway to be aimed (S1075). In general, Moving speed 1 is faster than Moving speed 2 because the closer to the center of the airway the steerable catheter is, the less risky the steerable catheter collide to the airway wall.

[0184] According to this embodiment, the steerable catheter can reach the target point faster with less risk for the steerable catheter to collide to the airwaywall.<Variable threshold>

[0185] FIG. 15 shows an exemplary flowchart to adjust the threshold based on the location of the steerable catheter in the lung.

[0186] FIG. 16 shows an exemplary the display at the parameter settings where two different thresholds are used. The user can set two thresholds to decide for the autonomous function to bend the steerable catheter or to move forward the linear translational stage based on the insertion depth at the parameter settings. In this example, Threshold 1 and Threshold 2 are located at the carina and at the region of interest in the planning view created from the preoperative CT image. Thus, the autonomous driving can start using Threshold 1 when the airway is large and the need for tight control and steering is not as great. Then, as the steerable catheter moves down the airways that narrow toward the peripheral area of the lung, a second threshold (Threshold 2) is needed, where Threshold 2 is smaller than Threshold 1. Since the airway is smaller the further into the lung the catheter moves, the robotic catheter needs to be bent more accurately towards the center of the airways before it moves forward.

[0187] During autonomous navigation, the autonomous navigation system can set the threshold at each frame of the camera view. When two thresholds are used as discussed above, the threshold can be a linearly interpolation of Threshold 1 and Threshold 2 based on the insertion depth (S1045). According to this embodiment, the steerable catheter can move faster and spends less time to bend around the carina, leading to less time for bronchoscopy, but maintains an accurate and precise navigation further into the periphery where a deviation from the center of the airway would increase risk to the patient.<Emergency situations >

[0188] FIG. 17 shows a flowchart with an exemplary method to abort the autonomous navigation when the blood is detected in the camera view. The criterion to abort bronchoscopy may be defined as the ratio of the number of pixels indicatingthe blood divided by the total number of pixels in a camera image. In this embodiment, an imaging processing library, e.g. OpenCV, is used to count the number of red pixels in a RGB camera view. If the ratio of the number of pixels indicating the blood divided by the total number of pixels in a camera image exceeds the predetermined ratio, the autonomous navigation is aborted. Similar to the blood, the mucous in the airway can be detected using an imaging processing library. For detecting mucus, the number of yellow pixels in a RGB camera view can be used. According to this embodiment, the steerable catheter can be automatically stopped during bronchoscopy when an emergency situation is detected.

[0189] The present disclosure and / or one or more components of devices, systems, and storage mediums, and / or methods, thereof also may be used in conjunction with continuum robot devices, systems, methods, and / or storage mediums and / or with endoscope devices, systems, methods, and / or storage mediums. Such continuum robot devices, systems, methods, and / or storage mediums are disclosed in at least: U.S. Provisional Pat. App. No. 63 / 150,859, filed on February 18, 2021, the disclosure of which is incorporated by reference herein in its entirety. Such endoscope devices, systems, methods, and / or storage mediums are disclosed in at least: U.S. Pat. App. No. 17 / 565,319, filed on December 29, 2021, the disclosure of which is incorporated by reference herein in its entirety; U.S. Pat. App. No. 63 / 132,320, filed on December 30, 2020, the disclosure of which is incorporated by reference herein in its entirety; U.S. Pat. App. No. 17 / 564,534, filed on December 29, 2021, the disclosure of which is incorporated by reference herein in its entirety; and U.S. Pat. App. No. 63 / 131,485, filed December 29, 2020, the disclosure of which is incorporated by reference herein in its entirety. Any of the features of the present disclosure may be used in combination with any of the features as discussed in U.S. Prov. Pat. App. No. 63 / 378,017, filed September 30, 2022, the disclosure of which is incorporated by reference herein in its entirety, and / or any of the features as discussed in U.S. Prov. Pat. App. No. 63 / 377,983, filed September 30, 2022, the disclosure of which is incorporated by reference herein in its entirety. Any of thefeatures of the present disclosure may be used in combination with any of the features as discussed in U.S. Pat. Pub. No. 2023 / 0131269, published on April 26, 2023, the disclosure of which is incorporated by reference herein in its entirety.

[0190] Further, the present disclosure and / or one or more components of devices, systems, and storage mediums, and / or methods, thereof also may be used in conjunction with continuum robotic systems and catheters, such as, but not limited to, those described in U.S. Patent Publication Nos. 2019 / 0105468; 2021 / 0369085; 2020 / 0375682; 2021 / 0121162; 2021 / 0121051; and 2022-0040450, each of which patents and / or patent publications are incorporated by reference herein in their entireties.

[0191] The present disclosure and / or one or more components of devices, systems, and storage mediums, and / or methods, thereof also may be used in conjunction with autonomous robot devices, systems, methods, and / or storage mediums and / or with endoscope devices, systems, methods, and / or storage mediums. Such continuum robot devices, systems, methods, and / or storage mediums are disclosed in at least: U.S. Prov. Pat. App. 63 / 497, 358, filed on April 20, 2023, which is incorporated by reference herein in its entirety.

[0192] While the present disclosure has been described with reference to exemplary embodiments, it is to be understood that the disclosure is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

Claims

CLAIMS1. An autonomous navigation robot system, comprising: a steerable catheter; one or more actuators to steer and drive the steerable catheter; a user input device, a display; and a processor with one or more memory storing instructions configured to execute the stored instructions through the actuators, wherein the stored instructions include at least two driving modes for the robot system, including an autonomous driving mode and a manual driving mode, wherein the at least two driving modes are selected based on a force applied to the actuator of the robotic system during steering and driving of the robotic system through a certain path, or wherein the at least two driving modes are selected based on a time spent to complete steering and driving of the robotic system through a certain path.

2. The autonomous navigation robot system of Claim 1, wherein the driving mode of the robotic system is changed from the autonomous driving mode to the manual driving mode in a case where a time spent to complete steering and driving of the robotic system through a certain path is more than a predetermined time period.

3. The autonomous navigation robot system of Claim 1, wherein the driving mode of the robotic system is changed from the autonomous driving mode to manual driving mode in a case where the detected force during steering and of the robotic system driving through a certain path is more than a predetermined force.4- The autonomous navigation robot system of Claim 1, wherein the manual mode includes at least one of a stop mode, a manual driving mode or another autonomous driving mode.

5. The autonomous navigation robot system of Claim 1, wherein the robotic system is inserted into an object by an advance of a stage attached to the robotic system, and wherein the certain path is a path that the robotic system passes in accordance with an advance of the stage for a predetermined distance.

6. The autonomous navigation robot system of Claim 1 wherein the certain path is determined based on a position of at least three consecutive branching points.

7. The autonomous navigation robot system of Claim 6 wherein a start point of the certain path is between a first branching point and a second branching point and an end point of the certain path is between the second branching point and the third branching point.

8. The autonomous navigation robot system of Claim 6 wherein a start point of the certain path is a midpoint between a first branching point and a second branching point and an end point of the certain path is a midpoint between the second branching point and the third branching point.

9. The autonomous navigation robot system of Claim 2 wherein the driving mode of the robotic tool is changed from the autonomous driving mode to a manual driving mode in a case where the time spent to complete bending of the robotic tool during the driving through the certain path is more than the predetermined period.

10. The autonomous navigation robot system of Claim 2 wherein the driving mode of the robotic tool is changed from a first autonomous driving mode to asecond autonomous driving mode which is different from the first autonomous driving mode in a case where the time spent to complete bending of the robotic tool during the driving through the certain path is more than the predetermined period.

11. The autonomous navigation robot system of Claim 2 wherein the predetermined period is equal to or more than 2.3 seconds and equal to or less than 6.03 seconds.

12. The autonomous navigation robot system of Claim 1, wherein the driving mode of the robotic tool is changed from the autonomous driving mode to the other driving mode in a case where the force or the time exceeds a threshold, wherein the threshold is defined as:(75th percentile)+i.5* interquartile range of a box-and-whisker plot for a dataset representing the force applied to the driving wire of the robotic tool during the driving through the certain path or the time spent to complete bending of the robotic tool during the driving through the certain path.

13. The autonomous navigation robot system of Claim 1, wherein the driving mode of the robotic tool is changed from the autonomous driving mode to the other driving mode in a case where the force or the time exceeds a threshold, wherein the threshold is defined as: the slope of regression line + 3*average of all measured data points of a dataset representing the force applied to the driving wire of the robotic tool during the driving through the certain path or the time spent to complete bending of the robotic tool during the driving through the certain path.

14. The autonomous navigation robot system of Claim 1, further comprising a storage that stores the force applied to a driving wire of the robotic tool during adriving through a certain path or the time spent to complete bending of the robotic tool during the driving through the certain path; wherein the driving mode of the robotic tool is changed from the autonomous driving mode to the other driving mode in a case where the force or the time exceeds a threshold, and wherein the threshold is updated based on the stored force or the stored time.

15. The autonomous navigation robot system of Claim 1, wherein the one or more processors further perform: obtaining, based on the at least one model, an index indicating a difficulty of an operation of a robotic tool in the object or a probability of success of an operation of the robotic tool in the object; outputting, based on the index, information about a selection of one of tools, the tools including a manual tool steered manually or the robotic tool steered robotically.

16. The autonomous navigation robot system of Claim 1, wherein the one or more processors further configured to execute the stored instructions to perform: obtaining an index indicating a difficulty of an operation of a robotic tool in the object or a probability of success of an operation of the robotic tool in the object for at least a part of the path of the robotic tool; and determining, based on the index corresponding to a part of the path, a steering mode of the robotic tool for the path, from among steering modes including a manual steering mode and one or more autonomous steering mode.

17. The autonomous navigation robot system of Claim 16, wherein the one or more processors further configured to execute the stored instructions to perform:switching, based on the index corresponding to a part of the path, a steering mode of the robotic tool for the path, from a first autonomous mode to a second autonomous mode which is different from the first autonomous mode.

18. The autonomous navigation robot system of Claim 1, wherein the robotic tool is a robotic catheter.

19. A non-transitory computer readable storage medium storing instructions executed by one or more processors to perform: obtaining at least one model of an object; controlling a robotic system to move through the object based on the at least one model of the object; wherein a driving mode of the robotic system is changed from an autonomous driving mode to a manual driving mode based on a force applied to a driving wire of the robotic system during a driving through a certain path or a time spent to complete bending of the robotic system during the driving through the certain path.

20. The autonomous navigation robot system of Claim 1, wherein: one or more memories storing instructions; and one or more processors configured to execute the stored instructions to perform: obtaining at least one model of an object; obtaining, based on the at least one model, an index indicating a difficulty of an operation of a robotic system in the object or a probability of success of an operation of the robotic system in the object; outputting, based on the index, information about a selection of one of tools, the tools including a manual tool steered manually or the robotic tool steered robotically.

Images