Device for autonomous lithotripsy and methods for displaying corrective actions therefor

Abstract

Disclosed are a device and a method of autonomous stone ablation during laser lithotripsy, the method including inserting a catheter into a lumen; navigating the catheter through the lumen along an insertion trajectory; obtaining at least one image of an object within the lumen; segmenting the at least one image; determining a size of the object; and in response to the size exceeding or being equal to a predetermined size: defining a region of lasing of the object, aligning a tip of the catheter with the region of lasing, and performing lithotripsy.

Description

DEVICE FOR AUTONOMOUS LITHOTRIPSY AND METHODS FOR DISPLAYING CORRECTIVE ACTIONS THEREFOR BACKGROUND Priority

[0001] The present application claims the benefit of and priority to U.S. Provisional Application No. 63 / 561,996, which was filed with the U.S. Patent and Trademark Office on March 6, 2024, the entire contents of which is incorporated herein by reference. Field of Disclosure

[0002] The present disclosure relates to medical devices. More particularly, the disclosure is directed to autonomous stone ablation performed during lithotripsy. Description of Related Art

[0003] Endoscopy is a medical procedure allowing remote inspection, manipulation, and / or treatment of internal organs using flexile devices inserted through natural orifices. The complexity of anatomical pathways in a patient, the limited flexibility of existing instruments, and the need to use multiple instruments limit such procedures. Endoscopes are generally composed of a passive proximal section and an active distal section. The proximal passive section can be rigid, semi-rigid, or flexible. The active distal section includes a steerable tip that is manually controlled or remotely actuated by control wires connected to actuation wheels located on the handle of the device. Endoscopes are typically equipped with an imaging device (a camera), a light source, irrigation and / or suction channels, and at least one instrument channel for passing interventional tools. Typical endoscopic instruments have at least three degrees of freedom (3DOF) which allow for insertion, rotation and grasping operations. However, confined workspaces and limited visibility of target organs further limit the usability of these systems and require one or more highly skilled endoscopists to perform dexterous tasks. For example, four hands are required to manually control the endoscope and its instruments during a simple procedure. To that end, an endoscopist needs to master a combination of accurate tip angulations, shaft management, and instrument insertion procedures, while communicating with an endoscopist assistant to actuate the instrument and hold the endoscopic shaft in a correct position. To alleviate the complexity of such procedures, several innovative techniques have been proposed. In particular, robotic endoscope systems have been developed.

[0004] During a manual ureteroscope procedure, a physician may insert a ureteroscope and guide the ureteroscope to a position identified as being located near a urinary stone. US2021 / 0298590 to Ayvali discusses an application that records urinary stone locations, identifies target papilla, and records position(s) of target papilla. US 2004 / 0249267 to Gilboa discusses a system to facilitate navigation to a target within a branched structure, e.g., the bronchial tree, to guide a medical tool to the target.

[0005] Ureteroscopy for transurethral lithotripsy may be performed with a robotic ureteroscope or continuum robot that includes a bending section with flexible body. A physician may control the robotic ureteroscope by control via a joystick of the bending section to navigate through the urinary system. A type of continuum robot is discussed by US 11,685,046 to Takagi et al. Gauhar, V., et al., Robotic Retrograde Intrarenal Surgery: A Journey from “Back to the Future”, Journal of Clinical Medicine (Vol. 11, Issue 18) 2022, discusses robotic platforms available for flexible ureteroscopy, including that a robotic device with a manual ureteroscope for recording and replaying input.

[0006] Kidney stones are characterized as mineral deposits found in the renal pelvis and calyces. They can affect around 5% of women and 12% of men in America, can cause nausea, vomiting, pain, and hematuria, and the patient might end up with infection. Diagnosis of kidney stones is based on radiological imaging and treatment can vary from mild with the use of analgesics or antibiotics, in case infection occurs, to more intense like lithotripsy. Lithotripsy is the physical destruction of kidney stones and is a more permanent treatment. Depending on the stone size, different lithotripsy techniques have been developed. Extracorporeal shockwave therapy (ESWT) is a non-invasive lithotripsy method in which shock waves generated outside the body are focused upon the stone at a rate of one or per second. The stone is transformed into fragments which are small enough to pass through the urethra. ESWT lithotripsy can be applied to kidney stones with size less than 2 cm while for stones greater than 2 cm size laser lithotripsy was introduced.

[0007] Laser lithotripsy is a minimally invasive procedure typically performed by a urologist for treatment of a urinary tract stone by fragmentation. The ureteroscope is passed through the urethra and bladder all the way up to the ureter at the point where the stone is located. Laser lithotripsy may use a ureteroscope, which includes a flexible laser fiber and a camera, which can live show images. The camera is used to visually target the stone and the laser is used to physically break the stones into fragments. The fragments can then be removed by using a basket-like instrument. Laser lithotripsy has several advantages such as: reduced recovery time, effectiveness and versatility since it can be applied to multiple types of stones. On the other hand, disadvantages may include difficulty in retrieving fragments and the risk of complications during the procedure. Those risks involve the injury of thesurrounding tissues, infections, or even damage to the ureter.

[0008] A factor that can affect the clinical complications and prolong operation time is the retro-pulsive movement of the stone during laser ablation. Reduced stone movement and retropulsion provide more efficient the treatment since there is no increased need to chase the stones. Additionally, slow movement of the stone reduces the chance of traumatizing the kidney tissue since the operator can follow the target easier. However, when the stone movement is fast and when the user is inexperienced, risk of trauma may increase.

[0009] U.S. Pub. 2021 / 0177444 to Shelton et al. discusses a lithotripsy apparatus that includes a lithotripsy wave guide shaft configured to transmit energy to a urinary tract stone, in which a lithotripter collects signal data, provides feedback to a user, and determines if the lithotripsy wave guide shaft is in contact with a tissue; if the lithotripsy wave guide shaft is in contact with a stone; a type of stone; if a user is applying force in excess of a predetermined threshold; and physical characteristics of a stone. In Shelton, the laser must touch the tissue. Also, Shelton does not recognize when the tissue is being traumatized.

[0010] WO 2022 / 256632 Al to Cheng et al. discusses a ureteroscope that includes an elongate flexible shaft; a camera at a distal end of the shaft; and an image processing module coupled with the ureteroscope that includes a console with a processor for receiving imaging data including a first image that includes a plurality of objects including a first subset of the plurality of objects that obstructs visibility of one or more objects of a second subset of the plurality of objects. The processor generates a second image including the second subset of the plurality of objects visibly unobstructed, and renders of the first image or the second image on a display. Although Cheng helps to not apply the laser to small fragments covering a larger fragment, Cheng does not prevent traumatizing surrounding tissue.

[0011] S. Gupta, et al., Multi-class motion-based semantic segmentation for ureteroscopy and laser lithotripsy, discusses automated segmentation of kidney stones and the laser fiber for performing automated quantitative analysis, particularly stone-size estimation, that a surgeon can use to decide if a stone requires further fragmentation. Stone detection is a crucial step of this autonomous process and involves the automatic segmentation thereof. Gupta discusses factors such as turbid fluid inside the cavity, specularities, motion blur due to kidney movements and camera motion, bleeding, and stone debris as impacting the quality of vision within the kidney, that may lead to extended operative times. Gupta shows the stone in an ureteroscopic video in order to perform accurate measurements. However, Gupta does not segment an image during lithotripsy. Nor does Gupta use the segmentation result to guide the ureteroscope.SUMMARY

[0012] To overcome the shortcomings of conventional systems, disclosed is a device for autonomous lithotripsy and methods for automated stone targeting during lithotripsy, to minimize the risk of clinical complications.

[0013] An aspect of the present disclosure provides a method for performing laser lithotripsy, with the method including inserting a catheter into a lumen; navigating the catheter through the lumen along an insertion trajectory; obtaining at least one image of one or more objects within the lumen; segmenting the at least one image; detecting at least one object of the one or more objects having a size exceeding a threshold as at least one target object; and defining a region of lasing (ROL) of the at least one target object, aligning a tip of the catheter with the ROL, and performing lithotripsy.

[0014] A further aspect of the present disclosure provides a method for performing lithotripsy, the method including inserting a catheter into a lumen; navigating the catheter through the lumen along an insertion trajectory; obtaining at least one image of one or more objects within the lumen; segmenting the at least one image; identifying at least two targets among the segmented at least one image; identifying at least one parameter of the targets; prioritizing the targets based on at least one parameter of each target; defining an ROL for a highest priority target; aligning a tip of the catheter with the ROL; and performing lithotripsy.

[0015] Yet another aspect of the present disclosure provides an information processing apparatus to control a steerable catheter, the information processing apparatus including at least one memory configured to store instructions; and at least one processor configured to execute the stored instructions to cause the steerable catheter to: obtain at least one image of one or more objects within a lumen; segment the at least one image; detect at least one object of the one or mor objects having a size exceeding a threshold as at least one target object; and define an ROL of the at least one target object, align a catheter tip with the ROL, activate a laser on a distal end of the steerable catheter, and perform lithotripsy.

[0016] A further aspect of the present disclosure provides an autonomous navigation robot system that includes a steerable catheter; a camera at the distal end of the steerable catheter; one or more actuators configured to automatically move the steerable catheter; and a controller that is configured to: obtain, from the camera, at least one image of an object within a lumen; segment the at least one image; determine a size of the object; and in response to the size of the object exceeding or being equal to a predetermined size: detect an ROL of the object, align the catheter tip with the ROL, activate a laser on a distal end of the steerable catheter, and perform lithotripsy.

[0017] The various embodiments disclosed in the present disclosure provide several advantages over conventional methods and systems, including automatic calculation of the movement of the ureteroscope and reduction of potential, as well as reduction of unintentional trauma potentially caused by the laser during lithotripsy.

[0018] It is to be understood that both the foregoing summary and the detailed description are exemplary and explanatory in nature and are intended to provide a complete understanding of the present disclosure without limiting the scope of the present disclosure. Additional objects, features, and advantages of the present disclosure will become apparent to those skilled in the art upon reading the following detailed description of exemplary embodiments, when taken in conjunction with the appended drawings, and provided claims. BRIEF DESCRIPTION OF DRAWINGS

[0019] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate various embodiments, objects, features, and advantages of the present disclosure.

[0020] FIG. 1 illustrates a simplified representation of a medical environment, such as an operating room, where a robotic catheter system can be used.

[0021] FIG.2 illustrates a functional block diagram of the robotic catheter system.

[0022] FIG.3 represents the catheter and bending thereof.

[0023] FIG.4 is a block diagram illustrating components of the robotic catheter system.

[0024] FIG.5 is a block diagram illustrating components of the system controller and / or the display controller.

[0025] FIG. 6 illustrates a method of performing autonomous lithotripsy according to an embodiment.

[0026] FIGS. 7A-7D illustrate kidney and stone phantoms for validating data for an autonomous stone ablation method according to an embodiment.

[0027] FIGS. 8A-8C are images derived from the recorded videos of extraction of a phantom stone according to an embodiment.

[0028] FIG. 9 is an image dataset used to train a U-net algorithm according to an embodiment.

[0029] FIG.10 illustrates a method of autonomous ablation that is performed upon reaching the stone according to an embodiment.

[0030] FIGS. 11A and 11B provide results of stone target evaluation according to an embodiment.

[0031] FIGS. 12A and 12B present application examples of the stone targeting procedureaccording to an embodiment.

[0032] FIG. 13 schematically presents autonomous movement of the catheter, according to an embodiment. DETAILED DESCRIPTION

[0033] Aspects of the present disclosure can be understood by reading the following detailed description in light of the accompanying figures. It is noted that, in accordance with standard practice, the various features of the drawings are not drawn to scale and do not represent actual components. Details such as dimensions of the various features may be arbitrarily increased or reduced for ease of illustration. In addition, reference numerals, labels and / or letters are repeated in the various examples to depict similar components and / or functionality. This repetition is for the purpose of simplicity and clarity and does not in itself limit the various embodiments and / or configurations the same components discussed.

[0034] Before the various embodiments are described in further detail, it shall be understood that the present disclosure is not limited to any particular embodiment. It is also to be understood that the terminology used herein is for the purpose of describing exemplary embodiments only, and is not intended to be limiting. Embodiments of the present disclosure may have many applications within the field of medical treatment or minimally invasive surgery (MIS).

[0035] Throughout the figures, the same reference numerals, and characters, unless otherwise stated, are used to denote like features, elements, components, or portions of the illustrated embodiments. In addition, while the subject disclosure is described in detail with reference to the enclosed figures, it is done so in connection with illustrative exemplary embodiments. It is intended that changes and modifications can be made to the described exemplary embodiments without departing from the true scope of the subject disclosure as defined by the appended claims. Although the drawings represent some possible configurations and approaches, the drawings are not necessarily to scale and certain features may be exaggerated, removed, or partially sectioned to better illustrate and explain certain aspects of the present disclosure. The descriptions set forth herein are not intended to be exhaustive or otherwise limit or restrict the claims to the precise forms and configurations shown in the drawings and disclosed in the following detailed description.

[0036] Those skilled in the art will recognize that, in general, terms used herein, and especially in the appended claims (e.g., bodies of the appended claims) are generally intended as "open" terms (e.g., the term "including" should be interpreted as "including but not limited to," the term "having" should be interpreted as "having at least," the term "includes" shouldbe interpreted as "includes but is not limited to," etc.). It will be further understood by those within the art that if a specific number of an introduced claim recitation is intended, such an intent will be explicitly recited in the claim, and in the absence of such recitation no such intent is present. For example, as an aid to understanding, the following appended claims may contain usage of the introductory phrases "at least one" and "one or more" to introduce claim recitations. However, the use of such phrases should not be construed to imply that the introduction of a claim recitation by the indefinite articles "a" or "an" limits any particular claim containing such introduced claim recitation to claims containing only one such recitation, even when the same claim includes the introductory phrases "one or more" or "at least one" and indefinite articles such as "a" or "an" (e.g., "a" and / or "an" should typically be interpreted to mean "at least one" or "one or more"); the same holds true for the use of definite articles used to introduce claim recitations.

[0037] In addition, even if a specific number of an introduced claim recitation is explicitly recited, those skilled in the art will recognize that such recitation should typically be interpreted to mean at least the recited number (e.g., the bare recitation of "two recitations," without other modifiers, typically means at least two recitations, or two or more recitations). Furthermore, in those instances where a convention analogous to "at least one of A, B, and C, etc." is used, in general such a construction is intended in the sense one having skill in the art would understand the convention (e.g., "a system having at least one of A, B, and C" would include but not be limited to systems that have A alone, B alone, C alone, A and B together, A and C together, B and C together, and / or A, B, and C together, etc.). In those instances where a convention analogous to "at least one of A, B, or C, etc." is used, in general such a construction is intended in the sense one having skill in the art would understand the convention (e.g., "a system having at least one of A, B, or C" would include but not be limited to systems that have A alone, B alone, C alone, A and B together, A and C together, B and C together, and / or A, B, and C together, etc.). It will be further understood by those within the art that typically a disjunctive word and / or phrase presenting two or more alternative terms, whether in the description, claims, or drawings, should be understood to contemplate the possibilities of including one of the terms, either of the terms, or both terms unless context dictates otherwise. For example, the phrase "A or B" will be typically understood to include the possibilities of "A" or "B" or "A and B."

[0038] When a feature or element is herein referred to as being "on" another feature or element, it can be directly on the other feature or element or intervening features and / or elements may also be present. In contrast, when a feature or element is referred to as being"directly on" another feature or element, there are no intervening features or elements present. It will also be understood that, when a feature or element is referred to as being "connected", "attached", "coupled" or the like to another feature or element, it can be directly connected, attached or coupled to the other feature or element or intervening features or elements may be present. In contrast, when a feature or element is referred to as being "directly connected", "directly attached" or "directly coupled" to another feature or element, there are no intervening features or elements present. Although described or shown with respect to one embodiment, the features and elements so described or shown in one embodiment can apply to other embodiments. It will also be appreciated by those of skill in the art that references to a structure or feature that is disposed "adjacent" to another feature may have portions that overlap or underlie the adjacent feature.

[0039] The terms first, second, third, etc. may be used herein to describe various elements, components, regions, parts and / or sections. It should be understood that these elements, components, regions, parts and / or sections are not limited by these terms of designation. These terms of designation have been used only to distinguish one element, component, region, part, or section from another region, part, or section. Thus, a first element, component, region, part, or section discussed below could be termed a second element, component, region, part, or section merely for purposes of distinction but without limitation and without departing from structural or functional meaning.

[0040] As used herein, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should be further understood that the terms "includes" and / or "including", “comprises” and / or “comprising,” “consists” and / or “consisting” when used in the present specification and claims, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof not explicitly stated. Further, in the present disclosure, the transitional phrase “consisting of” excludes any element, step, or component not specified in the claim. It is further noted that some claims or some features of a claim may be drafted to exclude any optional element; such claims may use exclusive terminology as "solely," "only" and the like in connection with the recitation of claim elements, or it may use of a "negative" limitation.

[0041] The term “about” or “approximately” as used herein means, for example, within 10%, within 5%, or less. In some embodiments, the term “about” may mean within measurement error. In this regard, where described or claimed, all numbers may be read as ifprefaced by the word "about" or "approximately," even if the term does not expressly appear. The phrase "about" or "approximately" may be used when describing magnitude and / or position to indicate that the value and / or position described is within a reasonable expected range of values and / or positions. For example, a numeric value may have a value that is + / - 0.1% of the stated value (or range of values), + / -1% of the stated value (or range of values), + / -2% of the stated value (or range of values), + / -5% of the stated value (or range of values), + / -10% of the stated value (or range of values), etc. Any numerical range, if recited herein, is intended to be inclusive of end values and includes all sub-ranges subsumed therein, unless specifically stated otherwise. As used herein, the term “substantially” is meant to allow for deviations from the descriptor that do not negatively affect the intended purpose. For example, deviations that are from limitations in measurements, differences within manufacture tolerance, or variations of less than 5% can be considered within the scope of substantially the same. The specified descriptor can be an absolute value (e.g. substantially spherical, substantially perpendicular, substantially concentric, etc.) or a relative term (e.g. substantially similar, substantially the same, etc.).

[0042] Unless specifically stated otherwise, as apparent from the following disclosure, it is understood that, throughout the disclosure, discussions using terms such as "processing," "computing," "calculating," "determining," "displaying," or the like, refer to the action and processes of a computer system, or similar electronic computing device, or data processing device that manipulates and transforms data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission or display devices. Computer or electronic operations described in the specification or recited in the appended claims may generally be performed in any order, unless context dictates otherwise. Also, although various operational flow diagrams are presented in a sequence(s), it should be understood that the various operations may be performed in other orders than those which are illustrated or claimed, or operations may be performed concurrently. Examples of such alternate orderings may include overlapping, interleaved, interrupted, reordered, incremental, preparatory, supplemental, simultaneous, reverse, or other variant orderings, unless context dictates otherwise. Furthermore, terms like "responsive to," “in response to”, "related to," “based on”, or other like past-tense adjectives are generally not intended to exclude such variants, unless context dictates otherwise.

[0043] As used herein, the term “real-time” is meant to describe processes or eventscommunicated, shown, presented, etc. substantially at the same time as those processes or events actually occur. Real time refers to a level of computer responsiveness that a user senses as sufficiently immediate or that enables the computer to keep up with some external process. For example, in computer technology, the term real-time refers to the actual time during which something takes place and the computer may at least partly process the data in real time (as it comes in). As another example, in signal processing, “real-time” processing relates to a system in which input data is processed within milliseconds so that it is available virtually immediately as feedback, e.g., in a missile guidance, an airline booking system, or the stock market real-time quotes (RTQs).

[0044] The present disclosure generally relates to medical devices, and it exemplifies embodiments of an endoscope or catheter, and more particular to a steerable catheter controlled by a medical continuum robot (MCR). The embodiments of the endoscope or catheter and portions thereof are described in terms of their state in a three-dimensional space. As used herein, the term “position” refers to the location of an object or a portion of an object in a three-dimensional space (e.g., three degrees of translational freedom along Cartesian X, Y, Z coordinates); the term “orientation” refers to the rotational placement of an object or a portion of an object (three degrees of rotational freedom--e.g., roll, pitch, and yaw); the term “posture” refers to the position of an object or a portion of an object in at least one degree of translational freedom and to the orientation of that object or portion of object in at least one degree of rotational freedom (up to six total degrees of freedom); the term "shape" refers to a set of posture, positions, and / or orientations measured along the elongated body of the object.

[0045] As it is known in the field of medical devices, the terms “proximal” and “distal” are used with reference to the manipulation of an end of an instrument extending from the user to a surgical or diagnostic site. In this regard, the term “proximal” refers to the portion (e.g., a handle) of the instrument closer to the user, and the term “distal” refers to the portion (tip) of the instrument further away from the user and closer to a surgical or diagnostic site. It will be further appreciated that, for convenience and clarity, spatial terms such as "vertical", "horizontal", "up", and "down" may be used herein with respect to the drawings. However, surgical instruments are used in many orientations and positions, and these terms are not intended to be limiting and / or absolute. In that regard, all directional references (e.g., upper, lower, upward, downward, left, tight, leftward, rightward, top, bottom, above, below, vertical, horizontal, clockwise, and counterclockwise) are only used for identification purposes to aid the reader's understanding of the present disclosure, and do not create limitations, particularly as to the position, orientation, or use of the disclosure.

[0046] As used herein the term “catheter” generally refers to a flexible and thin tubular instrument made of medical grade material designed to be inserted through a narrow opening into an anatomical bodily lumen (e.g., an airway or a vessel) to perform a broad range of medical functions. The more specific term “steerable catheter” refers to a medical instrument comprising an elongated flexible shaft having at least one tool channel spanning through a plurality of bendable segments that are actuated by an actuator that applies an actuation force via drive wires arranged along a wall of the shaft.

[0047] As used herein the term “endoscope” refers to a rigid or flexible medical instrument which uses light guided by an optical probe to look inside a body cavity or organ. A medical procedure, in which an endoscope is inserted through a natural opening, is called an endoscopy. Specialized endoscopes are generally named for how or where the endoscope is intended to be used, such as the bronchoscope (mouth), sigmoidoscope (rectum), cystoscope (bladder), nephroscope (kidney), bronchoscope (bronchi), laryngoscope (larynx), otoscope (ear), arthroscope (joint), laparoscope (abdomen), and gastrointestinal endoscopes.

[0048] In the present disclosure, the terms “optical fiber”, “fiber optic”, or simply “fiber” refers to an elongated, flexible, light conducting waveguide capable of conducting light from one end to another end due to the effect known as total internal reflection. The terms “light guiding component” or “waveguide” may also refer to, or may have the functionality of, an optical fiber. The term “fiber” may refer to one or more light conducting fibers. <Robotic Catheter System>

[0049] An embodiment of a robotic catheter system 100 is described in reference to FIG. 1 through FIG. 4. FIG. 1 illustrates a simplified representation of a medical environment, such as an operating room, where a robotic catheter system 100 can be used. FIG. 2 illustrates a functional block diagram of the robotic catheter system 100. FIG. 3 represents the catheter and bending thereof. FIG. 4 illustrate a logical block diagram of the robotic catheter system 100.

[0050] In this example, the system 100 includes a system console 102 operatively connected to a steerable catheter / ureteroscope 104 via a robotic platform 106. The robotic platform 106 includes one or more than one robotic arm 108 and a linear translation stage 110.

[0051] As illustrated in FIG. 1, a user 112 (e.g., a physician) controls the robotic catheter system 100 via a user interface unit (operation unit) to perform an intraluminal procedure on a patient 114 positioned on an operating table 116. The user interface may include at least one of a main display 118 (a first user interface unit), a secondary display 120 (a second user interface unit), and a handheld controller 124 (a third user interface unit). The main display118 may include, for example, a large display screen attached to the system console 102 or mounted on a wall of the operating room and may be, for example, designed as part of the robotic catheter system 100 or be part of the operating room equipment. Optionally, there is a secondary display 120 that is a compact (portable) display device configured to be removably attached to the robotic platform 106. Examples of the secondary display 120 include a portable tablet computer or a mobile communication device (a cellphone).

[0052] The steerable catheter 104 is actuated via an actuator unit 122. The actuator unit 122 is removably attached to the linear translation stage 110 of the robotic platform 106. The handheld controller 124 may include a gamepad-like controller with a joystick having shift levers and / or push buttons. It may be a one-handed controller or a two-handed controller. In one embodiment, the actuator unit 122 is enclosed in a housing having a shape of a catheter handle. One or more access ports 126 are provided in or around the catheter handle. The access port 126 is used for inserting and / or withdrawing end effector tools and / or fluids when performing an interventional procedure of the patient 114.

[0053] The system console 102 includes a system controller 128, a display controller 130, and the main display 118. The main display 118 may include a conventional display device such as a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, a quantum dot light-emitting diode (QLED) display or the like. The main display 118 provides a graphic interface unit (GUI) configured to display one or more views. These views may include live view image 132, an intraoperative image 134, and a preoperative image 136, and other procedural information 138. Other views that may be displayed include a model view, a navigational information view, and / or a composite view. The live image view 132 may be an image from a camera at the tip of the catheter. This view may also include, for example, information about the perception and navigation of the catheter 104. The preoperative image 136 may include pre-acquired three dimensional (3D) or two dimensional (2D) medical images of the patient 114 acquired by conventional imaging modalities such as computer tomography (CT), magnetic resonance imaging (MRI), or ultrasound imaging. The intraoperative image 134 may include images used for image guided procedure such images may be acquired by fluoroscopy or CT imaging modalities. Intraoperative image 134 may be augmented, combined, or correlated with information obtained from a sensor, camera image, or catheter data.

[0054] In the various embodiments where a catheter tip tracking sensor 140 is used, the sensor may be located at the distal end of the catheter. The catheter tip tracking sensor 140 may be, for example, an electromagnetic (EM) sensor. If an EM sensor is used, a catheter tipposition detector 142 may be included in the robotic catheter system 100; this catheter tip position detector 142 would include an EM field generator operatively connected to the system controller 128. Suitable electromagnetic sensors for use with a steerable catheter are well-known and described, for example, in U.S.6,201,387 and WO 2020 / 194212 A1.

[0055] Similar to FIG. 1, the diagram of FIG. 2 illustrates the robotic catheter system 100 includes the system controller 128 operatively connected to the display controller 130, which is connected to the display unit 118, and to the hand-held control 124. The system controller 128 is also connected to the actuator unit 122 via the robotic platform 106, which includes the linear translation stage 110. The actuator unit 122 includes a plurality of motors that control a plurality of drive wires 160. These drive wires travel through the steerable catheter 104. One or more access ports 126 may be located on the catheter. The catheter includes a proximal section 148 located between the actuator and the proximal bending section 152 where they actuate the proximal bending section. Three of the six drive wires 160 continue through the distal bending section 156 where they actuate this section and allow for a range of movement. As shown, at least two bendable sections (152 and 156) may be provided. Other embodiments as described herein can have three bendable sections. In some embodiments, a single bending section may be provided, or alternatively, four or more bendable sections may be present in the catheter.

[0056] FIG. 3 shows an exemplary embodiment of a steerable catheter 104. The steerable catheter 104 includes a non-steerable proximal section 148, a steerable distal section, and a catheter tip 158. The proximal section 148 and distal bendable section (including 152, 154 and 156) are joined to each other by a plurality of drive wires 160 arranged along the wall of the catheter. The proximal section 148 is configured with thru-holes or grooves or conduits to pass drive wires 160 from the distal section to the actuator unit 122. The distal section is comprised of a plurality of bending segments including at least a distal segment 156, a middle segment 154, and a proximal segment 152 to form a multi-section catheter. Each bending segment is bent by actuation of at least some of the plurality of drive wires 160 (driving members). The posture of the catheter may be supported by non-illustrated supporting wires (support members) also arranged along the wall of the catheter (see US 2021 / 0308423). The proximal ends of drive wires 160 are connected to individual actuators or motors of the actuator unit 122, while the distal ends of the drive wires 160 are selectively anchored to anchor members in the different bending segments of the distal bendable section.

[0057] Each bending segment is formed by a plurality of ring-shaped components (rings) with thru-holes, grooves, or conduits along the wall of the rings. The ring-shaped componentsare defined as wire-guiding members 162 or anchor members 164 depending on their function within the catheter. Anchor members 164 are ring-shaped components onto which the distal end of one or more drive wires 160 are attached. Wire-guiding members 162 are ring-shaped components through which some drive wires 160 slide through (without being attached thereto).

[0058] Detail “A” in FIG. 3 illustrates an exemplary embodiment of a ring-shaped component (a wire-guiding member 162 or an anchor member 164). Each ring-shaped component includes a central opening which forms the tool channel 168, and plural conduits 166 (grooves, sub-channels, or thru-holes) arranged lengthwise equidistant from the central opening along the annular wall of each ring-shaped component. Inside the ring-shaped component, an inner cover such as is described in US 2021 / 0369085 and US 2022 / 0126060, may be included to provide a smooth inner channel and provide protection. The non-steerable proximal section 148 is a flexible tubular shaft and can be made of extruded polymer material. The tubular shaft of the proximal section 148 also has a central opening or tool channel 168 and plural conduits 166 along the wall of the shaft surrounding the tool channel 168. An outer sheath may cover the tubular shaft and a steerable section. In this manner, at least one tool channel 168 formed inside the steerable catheter 104 provides passage for an imaging device and / or end effector tools from the access port 126 to the distal end of the steerable catheter 104.

[0059] The actuator unit 122 includes one or more servo motors or piezoelectric actuators. The actuator unit 122 bends one or more of the bending segments of the catheter by applying a pushing and / or pulling force to the drive wires 160. As shown in FIG. 3, each of the three bendable segments of the steerable catheter 104 has a plurality of drive wires 160. If each bendable segment is actuated by three drive wires 160, the steerable catheter 104 may have nine driving wires arranged along the wall of the catheter. Each bendable segment of the catheter is bent by the actuator unit 122 by pushing or pulling at least one of these nine drive wires 160. Force is applied to each individual drive wire in order to manipulate / steer the catheter to a desired pose. The actuator unit 122 assembled with steerable catheter 104 is mounted on the linear translation stage 110. Linear translation stage 110 includes a slider and a linear motor. In other words, the linear translation stage 110 is motorized, and can be controlled by the system controller 128 to insert and remove the steerable catheter 104 to / from the patient’s bodily lumen.

[0060] In some embodiments, the catheter as described herein is the steerable multi-section catheter described in one or more of U.S. Patents 11,007,641; 11,051,892; 11,096,552;11,278,366; 11,559,190; 11,622,828; 11,730,551; and 11,786,106; U.S. Publications 2019 / 0105468; 2021 / 0121162; 2021 / 0308423; 2021 / 0362323; 2021 / 0369085; 2021 / 0386972; 2022 / 0039635; 2022 / 0126060; 2022 / 0202277; 2023 / 0016761; and 2023 / 0083702; and PCT Publications WO / 2023 / 150517; WO / 2023 / 150761; WO / 2023 / 147414; WO / 2023 / 154352; WO / 2023 / 154825; WO / 2023 / 164047; WO / 2023 / 164275; and WO / 2022 / 032162, each of which are herein incorporated by reference in their entirety.

[0061] An imaging device 170 that can be inserted through the tool channel 168 includes an endoscope camera (videoscope) along with illumination optics (e.g., optical fibers or LEDs). The illumination optics provides light to irradiate the lumen and / or a lesion target which is a region of interest within the patient 114. End effector tools refer endoscopic surgical tools including clamps, graspers, scissors, staplers, ablation or biopsy needles, and other similar tools, which serve to manipulate body parts (organs or tumorous tissue) during examination or surgery. The imaging device 170 may be what is commonly known as a chip-on-tip camera and may be color or black-and-white.

[0062] A laser may be inserted through the tool channel 168.

[0063] In some embodiments, a tracking sensor 140 (e.g., an EM tracking sensor) is attached to the catheter tip 158. In this embodiment, steerable catheter 104 and the tracking sensor 140 can be tracked by the tip position detector. Specifically, the tip position detector detects a position of the tracking sensor 140, and outputs the detected positional information to the system controller 100. The system controller 128, receives the positional information from the tip position detector, and continuously records and displays the position of the steerable catheter 104 with respect to the patient’s coordinate system. The system controller 128 controls the actuator unit 122 and the linear translation stage 110 in accordance with the manipulation commands input by the user 112 via one or more of the user interface units (the handheld controller 124, a GUI at the main display 118 or touchscreen buttons at the secondary display 120).

[0064] FIG.4 is a block diagram illustrating components of the robotic catheter system 100.

[0065] By use of the components shown in FIG. 4, the system controller 128 may execute software programs and controls the display controller 130 to display a navigation screen (e.g., a live view image 132) on the main display 118 and / or the secondary display 120. The display controller 130 may include a graphics processing unit (GPU) or a video display controller (VDC).

[0066] FIG. 5 is a block diagram illustrating components of the system controller 128 and / or the display controller 130. The system controller 128 and the display controller 130may be configured separately. Alternatively, the system controller 128 and the display controller 130 can be configured as one device. In either case, the system controller 128 and the display controller 130 comprise substantially the same components. Specifically, the system controller 128 and display controller 130 may include a central processing unit (CPU) 182 comprised of one or more processors (microprocessors), a random access memory (RAM) 184 module, an input / output (I / O) 186 interface, a read only memory (ROM) 180, and data storage memory (e.g., a hard disk drive (HDD) 188 or solid state drive (SSD))).

[0067] The ROM 180 and / or HDD 188 store the operating system (OS) software, and software programs necessary for executing the functions of the robotic catheter system 100 as a whole. The RAM 184 is used as a workspace memory. The CPU 182 executes the software programs developed in the RAM 184. The I / O 186 inputs, for example, positional information to the display controller 130, and outputs information for displaying the navigation screen to the one or more displays (main display 118 and / or secondary display 120). The navigation screen may be a GUI generated by a software program but, it may also be generated by firmware, or a combination of software and firmware.

[0068] The system controller 128 may control the steerable catheter 104 based on any known kinematic algorithms applicable to continuum or snake-like catheter robots. For example, the system controller controls the steerable catheter 104 based on an algorithm known as follow the leader (FTL) algorithm. By applying the FTL algorithm, the most distal segment 156 of the steerable section is actively controlled with forward kinematic values, while the middle segment 154 and the proximal segment 152 (following sections) of the steerable catheter 104 move at a first position in the same way as the distal section moved at the first position or a second position near the first position.

[0069] The display controller 130 acquires position information of the steerable catheter 104 from system controller 102. Alternatively, the display controller 130 may acquire the position information directly from the tip position detector. The steerable catheter 104 may be a single-use or limited-use catheter device. In other words, the steerable catheter 104 can be attachable to, and detachable from, the actuator unit 122 to be disposable.

[0070] During a procedure, the display controller 130 may generate and outputs a live-view image or other view(s) or a navigation screen to the main display 118 and / or the secondary display 120. This view can optionally be registered with a 3D model of a patient’s anatomy (a branching structure) and the position information of at least a portion of the catheter (e.g., position of the catheter tip 158) by executing pre-programmed software routines. Upon completing navigation to a desired target, one or more end effector tools can be insertedthrough the access port 126 at the proximal end of the catheter, and such tools can be guided through the tool channel 168 of the catheter body to perform an intraluminal procedure from the distal end of the catheter.

[0071] The tool may be a medical tool such as an endoscope camera, forceps, a needle or other biopsy or ablation tools. In one embodiment, the tool may be described as an operation tool or working tool. The working tool is inserted or removed through the working tool access port 126. In the embodiments below, an embodiment of using a steerable catheter to guide a tool to a target is explained. The tool may include an endoscope camera or an end effector tool, which can be guided through a steerable catheter under the same principles. In a procedure there is usually a planning procedure, a registration procedure, a targeting procedure, and an operation procedure.

[0072] Use of a laser as an operation tool involves risk of unintentional trauma. Such risk may be caused by the laser when the stones are out of target due to retro-pulsive movement of the stones.

[0073] To minimize clinical complications, an autonomous stone ablation method that is performed during lithotripsy is provided. As discussed herein, the method segments the stone during lithotripsy in order to autonomously guide the catheter / ureteroscope 104. The method is designed and validated using a phantom specifically fabricated for investigating feasibility of autonomous stone ablation. <Autonomous Stone Ablation>

[0074] FIG. 6 illustrates a method of performing autonomous lithotripsy according to an embodiment.

[0075] As illustrated in Fig.6, in Step S601 phantom data is input for U-net segmentation in Step S603. The phantom data is acquired by two different components a renal pelvis phantom illustrated in FIGS.7A and 7C and a stone phantom illustrated in FIGS.7B and 7D.

[0076] In Step S605 of FIG. 6, data is received from 3D Slicer software and, in Step S607, data is received from the robotic system. The 3D Slicer software is described by A. Fedorov, et al., 3D Slicer as an image computing platform for the Quantitative Imaging Network. Magnetic Resonance Imaging 30(9), 1323-1341 (112012).

[0077] Data received in Steps S605 and S607 are integrated in Step S609 and is output for performing autonomous lithotripsy in Step S611. As described herein, a segmentation algorithm is trained using generated phantom data to identify stones. The segmentation algorithm is integrated with a training model with the 3D Slicer software and the above- described robotic system. As described herein, the autonomous stone targeting method wastested with live captured images.

[0078] FIGS. 7A-7D illustrate kidney and stone phantoms for validating data for an autonomous stone ablation method according to an embodiment.

[0079] FIGS. 7A and 7C illustrate a kidney phantom for data validation for the autonomous stone ablation method. The phantom is composed by two different components: a renal pelvis phantom and a stone phantom (FIGS. 7B and 7D). The pelvis phantom was created using a CT scan acquired under IRB approval protocol #2021P001848, in a Toshiba scanner at Brigham and Women’s Hospital based on a patient diagnosed with blood in the urine. The patient’s renal pelvis were manually delineated from the CT scan through 3D Slicer software’s modules of “Thresholding” and “Grow from Seeds.” See, Pinter, A, et al. “Polymorph segmentation representation for medical image computing”, Computer Methods and Programs in Biomedicine, Volume 171, p19-26, 2019.

[0080] The 3D volume of the renal pelvis was 3D printed using ABS plastic. The printed mold was then filled with a silicone rubber compound, and a patient-specific renal pelvis phantom was created.

[0081] Similar to renal pelvis phantom, the stone phantoms illustrated in FIGS. 7B and 7D were created using 3D slicer and 3D printing.3D volumes of various shapes were created and 3D printed using ABS plastic. Then, the plastic 3D prints were manually crafted to imitate the stone size length, e.g., approximately 1.5 mm to 2.5 mm, and were attached to copper wires using glue. The copper wire allowed maneuverability inside the silicone pelvis phantom. The renal pelvis phantom, the stone phantom and an application example of maneuverability of the silicone phantom are illustrated in FIG.7C.

[0082] FIGS. 8A-8C are images derived from the recorded videos of extraction of the phantom stone according to an embodiment. Based on at least one image obtained from the recorded videos, a stone identified in each frame was annotated by an expert to create a segmentation ground truth. A U-net, as illustrated in FIG. 9, was implemented and trained using the ground truth in order the stones to be automatically segmented in every frame of the phantom video. See, N. Siddique, et al., U-Net and Its Variants for Medical Image Segmentation: A Review of Theory and Applications, IEEE Access, vol. 9, pp. 82031-82057, 2021.

[0083] FIG. 9 is an image dataset used to train a U-net algorithm according to an embodiment. 9. Training of the stone segmentation algorithm may be performed before inserting the catheter tip into the bodily lumen in Step 1001 of FIG.10.

[0084] The U-net algorithm in FIG.9 is visualized using a Visualkeras Python package. TheU-net consists of a U-shape architecture which contracts and expands the convolutions. The left side of FIG. 9 is a contracting path and the right side of FIG. 9 is an expansive path. For the contracting path, a block of 4 repetitions were used: 3x3 convolution followed by a dropout and a second 3x3 convolution followed by a 2x2 maxpooling. Then, a 3x3 convolution followed by a dropout and a second 3x3 convolution which completed the contracting path. For the expansive path, 4 repetitions were used of: 2x2 2D transposed convolution followed by a concatenate layer, then a 3x3 convolution followed by a dropoout and a second 3x3 convolution.

[0085] Using a binary image of the ground truth and their corresponding RBG images, network was trained. During training, the energy function may be calculated using the following pixel oriented soft max: … (1) , where ac(i) is thenumber of classes.

[0086] The cross entropy function penalizes the deviation of sl(i) from 1 as: , where 1 is the groundas: (3) , whereis the distance to the second nearest.

[0087] FIG.10 illustrates a method of autonomous ablation that is performed upon reaching the stone according to an embodiment.

[0088] In Step S1001 autonomous lithotripsy may commence after insertion of a catheter tip into a bodily lumen and navigation of the catheter through the lumen along an insertion trajectory. In Step S1001 a laser that is configured to perform lithotripsy is turned off, i.e. is in a deactivated state.

[0089] In Step S1003, live imaging may be performed, and autonomous lithotripsy may be performed in Step S1005. In Step S1007 at least one image may be obtained from the live imaging by the imaging device (camera) 170 within the bodily lumen.

[0090] In Step S1009 segmentation may be performed of the at least one live image using a convolutional neural network (CNN). The at least one image may be segmented by fully convolutional network (FCN) segmentation or U-Net segmentation. In Step S1011 an object is detected in the at least one live image, a diameter or size of the object may be determined and a determination may be made of whether a maximum size of the object exceeds or is equal to a predetermined size, for example approximately 1.5 mm. The diameter of the object may be determined based on the segmented at least one image and segmenting the at least one image may be automated. The predetermined size may be set by the user prior to the lithotripsy or may be from a lookup table. In some embodiments, the predetermined size may be defined as the size of a kidney stone for which laser lithotripsy is indicated. In some embodiments, the predetermined size may take into account properties of the kidney stone such as stone density, stone shape, and Guy's stone score.

[0091] The size of the object may be determined based on at least one physical parameter of the object, with the physical parameter being one or more of a diameter, a radius, a circumference, or an area of the object.

[0092] If in Step S1013 the maximum size of the object is less than the predetermined size, the method returns to Step S1001, with the laser in the deactivated state.

[0093] If in Step S1013 the maximum size of the object exceeds or is equal to the predetermined size, a region of lasing (ROL) of the object, i.e., the kidney stone, is detected. The ROL may be a perimeter around a center of gravity of the segmented object and an overlapping area of previous detected stone objects. In Step S1015 a tip of the catheter may be aligned with the ROL.

[0094] When the laser is aligned with the ROL, a notification may be output to autonomously perform the lithotripsy, with the laser maintaining a minimum predefined distance from kidney tissue.

[0095] Upon detecting the ROL, the catheter tip automatically moves into alignment with the ROL. Then, in Step S1017 the laser is activated and lithotripsy is performed on the object. After irradiating the ROL of the stone, the laser may be deactivated.

[0096] Another image of the object may then be obtained, the another image may be segmented, and an updated size of the object may be determined. In response to the size of the object exceeding or being equal to the predetermined size, another ROL of the object may be detected, the catheter tip may be aligned with the another ROL, the laser may be activated, and automated lithotripsy may be performed.

[0097] To minimize clinical complications, stone ablation is performed during lithotripsy byautonomously guiding the ureteroscope during lithotripsy. During lithotripsy, the catheter 104 may be inserted to the kidney manually, robotically and / or autonomously. Navigation of the catheter 104 is performed to an identified stone. Upon reaching the stone, the imaging device (camera) 170 is used to physically view the stone and the user 112 may autonomously target the stone and enable the laser for ablation. After the autonomously targeting, the stone is then automatically segmented and the tip of the catheter 104 automatically targets the stone.

[0098] The U-net network was trained using 884 (1024-120 for testing) frames coming from three recorded videos using an OVM6946 camera and phantom data for components of the renal pelvis phantom, as described in FIGS. 7A and 7C. The network was trained on 500 epochs and dynamic augmentations were applied. In dynamic augmentations random transformations are applied (rotation, shifting, shearing, blurring, added noise etc.) to increase the training data size during the course of model training. At each epoch, a new randomly augmented dataset is created. Therefore, the size of training data is larger when compared to static augmentation (each epoch pulls the data from the same augmented dataset). Moreover, the model has less risk to overfit. See, GM, H., Mori, K., Verma, S., Athanasiou, L.: The influence of image cropping sizes on mammographic breast cancer classification using CNN. In: IEEE International Conference on Bioinformatics and Bioengineering (BIBE) (2023). With a goal to target the center of gravity of the stone, the U-net’s performance was evaluated in terms of stone target; the Euclidean distance difference between the center of gravity (COG) of the detected and annotated stones was calculated and then it was compared with the mean distance of each annotation. As mean distance of each annotation, the mean of the maximum distances of each perimeter point of the annotation is defined to all the other perimeter points of the same annotation. If the COG difference is greater than the mean distance of the annotation points, the catheter will not target the stone. The stone target results presented in FIGS. 11A and 11B illustrate 99.97% target accuracy and excellent area agreement.

[0099] FIGS. 11A and 11B provide results of stone target evaluation according to an embodiment. FIG. 11A illustrates Euclidean distance difference between the COG of the detected and annotated stones (triangles), and the mean of the maximum distances (circles) of each perimeter point of the annotation to all the other perimeter points of the same annotation. The out of target frames are highlighted with a perpendicular (vertical) line. FIG. 11B illustrates regression analysis of detected versus predicted-annotated stone percentage areas.

[0100] FIGS. 12A and 12B present application examples of the stone targeting procedure according to an embodiment.

[0101] FIG. 12A illustrates application examples of the out of target stones in frames 24, 25 and 77. FIG.12B illustrates application examples of three successfully targeted stones.

[0102] When a target or stone goes outside of a field of view (FOV), size information may be added and / or an icon, e.g., a triangle may be displayed at the edge of the display, with a number of displayed icons corresponding to a number of detected target that have moved outside the FOV.

[0103] FIG. 13 schematically presents autonomous movement of the catheter, according to an embodiment.

[0104] The upper region of FIG. 13 provides an application example of an autonomous robotic integrated system, with the left image showing the live image captured from the catheter’s camera, the center image being the live image with the binary U-net result overlaid, and the right image being binary U-net result. The center of gravity (red dot) of the binary U- net result which is the target is overlaid in all 3 images. The bottom of FIG. 13 illustrates a demonstration of the catheter autonomous movement, with the green arrow pointing to the stone phantom and the orange arrow pointing to the catheter tip, with the catheter autonomously following the stone.

[0105] The autonomous movement of the catheter is illustrated in FIG. 13 with four different images of a live demonstration being presented. For demonstration purposes, the phantom stone was placed in the entrance of the pelvis phantom since the catheter movement cannot be depicted inside the phantom since the material is not transparent.

[0106] One or more stone fragments are detected and one or more potential targets are identified, with small targets being ignored. The target identification may define the targets by prioritization by size, distance, and / or combination thereof. Tracing is performed and at least one target is broken into first generation fragments. The first generation fragments may be reprioritized and broking into second generation fragments. The defining is repeated for creation of each fragment generation.

[0107] Disclosed is a method for autonomous targeting of the kidney stones based on integration of a kidney phantom and a robotic system. The autonomous laser lithotripsy provides benefits that include reduced trauma that may occur during the procedure due to stone movement and inexperienced users.

[0108] The autonomous catheter operated to automatically followed the kidney stone during lithotripsy, based on accuracy of the U-net results. When the stone phantom touches the wall and the camera light reflects to the silicon renal pelvis phantom, some peripheral tissue parts may be detected as stones. However, such detection of some peripheral tissue parts as beingstones occurred in only three out of 120 frames, which is considered a successful targeting result, with, in a clinical scenario, manually controlling the level of autonomy may prevent detection of some peripheral tissue parts as being stones, and may be further help the user in cases where more than one stone is in the camera’s field of view.

[0109] Also provided is an information processing apparatus that controls a steerable catheter, with the information processing apparatus including at least one memory configured to store instructions and at least one processor configured to execute the stored instructions to cause the steerable catheter to obtain at least one image of an object within a lumen; segment the at least one image; determine a size of the object; and in response to the size of the object exceeding or being equal to a predetermined size: detect an ROL of the object, align the catheter tip with the ROL, activate a laser on a distal end of the steerable catheter, and perform lithotripsy.

[0110] In addition, provided is an autonomous navigation robot system that includes a steerable catheter, a camera at the distal end of the steerable catheter, one or more actuators configured to automatically move the steerable catheter, and a controller, which is configured to obtain, from the camera, at least one image of an object within a lumen; segment the at least one image; determine a size of the object; and in response to the size of the object exceeding or being equal to a predetermined size detect an ROL of the object, align the catheter tip with the ROL, activate a laser on a distal end of the steerable catheter, and perform lithotripsy.

[0111] At least certain aspects of the exemplary embodiments described herein can be realized by a computer of a system or apparatus that reads out and executes computer executable instructions (e.g., one or more programs or executable code) recorded on a memory such as an SSD or a storage medium (which may also be referred to as a non- transitory computer-readable storage medium) to perform functions of one or more block diagrams, systems, or flowchart described above.

[0112] The detector interface also provides communication interfaces to input and output devices. The detector may include, for example a photomultiplier tube (PMT), a photodiode, an avalanche photodiode detector (APD), a charge-coupled device (CCD), multi-pixel photon counters (MPPC), or other. Also, the function of detector may be realized by computer executable instructions (e.g., one or more programs) recorded on a storage / RAM.

[0113] In referring to the description, specific details are set forth in order to provide a thorough understanding of the examples disclosed. In other instances, well-known methods, procedures, components, and circuits have not been described in detail as not to unnecessarilylengthen the present disclosure.

[0114] Any patent, pre-grant patent publication, or other disclosure, in whole or in part, that is said to be incorporated by reference herein is incorporated only to the extent that the incorporated materials do not conflict with standard definitions or terms, or with statements and descriptions set forth in the present disclosure. As such, and to the extent necessary, the disclosure as explicitly set forth herein supersedes any conflicting material incorporated by reference.

[0115] Reference Numbers: 100 Robotic Catheter System 102 System Console188 Data Storage Memory HDD

[0116] US 11,007,641 US 11,051,892 US 11,096,552 US 11,278,366 US 11,559,190 US 11,622,828 US 11,730,551 US 11,786,106 US 2019 / 0105468 US 2021 / 0121162 US 2021 / 0362323 US 2021 / 0386972 US 2022 / 0039635 US 2022 / 0202277 US 2023 / 0016761 US 2023 / 0083702 WO / 2023 / 150517 WO / 2023 / 150761 WO / 2023 / 147414 WO / 2023 / 154352 WO / 2023 / 154825 WO / 2023 / 164047 WO / 2023 / 164275 WO / 2022 / 032162 US 6,201,387 to Govari US 2021 / 0308423 to Kincaid US 2021 / 0369085 to Kato US 2022 / 0126060 to Shia US 2021 / 0298590 to Ayvali US 2004 / 0249267 to Gilboa US 11,685,046 to Takagi et al.US 2021 / 0177444 to Shelton et al. US 2975785 to Sheldon US 3253524 to Ashizawa US 3610231 to Takahashi US 3788303 to Hall US 4207873 to Kruy US 5355871 to Hurley US 8657781 to Sewell US 9737373 to Schuh US 10470831 to Cohen US 20100041949 to Talkowshy US 20120078053 to Phee US 20190261830 to Banik WO 2020 / 194212 A1 to Izmirli et al. WO 2022 / 256632 Al to Cheng et al. Pierre Berthet-Rayne, et al., "The i2Snake Robotic Platform for Endoscopic Surgery", Ann Biomed Eng.2018 Oct;46(10):1663-1675, 2018; and Esther D. Rozeboom, et al., "Feasibility of joystick guided colonoscopy", J Robotic Surg.9:173-178, 2015. Gauhar, V., et al., Robotic Retrograde Intrarenal Surgery: A Journey from “Back to the Future”, Journal of Clinical Medicine (Vol.11, Issue 18), Available at https: / / doi.org / 10.3390 / jcm11185488. S. Gupta, et al., Multi-class motion-based semantic segmentation for ureteroscopy and laser lithotripsy. Computerized Medical Imaging and Graphics (2022), Available at: https: / / doi.org / 10.1016 / j.compmedimag.2022.102112 Dupourqué, L., Masaki, F., Colson, Y.L., Kato, T., Hata, N.: Transbronchial biopsy catheter enhanced by a multisection continuum robot with follow-the-leader motion. International journal of computer assisted radiology and surgery 14(11), 2021–2029 (11 2019). https: / / doi.org / 10.1007 / S11548-019-02017 - W, https: / / pubmed.ncbi.nlm.nih.gov / 31289997 / Cramer, J., Finet, J., Fillion-Robin, J.C., Pujol, S., Bauer, C., Jennings, D., Fennessy, F., Sonka, M., Buatti, J., Aylward, S., Miller, J.V., Pieper, S., Kikinis, R.: 3D Slicer as an image computing platform for the Quantitative Imaging Network. Magnetic Resonance Imaging 30(9), 1323–1341 (112012). https: / / doi.org / 10.1016 / J.MRI.2012.05.001N. L. Miller, A. P. Evan, and J. E. Lingeman, “Pathogenesis of Renal Calculi,” Urologic Clinics of North America, vol.34, no.3, pp.295–313, Aug.2007, doi: 10.1016 / J.UCL.2007.05.007. T. G. Leighton and R. O. Cleveland, “Lithotripsy,” http: / / dx.doi.org / 10.1243 / 09544119JEIM588, vol.224, no.2, pp.317–342, Oct.2009, doi: 10.1243 / 09544119JEIM588. E. Emiliani, A. Kanashiro, and O. Angerri, “Lasers for stone lithotripsy: advantages / disadvantages of each laser source,” Curr Opin Urol, vol.33, no.4, pp.302–307, Jul.2023, doi: 10.1097 / MOU.0000000000001092. M. Mohammadzadeh, J. M. Mercado, and C. D. Ohl, “Bubble Dynamics in Laser Lithotripsy,” J Phys Conf Ser, vol.656, no.1, p.012004, Nov.2015, doi: 10.1088 / 1742- 6596 / 656 / 1 / 012004. A. Fedorov, R. Beichel, J. Kalpathy-Cramer, J. Finet, J.C. Fillion-Robin, S. Pujol, C. Bauer, D. Jennings, F. Fennessy, M. Sonka, J. Buatti, S. Aylward, J.V. Miller, S. Pieper, R. Kikinis: 3D Slicer as an image computing platform for the Quantitative Imaging Network. Magnetic Resonance Imaging 30(9), 1323–1341 (112012). N. Siddique, S. Paheding, C. P. Elkin and V. Devabhaktuni, “U-Net and Its Variants for Medical Image Segmentation: A Review of Theory and Applications,” in IEEE Access, vol.9, pp.82031-82057, 2021, doi: 10.1109 / ACCESS.2021.3086020. Cs. Pinter, et al, “Polymorph segmentation representation for medical image computing”, Computer Methods and Programs in Biomedicine, Volume 171, p19-26, 2019 : http: / / perk.cs.queensu.ca / sites / perkd7.cs.queensu.ca / files / Pinter2019_Manuscript.pdf. GM, H., Mori, K., Verma, S., Athanasiou, L.: The influence of image cropping sizes on mammographic breast cancer classification using CNN. In: IEEE International Conference on Bioinformatics and Bioengineering (BIBE) (2023)

[0117] These publications are incorporated by reference herein in their entirety.

[0118] While the present disclosure has been described with reference to exemplary embodiments, it is to be understood that the present disclosure is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

Claims

CLAIMS What is claimed is:

1. A method for performing lithotripsy, the method comprising: inserting a catheter into a lumen; navigating the catheter through the lumen along an insertion trajectory; obtaining at least one image of one or more objects within the lumen; segmenting the at least one image; detecting at least one object of the one or more objects having a size exceeding a threshold as at least one target object; and defining a region of lasing (ROL) of the at least one target object, aligning a tip of the catheter with the ROL, and performing lithotripsy.

2. The method of claim 1, wherein the catheter is configured to transmit laser light.

3. The method of claim 2, wherein, upon defining the ROL, the catheter tip automatically moves into alignment with the ROL.

4. The method of claim 3, wherein, upon moving into alignment with the ROL, the laser light is transmitted.

5. The method of claim 1, wherein a laser configured to output laser light is provided on or adjacent to the catheter tip.

6. The method of claim 5, wherein, when the laser is aligned with the ROL, a notification is output to autonomously perform the lithotripsy.

7. The method of claim 5, wherein the at least one object is a kidney stone and the laser maintains a minimum predefined distance from kidney tissue.

8. The method of claim 1, wherein at least one object of the one or more objects is a kidney stone, and the ROL is at least one of a perimeter around a center of gravity of the segmented at least one image or an overlapping area of a previously detected kidney stone.

9. The method of claim 5, wherein, in response to a diameter of the at least one target object being less than a predetermined size, the laser is maintained in a deactivated state.

10. The method of claim 5, wherein, in response to the size exceeding or being equal to a predetermined size, the laser is activated.

11. The method of claim 10, wherein, in response to activating the laser, the method further comprises: obtaining at least one image of another object within the lumen; segmenting the at least one image of the another object; determining a size of the another object; in response to the size of the another object being less than the predetermined size, maintaining the laser in a deactivated state.

12. The method of claim 11, wherein the another object is a part of the object after performing the lithotripsy targeted on the ROL of the object.

13. The method of claim 11, wherein, in response to the size of the another object exceeding or being equal to the predetermined size, the method further comprises: defining an ROL of the another object, aligning the catheter tip with the ROL of the another object, and performing lithotripsy.

14. The method of claim 1, wherein the size of the one or more objects is defined based on the segmented at least one image.

15. The method of claim 1, wherein the size of the one or more objects is defined based on at least one physical parameter of a respective object of the one or more objects.

16. The method of claim 15, wherein the at least one physical parameter is one or more of a diameter, a radius, a circumference, an area of the respective object, a size, a distance from the catheter tip, a morphological feature, and a shape.

17. The method of claim 1, wherein segmenting the at least one image is automated.

18. The method of claim 1, further comprising: generating phantom data; and training a stone segmentation algorithm using the generated phantom data.

19. The method of claim 18, wherein the training of the stone segmentation algorithm is performed before inserting the catheter tip into the lumen.

20. The method of claim 1, wherein the threshold is set by a user or is obtained from a lookup table.

21. The method of claim 1, wherein the threshold is at least 1.5 mm.

22. The method of claim 1, wherein the catheter is a steerable multi-section catheter.

23. The method of claim 1, wherein the segmenting is performed using a convolutional neural network.

24. A method for performing lithotripsy, the method comprising: inserting a catheter into a lumen; navigating the catheter through the lumen along an insertion trajectory; obtaining at least one image of one or more objects within the lumen; segmenting the at least one image; identifying at least two targets among the segmented at least one image; identifying at least one parameter of the targets; prioritizing the targets based on at least one parameter of each target; defining a region of lasing (ROL) for a highest priority target; aligning a tip of the catheter with the ROL; and performing lithotripsy.

25. The method of claim 24, wherein the at least one parameter is a physical parameter.

26. The method of claim 25, wherein the physical parameter is one or more of a diameter, a radius, a circumference, an area of a respective object, a size, a distance from the catheter tip, a morphological feature, and a shape.

27. The method of claim 25, wherein the physical parameter is size, and any target smaller than or equal to a threshold size is removed from the prioritized targets.

28. The method of claim 24, further comprising: after performing lithotripsy, obtaining another at least one image; segmenting the another at least one image; identifying one or more secondary targets corresponding to objects in the at least one image having a size exceeding a threshold; and prioritizing the one or more secondary targets based on a physical parameter.

29. The method of claim 28, wherein the prioritized one or more secondary targets are output as a list for modification by a user.

30. An information processing apparatus to control a steerable catheter, the information processing apparatus comprising: at least one memory configured to store instructions; and at least one processor configured to execute the stored instructions to cause the steerable catheter to: obtain at least one image of one or more objects within a lumen; segment the at least one image; detect at least one object of the one or mor objects having a size exceeding a threshold as at least one target object; and define a region of lasing (ROL) of the at least one target object, align a catheter tip with the ROL, activate a laser on a distal end of the steerable catheter, and perform lithotripsy.

31. An autonomous navigation robot system, comprising: a steerable catheter; a camera at the distal end of the steerable catheter; one or more actuators configured to automatically move the steerable catheter; anda controller configured to: obtain, from the camera, at least one image of an object within a lumen; segment the at least one image; determine a size of the object; and in response to the size of the object exceeding or being equal to a predetermined size: detect a region of lasing (ROL) of the object, align the catheter tip with the ROL, activate a laser on a distal end of the steerable catheter, and perform lithotripsy.

Images