Control scheme calibration for medical devices

Abstract

Discussed herein are methods, systems, and devices for calibrating medical instruments. For example, a first instrument can be configured to access an anatomical site via a first access path, and a second instrument can be configured to access the anatomical site via a second access path. The second instrument can include an imaging component configured to provide image data representative of the anatomical site and the first instrument. A difference between a first coordinate system associated with the first instrument and a second coordinate system associated with the second instrument can be identified. A control reference coordinate system associated with the first instrument can be updated based at least in part on the difference.

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims priority to U.S. Provisional Patent Application No. 63 / 042,457, filed June 22, 2020, and entitled "CONTROL SCHEME CALIBRATION FOR MEDICAL INSTRUMENTS," the disclosure of which is incorporated herein by reference in its entirety.

[0002] FIELD OF THE INVENTION FIELD OF THE DISCLOSURE The present disclosure relates to the field of medical devices and procedures. [Background technology]

[0003] Various medical procedures involve the use of one or more scopes and / or percutaneous access instruments. Certain surgical processes may involve inserting one or more devices through a patient's skin and other anatomical structures to reach a treatment site and extract an object, such as a urinary stone, from a patient. Improper positioning or advancement of such devices may result in certain physiological and procedural complications. Summary of the Invention [Means for solving the problem]

[0004] In some implementations, the present disclosure relates to a medical system comprising: a first instrument configured to access an anatomical site via a first access path; a second instrument including an imaging component configured to provide image data representative of the anatomical site and the first instrument; and one or more computer-readable media storing executable instructions that, when executed by control circuitry, cause the control circuitry to identify a difference between a first coordinate system associated with the first instrument and a second coordinate system associated with the second instrument, and update a control reference coordinate system associated with the first instrument based at least in part on the difference.

[0005] In some embodiments, the first coordinate system indicates the roll of the distal end of the first instrument and the second coordinate system indicates the roll of the distal end of the second instrument.

[0006] In some embodiments, the one or more computer-readable media further store executable instructions that, when executed by the control circuitry, cause the control circuitry to display an alignment indicator, the alignment indicator representing a first coordinate system, and receive input including an adjustment to the alignment indicator. A difference between the first coordinate system and the second coordinate system can be identified based at least in part on the adjustment to the alignment indicator. In some embodiments, the first instrument includes one or more markings thereon, and the alignment indicator represents an orientation of the one or more markings. Further, in some embodiments, the one or more computer-readable media further store executable instructions that, when executed by the control circuitry, cause the control circuitry to display a user interface. The user interface may include a graphical representation of the image data and the alignment indicator. The alignment indicator may include a ring and one or more marking indicators representing an orientation of the one or more markings.

[0007] In some embodiments, the medical system further comprises a robotic manipulator configured to connect to and control movement of the first instrument. The one or more computer-readable media may further store executable instructions that, when executed by the control circuitry, cause the control circuitry to determine a first coordinate system based on at least one of a position or an orientation of the robotic manipulator.

[0008] In some embodiments, the one or more computer-readable media may further store executable instructions that, when executed by the control circuit, cause the control circuit to receive a directional input signal from an input device and control movement of the first instrument based at least in part on the directional input signal and the control reference frame. The directional input signal may indicate a direction of movement of the first instrument.

[0009] In some implementations, the present disclosure relates to a method that includes receiving, by a control circuit, image data from a first instrument positioned at a target anatomical site, the image data representing at least a portion of a second instrument. The method further includes displaying a graphical representation of the image data and receiving an input indicating an orientation of the second instrument in the graphical representation, and the control circuit calibrating a control scheme for the second instrument based at least in part on the input.

[0010] In some embodiments, the method further includes receiving a directional input indicating a direction of movement of the second instrument, and controlling the movement of the second instrument based at least in part on the directional input and the control scheme.

[0011] In some embodiments, calibrating the control scheme includes determining a roll of the distal end of the first instrument relative to the second instrument based at least in part on the input, and adjusting a control reference frame used to control the second instrument based at least in part on the roll of the distal end of the first instrument relative to the second instrument. Determining the roll of the first instrument relative to the second instrument may include determining a roll of the distal end of the first instrument relative to the distal end of the second instrument.

[0012] In some embodiments, the method further includes displaying an alignment indicator representing an orientation of the second instrument. Receiving the input may include receiving an adjustment to the alignment indicator. The second instrument may include one or more markings on a distal end of the second instrument, and the alignment indicator may represent an orientation of the one or more markings. The first instrument may be an endoscope and the second instrument may be a catheter.

[0013] In some implementations, the present disclosure relates to one or more non-transitory computer-readable media storing computer-executable instructions that, when executed by a control circuit, cause the control circuit to perform operations including receiving image data from a direct access device positioned at a target anatomical location, the image data representing at least a portion of the percutaneous access device positioned at the target anatomical location, and generating roll data indicative of a roll of a distal end of the direct access device relative to the percutaneous access device based at least in part on the image data, and controlling movement of the percutaneous access device based at least in part on the roll data.

[0014] In some embodiments, the direct access device includes an endoscope and the percutaneous access device includes a catheter. The roll data can indicate an orientation of a coordinate system associated with the direct access device relative to a coordinate system associated with the percutaneous access device. The image data can represent a distal end of the percutaneous access device including one or more markings.

[0015] In some embodiments, the operations further include displaying a graphical representation of the image data, displaying an alignment indicator, and receiving input including an adjustment to the alignment indicator. The alignment indicator can represent an estimated orientation of the percutaneous access device. The roll data can be generated based at least in part on the adjustment to the alignment indicator. The operations can further include determining the estimated orientation of the percutaneous access device based on at least one of a position or an orientation of a robotic manipulator configured to control the percutaneous access device.

[0016] In some embodiments, the operations further include receiving a directional input signal from an input device. The directional input signal may indicate a direction of movement of the percutaneous access device. Controlling the movement of the percutaneous access device may include generating a control signal for controlling the movement of the percutaneous access device based at least in part on the roll data.

[0017] In some embodiments, the operations further include performing one or more image processing techniques using the image data and determining an orientation of the percutaneous access device relative to the direct access device based at least in part on the one or more image processing techniques. The roll data can be generated based at least in part on the orientation of the percutaneous access device relative to the direct access device.

[0018] In some embodiments, the operations further include: displaying a graphical representation of the image data via a user interface; displaying instructions via the user interface, the instructions indicating selecting a particular directional control on an input device; receiving a directional input signal from the input device, the directional input signal being associated with the particular directional control; and receiving input indicating a direction in which the percutaneous access device has been moved relative to the user interface. Roll data can be generated based at least in part on the input.

[0019] In some embodiments, the operations further include displaying a graphical representation of the image data via a user interface, displaying instructions via the user interface, the instructions indicating to move the percutaneous access device in a particular direction relative to the user interface, receiving a directional input signal from an input device, and controlling the percutaneous access device to move based at least in part on the directional input signal. Roll data can be generated based at least in part on the directional input signal.

[0020] In some implementations, the present disclosure relates to a system comprising: a first robotic manipulator configured to manipulate a direct access device; a second robotic manipulator configured to manipulate a percutaneous access device; and a control circuit communicatively coupled to the first and second robotic manipulators, wherein the control circuit is configured to perform operations including receiving image data from the direct access device, the image data representing at least a portion of the percutaneous access device, causing a graphical representation of the image data to be displayed, receiving an input indicating an orientation of the percutaneous access device in the graphical representation, and calibrating a control scheme for the percutaneous access device based at least in part on the input.

[0021] In some embodiments, the system further includes a direct access instrument configured to access the target anatomical site via a natural lumen of the patient. The percutaneous access instrument may be configured to access the target anatomical site via a percutaneous access pathway within the patient. The percutaneous access instrument may include one or more markings on a distal end. The system may further include a display configured to display a graphical representation and an alignment indicator including one or more marking indicators representing an orientation of the one or more markings. The input may include an adjustment to the alignment indicator. The direct access instrument may include an endoscope, and the percutaneous access instrument may include a catheter.

[0022] In some embodiments, calibrating the control scheme includes determining a roll of the distal end of the direct access device relative to the percutaneous access device based at least in part on the input, and adjusting a control reference frame used to control the percutaneous access device based at least in part on the roll of the distal end of the direct access device relative to the percutaneous access device.

[0023] In some embodiments, the operations further include receiving a first directional input signal indicating a direction of movement of the percutaneous access device, and controlling the movement of the percutaneous access device based at least in part on the first directional input signal and the control scheme.

[0024] For purposes of summarizing the present disclosure, certain aspects, advantages, and novel features have been described. It is to be understood that not necessarily all such advantages may be achieved in accordance with any particular embodiment. Thus, the disclosed embodiments may be practiced in a manner that achieves or optimizes one advantage or group of advantages as taught herein without necessarily achieving other advantages as may be taught or suggested herein. [Brief explanation of the drawings]

[0025] Various embodiments are depicted in the accompanying drawings for illustrative purposes, and should not be construed as limiting the scope of the present disclosure. In addition, various features of different disclosed embodiments can be combined to form additional embodiments that are part of the present disclosure. Throughout the drawings, reference numerals may be reused to indicate correspondence between referenced elements. [Figure 1] FIG. 1 illustrates an exemplary medical system for performing various medical procedures, in accordance with one or more embodiments of the present disclosure. [Figure 2] FIG. 1 illustrates an exemplary scope disposed in a portion of a patient's urinary system, in accordance with one or more embodiments. [Figure 3] FIG. 1 illustrates an exemplary catheter disposed in a patient's kidney, in accordance with one or more embodiments. [Figure 4-1] 10A-10C illustrate actuation of a catheter in direct control mode when the catheter is directly facing the scope in accordance with one or more embodiments. [Figure 4-2] 10A-10C illustrate actuation of a catheter in direct control mode when the catheter is directly facing the scope in accordance with one or more embodiments. [Figure 5-1] FIG. 10 illustrates actuation of a catheter in direct control mode when the catheter and scope are pointing in substantially the same direction, in accordance with one or more embodiments. [Figure 5-2] FIG. 10 illustrates actuation of a catheter in direct control mode when the catheter and scope are pointing in substantially the same direction, in accordance with one or more embodiments. [Figure 6-1] 10A-10C illustrate actuation of a catheter in an inverted control mode when the catheter is directly facing the scope, in accordance with one or more embodiments. [Figure 6-2] 10A-10C illustrate actuation of a catheter in an inverted control mode when the catheter is directly facing the scope, in accordance with one or more embodiments. [Figure 7] FIG. 1 illustrates an exemplary interface for controlling / navigating a medical instrument, according to one or more embodiments. [Figure 8]FIG. 1 is an example flow diagram of a process for controlling a medical device from the perspective of another medical device, according to one or more embodiments. [Figure 9-1] 10A-10C illustrate an example implementation of driving a medical instrument from a first-person perspective for different control modes relative to the coordinate system of the medical instrument, in accordance with one or more embodiments. [Figure 9-2] 10A-10C illustrate an example implementation of driving a medical instrument from a first-person perspective for different control modes relative to the coordinate system of the medical instrument, in accordance with one or more embodiments. [Figure 10-1] 10A-10C illustrate an example implementation of driving a medical instrument from a third-person perspective for different control modes relative to the coordinate system and control frame of the medical instrument, in accordance with one or more embodiments. [Figure 10-2] 10A-10C illustrate an example implementation of driving a medical instrument from a third-person perspective for different control modes relative to the coordinate system and control frame of the medical instrument, in accordance with one or more embodiments. [Figure 11] FIG. 1 illustrates an exemplary interface for calibrating a control scheme for a medical instrument, according to one or more embodiments. [Figure 12] FIG. 1 illustrates an exemplary interface for calibrating a control scheme for a medical instrument, according to one or more embodiments. [Figure 13] FIG. 10 is an example flow diagram of a process for calibrating a control scheme / control reference frame for a medical instrument, according to one or more embodiments. [Figure 14-1] FIG. 1 illustrates an exemplary implementation of calibrating a control frame for a medical instrument, according to one or more embodiments. [Figure 14-2] FIG. 1 illustrates an example implementation for calibrating a control frame for a medical instrument, according to one or more embodiments. [Figure 14-3] FIG. 1 illustrates an exemplary implementation of calibrating a control frame for a medical instrument, according to one or more embodiments. [Figure 14-4] FIG. 1 illustrates an exemplary implementation of calibrating a control frame for a medical instrument, according to one or more embodiments. [Figure 15] FIG. 2 illustrates a top view of the medical system of FIG. 1 arranged to assist in inserting a scope into a patient, in accordance with one or more embodiments. [Figure 16] FIG. 2 illustrates a top view of the medical system of FIG. 1 arranged to navigate a scope within a patient, in accordance with one or more embodiments. [Figure 17] 2 illustrates a top view of the medical system of FIG. 1 arranged to assist in inserting a needle into a patient, according to one or more embodiments. [Figure 18] 2 illustrates a top view of the medical system of FIG. 1 arranged to navigate a catheter within a patient, in accordance with one or more embodiments. [Figure 19] FIG. 2 is an exemplary detailed diagram of the robotic system of FIG. 1 in accordance with one or more embodiments. [Figure 20] FIG. 2 is an exemplary detailed diagram of the control system of FIG. 1 in accordance with one or more embodiments. [Figure 21A] FIG. 2 illustrates an exemplary detailed view of a controller according to one or more embodiments. [Figure 21B] FIG. 2 illustrates an exemplary detailed view of a controller according to one or more embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0026] The headings provided herein are for convenience only and do not necessarily affect the scope or meaning of the disclosure. While certain preferred embodiments and examples are disclosed below, the subject matter extends beyond the specifically disclosed embodiments to other alternative embodiments and / or uses, as well as to modifications and equivalents thereof. Accordingly, the scope of claims that may arise from this specification is not limited by any of the specific embodiments described below. For example, in any method or process disclosed herein, the acts or operations of the method or process may be performed in any suitable order and are not necessarily limited to any particular disclosed order. Various operations may be described as multiple separate operations in a manner that may be helpful in understanding a particular embodiment. However, the order of description should not be construed to imply that these operations are order-dependent. In addition, structures, systems, and / or devices described herein may be embodied as integrated or separate components. For purposes of comparing various embodiments, certain aspects and advantages of these embodiments are described. Not necessarily all such aspects or advantages are achieved by any particular embodiment. Thus, for example, various embodiments may be implemented in a manner that achieves or optimizes one advantage or group of advantages as taught herein without necessarily achieving other aspects or advantages that may also be taught or suggested herein.

[0027] Certain standard anatomical terms of location are used herein to refer to animal, i.e., human, anatomical structures with respect to preferred embodiments. While certain spatially relative terms, such as "outer," "inner," "superior," "lower," "below," "upper," "vertical," "horizontal," "top," "bottom," and similar terms, are used herein to describe the spatial relationship of one device / element or anatomical structure to another, it is understood that these terms are used herein for ease of description to describe the positional relationships between the element(s) / structure(s) as illustrated in the drawings. It should be understood that spatially relative terms are intended to encompass different orientations of the element(s) / structure(s) during use or operation in addition to the orientation depicted in the drawings. For example, an element / structure described as "above" another element / structure may represent a position below or beside such other element / structure relative to the intended patient or alternative orientations of the element / structure, and vice versa.

[0028] Overview The present disclosure relates to systems, devices, and methods for controlling and / or calibrating medical instruments to assist in certain medical procedures. While certain aspects of the present disclosure are described in detail herein in the context of renal, urinary, and / or nephrological procedures, such as kidney stone removal / treatment procedures, it should be understood that such context is provided for convenience and that the concepts disclosed herein are applicable to any suitable medical procedure. However, as noted above, a description of renal / urinary anatomical structures and associated medical problems and procedures is presented below to assist in describing the concepts disclosed herein.

[0029] Kidney stone disease, also known as urolithiasis, is a relatively common medical condition involving the formation of solid pieces of material within the urinary tract, referred to as "kidney stones," "urinary stones," "renal calculi," "nephrolithiasis," or "nephrolithiasis." Urinary stones can form and / or be found within the kidneys, ureters, and bladder (referred to as "bladder stones"). Urinary stones form as a result of concentrated minerals and can cause significant abdominal pain if they reach a size sufficient to obstruct urine flow through the ureters or urethra. Urinary stones can be formed from calcium, magnesium, ammonia, uric acid, cysteine, and / or other compounds.

[0030] Generally, there are several methods for treating patients with kidney stones, including observation, medical treatment (such as expulsion therapy), non-invasive treatment (such as extracorporeal shock wave lithotripsy (ESWL)), and surgical treatment (such as ureteroscopy and percutaneous nephrolithotomy ("PCNL")). In a surgical approach, a physician gains access to the lesion (i.e., the object to be removed, e.g., the stone) and breaks the stone into small pieces or fragments, and the relatively small stone fragments / particles are removed from the kidney.

[0031] In some cases, physicians may use a ureteroscope to remove urinary stones from the bladder and / or ureter. Typically, a ureteroscope includes an endoscope at its distal end configured to allow visualization of the urinary tract. The ureteroscope may also include a stone extraction mechanism for capturing or fragmenting urinary stones. During a ureteroscopy, a physician may insert the ureteroscope into the urinary tract through the urethra. One physician / technologist may control the position of the ureteroscope, while another physician / technologist may control the lithotomy mechanism(s).

[0032] In other cases, physicians may use percutaneous nephrolithotomy ("PCNL") techniques to remove stones that are too large or resistant to other forms of treatment. This technique involves inserting a nephroscope through the skin (i.e., percutaneously) to fragment and / or remove the stone(s). In some implementations, fluoroscopy is used during the procedure to help guide the nephroscope and / or other instruments. However, fluoroscopy generally increases the cost of nephrolithotomy procedures due to the cost of the fluoroscope itself as well as the cost of the technician operating the fluoroscope. Fluoroscopy also exposes patients to radiation for a relatively long period of time. Even with fluoroscopy, accurately making percutaneous incisions to access the kidney stone(s) can be difficult and unnecessarily inaccurate. Furthermore, some nephrolithotomy techniques involve a two- or three-day hospital stay. In short, certain nephrolithotomy solutions can be relatively expensive and problematic for patients.

[0033] According to certain medical procedures according to aspects of the present disclosure, a medical system may implement multiple medical instruments to remove urinary stones from a patient or perform another medical procedure. The medical system may include a robotic tool that engages with and / or controls one or more medical instruments to access and / or remove urinary stones from a patient. For example, a physician may operate the medical system to drive a scope through a natural access pathway within the patient to a treatment site, such as through the urethra to the kidney where the kidney stone is located. The physician may use the scope to designate a target site where a catheter will meet the scope to assist in removing the kidney stone. The physician may operate the medical system to insert a catheter through a percutaneous access pathway and navigate the catheter to the treatment site where the scope is located. In some embodiments, the medical system may provide functionality to assist the physician in driving the catheter and / or scope. For example, the medical system may enable the physician to drive the catheter from the perspective of the scope. To do so, the medical system may provide an interface with image data from the perspective of the scope (e.g., images from the scope). The image data can depict the catheter at the treatment site. The medical system can determine the orientation of the scope relative to the catheter and, when input from the physician to move the catheter is received, can use such information to drive the catheter in the appropriate direction relative to the scope. The physician can view the catheter movement through the interface.

[0034] In some embodiments, the medical system may facilitate one or more control modes to assist the physician in driving the catheter from the perspective of the scope. For example, the medical system may implement an inverted control mode in which the direction of catheter movement is inverted (e.g., when a right input is received, the catheter moves to the left relative to the catheter, and vice versa). This may be useful when the catheter is positioned more directly relative to the scope (e.g., the tip of the catheter faces the tip of the scope). For example, if the catheter faces the scope and a left input is received, the catheter may move to the left relative to the interface used to control the catheter. Additionally, the medical system may implement a direct control mode in which the direction of catheter movement is not inverted (e.g., when a left input is received, the catheter moves to the left relative to the catheter, and when a right input is received, the catheter moves to the right relative to the catheter). This may be useful when the catheter is positioned more parallel to the scope (e.g., the tip of the catheter and the tip of the scope are pointing in substantially the same direction). For example, if the catheter and scope are pointing in the same direction and a left input is received, the catheter may move to the left relative to the interface used to control the catheter. The medical system may automatically select the control mode based on input from a physician or the like.

[0035] Furthermore, in some embodiments, the medical system may provide the ability to calibrate a control scheme for a medical instrument. Such calibration may be useful when the medical system inaccurately tracks or otherwise does not recognize the orientation of a medical instrument, as this may result in directional errors when controlling another medical instrument, such as when controlling a catheter from the perspective of a scope. For example, to calibrate a control scheme for a catheter controlled from the perspective of a scope, the medical system may determine the orientation of the scope relative to the catheter, such as the orientation of the distal end of the scope relative to the distal end of the catheter. In some embodiments, the medical system may provide a user interface with image data depicting the catheter from the perspective of the scope and depicting one or more interface elements to allow a physician to specify the orientation of the catheter. Furthermore, in some embodiments, the medical system may analyze image data and / or other sensor data from the scope and / or catheter to identify the orientation of the scope relative to the catheter. Furthermore, other techniques may be used to identify the orientation of the scope relative to the catheter. In any case, the medical system may use the orientation information to adjust a control scheme associated with controlling the catheter. Such adjustments may allow the catheter to move in the proper direction relative to the scope when input is provided by the physician.

[0036] In some implementations, the present disclosure relates to robotic-assisted medical procedures, in which a robotic tool may enable a physician to perform endoscopic and / or percutaneous access and / or endoscopic and / or percutaneous treatment for a target anatomical site. For example, the robotic tool may engage with and / or control one or more medical instruments to access and / or perform treatment at a target site within a patient. In some cases, the robotic tool is guided / controlled by a physician. In other cases, the robotic tool operates automatically or semi-automatically. Although many techniques are discussed in the context of robotic-assisted medical procedures, the techniques may also be applicable to other types of medical procedures, such as procedures that do not implement a robotic tool or that implement a robotic tool only for relatively few operations (e.g., below a threshold number).

[0037] In some of the embodiments described herein, the object removal procedure relates to the removal of kidney stones from a kidney. However, the present disclosure is not limited to kidney stone removal alone. For example, the following description is also applicable to other surgical / medical operations or procedures involving the removal of objects from a patient, including any object that can be removed from a treatment site or a patient's body cavity (e.g., esophagus, ureter, intestine, eye, etc.) via percutaneous and / or endoscopic access, such as gallbladder stone removal, lung (pulmonary / transthoracic) tumor biopsy, or cataract removal.

[0038] medical system FIG. 1 illustrates an exemplary medical system 100 for performing various medical procedures in accordance with aspects of the present disclosure. The medical system 100 includes a robotic system 110 configured to engage with and / or control medical instruments 120, 130, and / or other medical instruments to perform the procedure on a patient 140. The medical system 100 also includes a control system 150 configured to interface with the robotic system 110 and provide information about the procedure and / or perform various other operations. For example, the control system 150 may include display(s) 152 that present certain information to assist the physician 160. The medical system 100 may include a table 170 configured to hold the patient 140. In some embodiments, the medical system 100 may also include an imaging device 180 that may be integrated into a C-arm and / or configured to provide imaging during the procedure, such as a fluoroscopy-type procedure. Various actions are described herein as being performed by the physician 160. It should be understood that these actions may be performed directly by the physician 160, by a user under the direction of the physician 160, by another user (e.g., a technician), by a combination thereof, and / or by any other user.

[0039] 1 , medical instrument 120 is implemented as a scope, and medical instrument 130 is implemented as a catheter. Thus, for ease of discussion, medical instrument 120 will be referred to as a "scope 120" or a "direct access / entry instrument 120," and medical instrument 130 will be referred to as a "catheter 130" or a "percutaneous access / entry instrument 130." However, medical instrument 120 and medical instrument 130 may each be implemented as any type of medical instrument, including, for example, a scope (sometimes referred to as an "endoscope"), a catheter, a needle, a guidewire, a nephrolithotriptor, a basket retrieval device, forceps, a vacuum, a needle, a scalpel, an imaging probe, jaws, scissors, a grasper, a needle holder, a microdissection instrument, a staple applier, a tacker, an aspiration / irrigation tool, a clip applier, etc. In some embodiments, the medical instrument is a steerable device, while in other embodiments, the medical instrument is a non-steerable device. In some embodiments, a surgical tool refers to a device configured to puncture or be inserted through the human anatomy, such as a needle, scalpel, guidewire, etc. However, a surgical tool may also refer to other types of medical instruments.

[0040] The terms "scope" or "endoscope" are used herein according to their broad and ordinary meaning and can refer to any type of elongated medical instrument having imaging, viewing, and / or capture functionality and configured to be introduced into any type of organ, cavity, lumen, chamber, and / or space in the body. For example, references herein to a scope or endoscope can refer to a ureteroscope (e.g., for accessing the urinary tract), a laparoscope, a nephroscope (e.g., for accessing the kidney), a bronchoscope (e.g., for accessing the airways such as the bronchi), a colonoscope (e.g., for accessing the colon), an arthroscope (e.g., for accessing a joint), a cystoscope (e.g., for accessing the bladder), a borescope, etc. A scope / endoscope may, in some cases, comprise a rigid or flexible tube and may be sized to be passed through an outer sheath, catheter, introducer, or other luminal device, or may be used without such a device.

[0041] The terms "direct entry" or "direct access" are used herein according to their broad and ordinary meaning and may refer to any entry of instrumentation through a natural or artificial opening in a patient's body. For example, with reference to FIG. 1, as discussed above, scope 120 may be referred to as a direct access instrument because it enters the urinary tract of patient 140 via the urethra.

[0042] The terms “percutaneous entry” or “percutaneous access” are used herein according to their broad and ordinary meaning and may refer to entry, such as by puncture and / or small incision of instrumentation, through a patient's skin and through any other body layers necessary to reach the target anatomical location associated with the procedure (e.g., the calyx rete of the kidney). Accordingly, a percutaneous access instrument may refer to a medical instrument, device, or assembly configured to puncture or be inserted through the skin and / or other tissue / anatomical structure, such as a needle, scalpel, guidewire, sheath, shaft, scope, catheter, etc. However, it should be understood that a percutaneous access instrument may refer to other types of medical instrument in the context of this disclosure. In some embodiments, a percutaneous access instrument refers to an instrument / device inserted or implemented through a patient's skin, along with a device that facilitates the puncture and / or small incision. For example, a catheter 130 may be referred to as a percutaneous access instrument when the catheter 130 is inserted through a sheath / shaft that punctures the skin of the patient 140.

[0043] In some embodiments, a medical instrument, such as scope 120 and / or catheter 130, includes sensors (sometimes referred to as position sensors) configured to generate sensor data. In examples, the sensor data may indicate the position and / or orientation of the medical instrument and / or may be used to determine the position and / or orientation of the medical instrument. For example, the sensor data may indicate the position and / or orientation of the scope, which may include the roll of the distal end of the scope. The position and orientation of the medical instrument may be referred to as the attitude of the medical instrument. The sensors may be positioned at the distal end of the medical instrument and / or at any other location. In some embodiments, the sensors may provide sensor data to control system 150 and / or other systems / devices, which may perform one or more localization techniques to determine / track the position and / or orientation of the medical instrument.

[0044] In some embodiments, the sensor may include an electromagnetic (EM) sensor having a coil of conductive material, where an EM field generator may provide an EM field that is detected by the EM sensor on the medical instrument. The magnetic field may induce a small current in the sensor coil of the EM sensor, which may be analyzed to determine the distance and angle between the EM sensor and the EM field generator. Additionally, the sensor may include other types of sensors, such as, for example, a camera, a range sensor, a radar device, a shape-sensing fiber, an accelerometer, a gyroscope, a satellite-based positioning sensor (e.g., a global positioning system (GPS)), a radio frequency transceiver, etc.

[0045] A medical instrument can be associated with a coordinate system that can include a set of two or more vectors (or axes) that are perpendicular to one another. For example, in three-dimensional space, the coordinate system can include three vectors (e.g., an x-vector, a y-vector, and a z-vector) that are perpendicular to one another. While various conventions can be used, for ease of illustration, the description herein often refers to the "forward" direction (e.g., insertion / retraction) as corresponding to positive z, the "right" direction as corresponding to positive x, and the "up" direction as corresponding to positive y. The z-vector can extend along the longitudinal axis of the medical instrument. Such a coordinate system can be referred to as a "left-handed coordinate system." However, the disclosure herein can equally be described / implemented in the context of a right-handed coordinate system. In embodiments, the coordinate system is established based on / correlates to the position of one or more elongated movement members (e.g., one or more pull wires) of a medical device. Further, in embodiments, the coordinate system is established based on / correlates to the position of an imaging device on the medical instrument, such as the distal end of the imaging device on the tip of a scope. Thus, the coordinate system can correspond to the reference frame of the camera. However, the coordinate system may be relative to / set elsewhere. In many instances, the coordinate system for a medical instrument is expressed / described relative to the distal end of the medical instrument (e.g., the end at the treatment site). However, the coordinate system may be positioned elsewhere.

[0046] As described above, control system 150 may be configured to provide various functions to assist in performing a medical procedure. In some embodiments, control system 150 may be coupled to robotic system 110 and operate in cooperation with robotic system 110 to perform a medical procedure on patient 140. For example, control system 150 may communicate with robotic system 110 via a wireless or wired connection to control scope 120 and / or catheter 130 connected to robotic system 110, receive image(s) acquired by scope 120, etc. Additionally or alternatively, control system 150 may provide fluid to robotic system 110 via one or more fluid channels, provide power to robotic system 110 via one or more electrical connections, provide optics to robotic system 110 via one or more optical fibers or other components, etc. In some embodiments, control system 150 may communicate with scope 120 (and / or catheter 130) to receive sensor data (via robotic system 110 and / or directly from scope 120 and / or catheter 130). In examples, the sensor data may indicate or be used to determine the position and / or orientation of the medical instrument. Further, in some embodiments, the control system 150 may communicate with the table 170 to position or otherwise control the table 170 in a particular orientation. Further, in some embodiments, the control system 150 may communicate with the EM field generator to control the generation of the EM field around the patient 140.

[0047] Control system 150 includes various I / O devices configured to assist physician 160 or others in performing medical procedures. In the example of FIG. 1 , control system 150 includes I / O device(s) 156 that physician 160 or other users use to navigate or otherwise control medical instruments. For example, physician 160 can provide input via I / O device(s) 156, and in response, control system 150 can send control signals to robotic system 110 to operate scope 120 / catheter 130. In examples, physician 160 can control scope 120 and / or catheter 130 (e.g., switch control between devices) using the same I / O device. In some embodiments, scope 120 is driven from a first-person perspective (e.g., from the perspective of scope 120) and / or catheter 130 is driven from a third-person perspective (e.g., from the perspective of scope 120), as discussed in more detail below. Although the I / O device(s) 156 are illustrated as controllers in the example of FIG. 1, the I / O device(s) 156 may be implemented as various types of I / O devices, such as a touchscreen, a touchpad, a mouse, a keyboard, etc.

[0048] 1, control system 150 may include display(s) 152 for providing various information regarding the procedure. For example, control system 150 may receive real-time images acquired by scope 120 and display the real-time images and / or virtual representations of the real-time images via display(s) 152. Display(s) 152 may present interface(s) 154, such as any of the interfaces discussed herein, which may include image data from scope 120 and / or another medical instrument.

[0049] In some embodiments, the control system 150 may provide the image data via the interface(s) 154 in a manner that maintains a constant orientation of the image data (sometimes referred to as the “original image view”). For example, the interface(s) 154 may maintain a constant relationship to the coordinate system of the scope 120 (e.g., so that up in the interface(s) 154 corresponds to the positive y vector of the coordinate system of the scope 120). For illustrative purposes, assume that a kidney stone depicted in the image data from the scope 120 initially appears on the left side in the interface(s) 154. If the scope rotates 180°, the kidney stone moves within the interface(s) 154 during the rotation and appears on the right side in the interface(s) 154 after the rotation. Here, the control system does not adjust the orientation of the image data displayed via the interface(s) 154. Thus, horizontal lines in the image data may be perceived as rolling.

[0050] In other embodiments, control system 150 may provide image data (sometimes referred to as “rotated images or virtual views”) via interface(s) 154 to update the orientation of the image data. For example, interface(s) 154 may update its relationship to the coordinate system of scope 120 (e.g., so that up in interface(s) 154 does not always correspond to the positive y vector of the coordinate system of scope 120). For illustrative purposes, assume that a kidney stone depicted in image data from scope 120 initially appears on the left side in interface(s) 154. If the scope rotates 180 degrees, the kidney stone still appears on the left side of interface(s) 154 after the rotation. Now, control system 150 may adjust the orientation of the image data displayed via interface(s) 154 as scope 120 rotates 180 degrees to maintain the same orientation of objects depicted in the image data (e.g., rotationally correct the image data). Thus, horizons in the image data may be perceived to remain the same.

[0051] Additionally or alternatively, control system 150 may output other information via display(s) 152. For example, control system 150 may receive signals (e.g., analog, digital, electrical, acoustic / sonic, pneumatic, tactile, hydraulic, etc.) from medical monitors and / or sensors associated with patient 140, and display(s) 152 may present information related to the health or environment of patient 140. Such information may include information displayed via medical monitors, including, for example, heart rate (e.g., ECG, HRV, etc.), blood pressure / blood velocity, muscle biosignals (e.g., EMG), body temperature, blood oxygen saturation (e.g., SpO2), CO2, brain waves (e.g., EEG), environmental temperature and / or local or core body temperature, etc.

[0052] To facilitate the functioning of control system 150, control system 150 may include various components (sometimes referred to as "subsystems"). For example, control system 150 may include control electronics / circuitry as well as one or more power sources, pneumatics, light sources, actuators, memory / data storage devices, and / or communication interfaces. In some embodiments, control system 150 includes control circuitry comprising a computer-based control system configured to store executable instructions that, when executed, implement various operations. In some embodiments, control system 150 is mobile, as shown in FIG. 1 , while in other embodiments, control system 150 is a fixed system. While various functionality and components are discussed as being implemented by control system 150, any of this functionality and / or components may be integrated into and / or performed by other systems and / or devices, such as robotic system 110, table 170, or even scope 120 and / or catheter 130.

[0053] The robotic system 110 can be arranged in a variety of ways depending on the particular procedure. The robotic system 110 can include one or more robotic arms 112 configured to engage and / or control medical instrument(s) to perform the procedure. As shown, each robotic arm 112 can include multiple arm segments coupled to joints, thereby providing multiple degrees of mobility. In the example of FIG. 1 , two of the robotic arms 112 are actuated to engage with the scope 120 to access the target site through the urethra of the patient 140, and one of the robotic arms 112 is actuated to engage with the catheter 130 to access the target site through a percutaneous access pathway. Once the robotic system 110 is properly positioned, the scope 120 and / or catheter 130 can be inserted and / or navigated within the patient 140 robotically using the robotic arms 112, manually by the physician 160, or a combination thereof. 1, the robotic arm 112 may also be connected to other medical instruments that may be exchanged during a procedure, such as an electromagnetic (EM) field generator that may be positioned near a treatment site during certain stages of the procedure. Additionally, while the robotic arm 112 is shown in various positions and coupled to various instruments, it should be understood that such configurations are shown for convenience and explanation, and that such robotic arm 112 may have different configurations over time during a medical procedure.

[0054] The robotic system 110 may also include a support structure 114 coupled to the one or more robotic arms 112. The support structure 114 may include control electronics / circuitry, one or more power sources, one or more pneumatics, one or more light sources, one or more actuators (e.g., motors that move the one or more robotic arms 112), memory / data storage, and / or one or more communication interfaces. In some embodiments, the support structure 114 includes input / output (I / O) device(s) 116 configured to receive input, such as user input, for controlling the robotic system 110 and / or provide output, such as a graphical user interface (GUI), information about the robotic system 110, information about the procedure, etc. The I / O device(s) 116 may include a display, a touchscreen, a touchpad, a projector, a mouse, a keyboard, a microphone, a speaker, etc. In some embodiments, the robotic system 110 is mobile (e.g., the support structure 114 includes wheels) so that the robotic system 110 can be positioned where appropriate or desired for the procedure. In other embodiments, the robotic system 110 is a fixed system. Additionally, in some embodiments, the robotic system 112 is integrated into the table 170.

[0055] The robotic system 110 may be coupled to any component of the medical system 100, such as the control system 150, the table 170, the scope 120, the catheter 130, and / or other devices / instruments. In one example, the robotic system 110 is communicatively coupled to the control system 150 and may be configured to receive control signals from the control system 150 to perform actions such as positioning the robotic arm 112 in a particular manner, manipulating the scope 120 and / or the catheter 130, etc. In another example, the robotic system 110 is configured to receive images from the scope 120 depicting the internal anatomical structures of the patient 140 and / or transmit the images to the control system 150, which may then be displayed on the display(s) 152. Additionally, in some embodiments, the robotic system 110 is coupled to components of the medical system 100, such as the control system 150, in a manner that allows it to receive fluids, optics, power, etc. therefrom.

[0056] Imaging device 180 may be configured to acquire / generate one or more images of patient 140 during a procedure, such as one or more X-ray or CT images. In an example, images from imaging device 180 may be provided in real time to visualize anatomical structures and / or medical instruments, such as scope 120 and / or catheter 130, within patient 140 to assist physician 160 in performing the procedure. Imaging device 180 may be used to perform fluoroscopy (e.g., using a contrast agent within patient 140) or another type of imaging technique. Although shown in FIG. 1 , in some embodiments, imaging device 180 is not implemented to perform the procedure and / or imaging device 180 (including a C-arm) is omitted.

[0057] The various components of the medical system 100 may be communicatively coupled to one another via a network, which may include wireless and / or wired networks. Exemplary networks include one or more personal area networks (PANs), local area networks (LANs), wide area networks (WANs), Internet area networks (IANs), cellular networks, the Internet, etc. Additionally, in some embodiments, the components of the medical system 100 are connected for data communication, fluid / gas exchange, power exchange, etc. via one or more supporting cables, tubes, etc.

[0058] As described above, the medical system 100 may enable the physician 160 to drive a medical instrument, such as to navigate the medical instrument within the patient 140. For example, the control system 150 may receive input signals from the I / O device(s) 156 indicating the direction of movement of the medical instrument. The control system 150 may determine the orientation / position of the medical instrument, the orientation / position of another medical instrument (if applicable) providing image data for the medical instrument, and / or the orientation of image data displayed via the interface(s) 154. The control system 150 may use such information to generate control signals to move the medical instrument in the appropriate direction relative to the coordinate system / control frame of the medical instrument. The control system 150 may send control signals to the robotic system 110 to manipulate the medical instrument.

[0059] In some embodiments, the medical system 100 allows the physician 160 to drive the medical instrument from the perspective of the medical instrument (also referred to as "first-person driving"). This type of driving may be useful in contexts where the medical instrument includes an imaging device that provides image data and / or in other contexts. For example, the control system 150 may allow the scope 120 to be driven from the perspective of the scope 120. The scope 120 may include an imaging device configured to provide image data to the control system 150. The control system 150 may display the image data via the interface(s) 154 to assist the physician 160 in driving the scope 120 from the perspective of the scope 120.

[0060] In the case of first-person operation, control system 150 may generally control the medical instrument to move relative to the orientation of image data from the medical instrument displayed via interface(s) 154. For example, assume that physician 160 is driving scope 120 from the perspective of scope 120 and physician 160 provides input via I / O device(s) 156 to move scope 120 in an upward direction relative to I / O device(s) 156, such as by selecting a move up control on I / O device(s) 156. Control system 150 may determine the orientation / position of scope 120 and / or the orientation of image data from scope 120 displayed via interface(s) 154. Control system 150 may use such orientation / position information to move scope 120 in the appropriate direction that appears as an upward direction on interface(s) 154.

[0061] To illustrate, if interface(s) 154 are displaying a static view of scope 120 (e.g., where up in interface(s) 154 corresponds to the positive y vector of the coordinate system of scope 120), control system 150 can move scope 120 along the positive y vector of the coordinate system for scope 120 in response to an up input on I / O device(s) 156. Furthermore, if interface(s) 154 are displaying a rotated image view of scope 120 (e.g., where up in interface(s) 154 does not always correspond to the positive y vector of the coordinate system for scope 120), control system 150 can determine an offset of the reference frame of the image data relative to the coordinate system for scope 120 to move scope 120 in the appropriate direction that will cause scope 120 to appear to move upward in interface(s) 154.

[0062] Additionally, in some embodiments, the medical system 100 may enable the physician 160 to drive a medical instrument from the perspective of another medical instrument (also referred to as "third-person driving"). This type of driving may be useful in situations where medical instruments merge with one another and / or where one of the medical instruments does not include an imaging device. For example, the control system 150 may enable the physician 160 to drive the catheter 130 from the perspective of the scope 120, which may be useful when the catheter 130 does not include an imaging device. Here, the scope 120 may provide image data, and the control system 150 may display the image data via the interface(s) 154. When the catheter 130 is within the field of view of the scope 120, the physician 160 may view the catheter 130 in the interface(s) 154 and drive the catheter 130 from the perspective of the scope 120.

[0063] In the case of third-person drive, control system 150 may generally implement a control scheme for controlling the movement of the medical instrument. For example, assume that physician 160 is driving catheter 130 from the perspective of scope 120 and physician 160 provides input via I / O device(s) 156 to move catheter 130 in an upward direction relative to I / O device(s) 156, such as by selecting a move up control on I / O device(s) 156. Control system 150 may implement a control scheme for catheter 130 that takes into account the orientation of scope 120 relative to catheter 130. This may allow catheter 130 to move in the appropriate direction, which appears as an upward direction on interface(s) 154.

[0064] A control scheme can be used to map input controls to control signals to move the medical instrument. In some embodiments, the control scheme includes a control frame (sometimes referred to as a "control reference frame") that can include an extracted coordinate system / set of vectors used to control the medical instrument / device. For example, the control frame can include a set of two or more vectors (or axes) that are perpendicular to each other. The control frame can generally be correlated to a coordinate system relative to the medical instrument. For example, the control frame for a medical instrument can be offset relative to the coordinate system of that medical instrument (e.g., offset 30 degrees about an axis / vector). In an example, the coordinate system remains static relative to the medical instrument (i.e., fixed to a point on the medical instrument), but the control frame can be dynamically updated based on the roll of the medical instrument, the orientation of image data via a user interface, etc. In an example, the control frame is correlated to the tip of the medical instrument. However, the control frame can be correlated / centered relative to other locations.

[0065] In some embodiments, the control scheme / frame is determined based on the orientation of the image data displayed via the interface. For example, the vertical / horizontal axis of the control frame may be correlated / aligned to the vertical / horizontal axis of the image data in the interface. To illustrate, when driving the catheter 130 from the perspective of the scope 120, the control frame for the catheter 130 may be correlated to the orientation of the image data from the scope 120, as displayed via the interface(s) 154. Exemplary control frames are described in more detail below. While control frames are often discussed in the context of third-person driving, control frames can be used in other contexts, such as first-person driving.

[0066] In some situations with third-person driving, it may be difficult for the physician 160 to drive the medical instrument. For example, if the physician 160 is driving the catheter 130 from the perspective of the scope 120 and the catheter 130 is substantially directly facing the scope 120 (e.g., the tip of the catheter 130 faces the tip of the scope 120), the physician 160 must provide left-right inversion inputs to move the catheter 130 in the appropriate direction relative to the interface(s) 154. For example, if the physician 160 wants to move the catheter 130 to the left relative to the interface(s) 154, the physician 160 may be required to provide right inputs via the I / O device(s) 156, or vice versa. In contrast, if the catheter 130 and the scope 120 are facing substantially the same direction (e.g., the tip of the catheter 130 and the tip of the scope 120 are facing the same direction), such inversion inputs are not required. In short, certain situations of third-party driving may make it difficult for the physician 160 to drive the medical instrument, which may lead to harm to the patient 140 (e.g., if the medical instrument is moved in an undesirable direction), an inefficient procedure, etc.

[0067] Thus, the medical system 100 may facilitate one or more control / drive modes to assist the physician 160 in driving the medical instrument. By using multiple control modes, the medical instrument may be driven in an effective manner for medical instruments with different orientations relative to each other. For example, when the catheter 130 is being driven from the perspective of the scope 120, the physician 160 may be able to see the catheter 130 moving in a direction on the interface(s) 154 that more intuitively corresponds to the input provided via the I / O device(s) 156. In some examples, the medical system 100 can switch to different control modes by reconfiguring the control system 150 (e.g., to process input signals from the I / O device(s) 156 and / or generate control signals for the robotic system 110 in a different manner), by reconfiguring the I / O device(s) 156 (e.g., to send different input control signals), and / or by reconfiguring the robotic system 110 (e.g., to control the robotic arm in a different manner). Although control modes are often discussed in the context of third-person driving, such control modes may be used in other contexts, such as first-person driving or any other driving scenario.

[0068] In some embodiments, control system 150 may implement a direct control mode (also referred to as a “parallel mode”) to drive a medical instrument in a manner corresponding to the coordinate / control frame of that medical instrument. For example, when driving catheter 130 from the perspective of scope 120 in direct control mode, if physician 160 selects a left input on I / O device(s) 156, control system 150 may control catheter 130 to move left relative to catheter 130. When catheter 130 is facing substantially the same direction as scope 120, physician 160 can see (e.g., from a third-person perspective) that catheter 130 is moving left within interface(s) 154. In contrast, when catheter 130 is facing directly toward scope 120, physician 160 can see that catheter 130 is moving right within interface(s) 154. Thus, direct control mode may often be implemented when catheter 130 and scope 120 are facing substantially the same direction.

[0069] Additionally or alternatively, control system 150 may implement an inverted control mode (also referred to as a "mirror mode") for driving the medical instrument in a manner that is inverted relative to the coordinate system / control frame of the medical instrument. For example, when driving catheter 130 from the perspective of scope 120 in inverted control mode, if physician 160 selects a left input on I / O device(s) 156, control system 150 may control catheter 130 to move to the right relative to catheter 130. If catheter 130 is facing directly at scope 120, physician 160 may see (e.g., from a third-person perspective) that catheter 130 is moving to the left within interface(s) 154. In contrast, if catheter 130 is facing substantially the same direction as scope 120, physician 160 may see that catheter 130 is moving to the right within interface(s) 154. Therefore, the direct control mode can often be implemented when the catheter 130 and the scope 120 are substantially directly opposed to each other.

[0070] In some embodiments, the inverted control mode is associated with inverting the horizontal movement of the medical instrument but not the vertical movement. For example, in the inverted control mode, the control system 150 may invert the horizontal component of the coordinate system / control frame but not the vertical component(s) (e.g., invert the sign of the x value but not the y value). However, in some cases of the inverted control mode, the vertical direction may also be inverted.

[0071] Additionally, in some embodiments, the control mode may indicate whether retraction and insertion are reversed. For example, in a direct control mode, a forward input control on the I / O device(s) 156 may be associated with inserting the catheter 130 further into the patient 140, and a reverse input control on the I / O device(s) 156 may be associated with retracting the catheter 130 from the patient 140. If the catheter 130 is being driven from the perspective of the scope 120, with the instruments facing each other, the physician 160 may see the catheter 130 moving toward the scope 120 for a forward input and moving away from the scope 120 for a reverse input. Additionally, in a reverse control mode, a forward input control may be associated with retracting the catheter 130, and a reverse input control may be associated with inserting the catheter 130. When the catheter 130 is driven from the perspective of the scope 120 and the instruments are facing each other, the physician 160 can see the catheter 130 moving away from the scope 120 in response to a forward input and moving towards the scope 120 in response to a reverse input.

[0072] The control mode may be selected in various ways. In some embodiments, the physician 160 may provide input for selecting the control mode through the interface(s) 154, through the I / O device(s) 156, or otherwise. This may allow the physician 160 to configure the medical system 100 to the physician's 160 preferences. Additionally, in some embodiments, the control system 150 may automatically select a control mode appropriate for a particular situation. For example, the control system 150 may perform one or more localization techniques to determine and / or track the position and / or orientation of a medical instrument and / or another object, as described in more detail below with reference to FIG. 20 . In some cases, the control system 150 may automatically select an inverted control mode when the catheter 130 and the scope 120 are facing forward, and / or a direct control mode when the catheter 130 and the scope 120 are facing substantially the same direction.

[0073] In some embodiments, the medical system 100 may learn to automatically select a particular control mode. For example, if a particular physician selects the inverted control mode more than a threshold number of times when the scope 120 is directly opposed to the catheter 130 (e.g., oriented at an angle relative to each other), the medical system 100 may learn to automatically select the inverted control mode whenever the scope 120 and catheter 130 are similarly oriented in the future and / or when the particular physician is logged into the medical system 100. The medical system 100 may learn when to select a particular control mode using various parameters, such as the type of procedure being performed and the type of medical instrument being implemented. Thus, the medical system 100 may automatically select a control mode for a particular situation.

[0074] Further, in some embodiments, the medical system 100 may implement other techniques for driving the medical instrument instead of or in addition to implementing the control mode(s). In one example, the catheter 130 may include an imaging device, and the interface(s) 154 may provide image data from the catheter 130 to drive the catheter 130 from a first-person perspective. In another example, the tip of the catheter 130 may include one or more markings, and the I / O device(s) 156 may include the same one or more markings to indicate a mapping of input controls to the direction of movement of the catheter 130. By way of example, the tip of the catheter 130 may include a red marking on one side of the tip (e.g., one half of the tip) and a blue marking on the other side of the tip (e.g., the other half of the tip). Here, the I / O device(s) 156 may include a right input control with a red marking that, when selected, moves the catheter 130 in the direction of the red marking on the tip of the catheter 130. Additionally, I / O device 156 can include a left input control with blue markings that, when selected, moves catheter 130 in the direction of the blue markings on the tip of catheter 130. Thus, physician 160 can view the markings on I / O device(s) 156 to determine which input control to select.

[0075] In some embodiments, the medical system 100 may provide the ability to calibrate the control scheme for the medical instrument. Such calibration techniques may be useful when the medical system does not recognize or inaccurately determines the orientation / position of the medical instrument. For example, the medical system 100 may not know the amount of roll associated with the scope 120 when the scope 120 is mounted on the arm 112 of the robotic system 110. In some cases, the scope 120 includes a lockout position associated with a particular roll of the scope 120. The lockout may be associated with a mechanism on the scope 120 and / or a mechanism on a mounting component of the arm 112 (e.g., a pin, a slot for receiving a pin, etc.). When placed in the lockout position, the scope 120 cannot roll until it is placed on a mounting component of the arm 112 (which releases the lockout). When the scope 120 is mounted on the robotic system 110, the medical system 100 may generally assume that the scope 120 is in the lockout position (e.g., a zero roll position). However, if the scope 120 is removed from the robotic system 110 in a non-locked position and / or manually rolled to any roll position while away from the robotic system 110, and then reloaded onto the robotic system 100, the medical system 100 may incorrectly assume the amount of roll of the scope 120 (e.g., incorrectly assume that the scope 120 is in the locked position).

[0076] Furthermore, the medical system 100 may lose track of the amount of roll associated with the scope 120 during a procedure. For example, friction or other forces may be applied to the scope 120 during a procedure due to the scope 120 being positioned within various anatomical structures of a patient. Such forces may prevent manipulation of the proximal end of the scope 120 from fully propagating to the distal end of the scope 120, resulting in an undetected amount of roll at the tip of the scope 120. For example, if the robotic system 110 manipulates a robotic arm connected to the scope 120 to rotate the scope 120 80 degrees, the tip of the scope 120 may only rotate 60 degrees, even though the control system 150 may track the scope 120 as having rotated 80 degrees. Furthermore, during certain stages of a procedure, an EM field generator may be implemented to track the position / orientation of a medical instrument. However, during other stages, the EM field generator may need to be removed, such as to mount a catheter 130 or other instrument on the same arm (as discussed in more detail below). Without the EM field generator, it can be difficult to track the position / orientation of the medical instrument. Additionally, the scope 120 may experience a parasitic roll, called "curvature alignment," which causes the distal end of the scope 120 to rotate undesirably.

[0077] Such unexplained roll of a medical instrument, such as the scope 120, can result in clocking / roll errors that can make it difficult to control the scope 120 or another medical instrument. For example, if the scope 120 is estimated to have a certain amount of roll, but such estimation is inaccurate, the medical system 100 may drive the scope 120 and / or catheter 130 incorrectly from the perspective of the scope 120 (e.g., the scope 120 / catheter 130 may move in the wrong direction relative to the interface(s) 154).

[0078] To address the roll error situation, the medical system 100 may perform one or more calibration techniques to update orientation / position information associated with the medical instrument(s). For example, the medical system 100 may determine the orientation of the distal end of the scope 120 relative to the orientation of the distal end of the catheter 130 and use such information to adjust a control scheme associated with controlling the catheter 130 from the perspective of the scope 120. For example, if the medical system 100 determines that the estimated orientation of the scope 120 relative to the catheter 130 is off by 30 degrees from the actual orientation of the scope 120 relative to the catheter 130, the medical system 100 may adjust the control frame by 30 degrees or based on the 30-degree offset.

[0079] In some embodiments, control system 150 may provide information via interface(s) 154 to enable physician 160 to calibrate the orientation of the medical instrument. For example, control system 150 may display image data acquired by scope 120, which may depict catheter 130. Control system 150 may also display an alignment indicator representing the orientation of catheter 130 relative to scope 120 (e.g., the estimated orientation of the coordinate system of catheter 130 relative to the coordinate system of scope 120). In some embodiments, catheter 130 may include one or more markings, such as on the distal end of catheter 130, and the alignment indicator may represent the orientation of the one or more markings on catheter 130. If the alignment indicator is not aligned in interface(s) 154 with the one or more markings on catheter 130 as depicted in the image data, physician 160 can adjust the alignment indicator to the appropriate orientation to indicate the actual orientation of the tip of catheter 130 relative to the tip of scope 120. Based on the orientation indicated by the physician 160, the control system 150 may, if necessary, update the control scheme used to control the catheter 130, such as by updating one or more parameters associated with the control frame (e.g., the orientation of one or more vectors).

[0080] Additionally, in some embodiments, control system 150 may perform one or more image processing techniques to calibrate the orientation of the medical instrument. For example, control system 150 may process image data acquired by scope 120 showing catheter 130 to identify the orientation of catheter 130 relative to scope 120 (e.g., the orientation of the coordinate system of catheter 130 relative to the coordinate system of scope 130). Such processing may identify one or more markings on the tip of catheter 130 (if included), the amount or direction of bending of catheter 130, and / or other characteristics of catheter 130. Based on the orientation identified by image processing, control system 150 may update the control scheme of catheter 130, as needed.

[0081] Additionally, in some embodiments, control system 150 may instruct physician 160 to perform a particular action. In one example, control system 150 instructs physician 160 to select a particular directional control on I / O device(s) 156 (e.g., select the up input control). In response to physician 160 selecting a particular directional control, control system 150 may control catheter 130 to move in any direction, depending on the accuracy of the control frame / scheme associated with catheter 130. Control system 150 may then receive further input from physician 160 indicating the direction in which catheter 130 moved relative to interface(s) 154. Based on any discrepancy between the direction indicated by physician 160 that catheter 130 moved (relative to the control frame / coordinate system) and the direction control system 150 estimated that catheter 130 moved (relative to the control frame / coordinate system), control system 150 may update the control frame / scheme for the catheter, as necessary. In another example, control system 150 may instruct physician 160 to move catheter 130 in a particular direction relative to interface 154 (e.g., text such as "move catheter right," a right arrow, or the like may be displayed via interface(s) 154). Physician 160 may provide input control via I / O device(s) 156 to move catheter 130 in the particular direction relative to interface(s) 154, which may require multiple attempts. Upon successfully moving catheter 130 in the particular direction, physician 160 may provide an additional input indicating completion of such command. Based on the input control provided upon successful movement of catheter 130 in the particular direction on interface(s) 154, the direction in which catheter 130 moved relative to the control frame / coordinate system for successful movement, and / or the direction in which control system 150 estimates catheter 130 moved (relative to the control frame / coordinate system), control system 150 may update the catheter's control frame / scheme as needed.

[0082] While many embodiments are described in the context of calibrating a control scheme associated with third-person actuation, the calibration techniques may be implemented in the context of first-person actuation and / or any other scenario. Furthermore, while many embodiments are discussed in the context of calibrating a control scheme for a catheter, the calibration techniques may additionally or alternatively be implemented to calibrate a control scheme for a scope and / or another medical instrument. For example, the catheter 130 may in some cases be associated with a roll function, and such a calibration technique may be implemented to correct for roll error of the catheter 130.

[0083] In some embodiments, the calibration technique may be performed as part of a troubleshooting function, such as when the physician 160 notices that the medical instrument is not moving in a manner that correlates with the inputs provided to control the instrument. The physician 160 can then enter a calibration interface. However, the calibration technique may be performed automatically, periodically, or at any time and / or based on various events.

[0084] In some embodiments, the catheter 130 includes or is associated with certain characteristics that aid in using the catheter 130 as a reference point for calibration. For example, the catheter 130 may be unable to roll, which may avoid any potential roll errors. Furthermore, the catheter 130 may be inserted along a path that is substantially straight compared to the path traveled by the scope 120. This may minimize propagation errors at the distal end of the catheter 130 compared to the scope 120. Furthermore, in some examples, the catheter 130 may be relatively short compared to the scope 120. Such characteristics of the catheter 130 may allow the position / orientation of the catheter 130 to be accurately determined / tracked based on, for example, the orientation / position of the robotic arm 112 to which the catheter 130 is attached (e.g., the position / orientation of the catheter 130 may be offset from the plane in which the catheter 130 is mounted to the robotic arm 112).

[0085] Additional techniques may be implemented to calibrate the medical instrument and / or maintain accurate orientation / position information for the medical instrument. For example, control system 150 may use the orientation / position of robotic arm 112 to determine the position / orientation of catheter 130. Control system 150 may also determine how catheter 130 appears in image data acquired by scope 120 (e.g., using image processing or receiving user input to determine the direction / angle at which catheter 130 is positioned or at which image data is input). Based on the position / orientation of catheter 130 and how catheter 130 appears in the image data, control system 150 may determine the orientation of catheter 130 relative to scope 120 and calibrate the control scheme. Furthermore, control system 150 may determine / calibrate the orientation of catheter 130 relative to scope 120 based on sensors included on catheter 130 and / or on scope 120, such as EM sensors, gyroscopes, accelerometers, etc., and / or based on other localization techniques described herein. Further, the control system 150 may identify air bubbles depicted in the image data (e.g., based on image processing, user input indicating the location of the air bubbles, or the direction the bubbles are traveling, etc.) and use such information (along with information indicating where air bubbles typically collect / travel) to calibrate the orientation of the scope 120 and the orientation / control scheme of the medical instrument, such as the control scheme for the catheter 130. Furthermore, the robotic system 110 may implement a mechanical keying to require the scope 120 to be rolled to a lockout position before loading onto the robotic arm 112. This may avoid a situation where the control system 150 is unaware of the roll of the scope 120 before loading. Additionally, the control system 150 may implement a roll-homing technique in which the scope 120 is automatically (or manually) rolled to a hard-stop position each time the scope 120 is loaded onto the robotic arm 112. This assists the control system 150 in identifying the initial roll of the scope 120.

[0086] In examples, the medical system 100 may be used for percutaneous and / or endoscopic (e.g., ureteroscope) procedures. Certain ureteroscope procedures involve the treatment / removal of kidney stones. In some implementations, kidney stone treatment may benefit from the assistance of certain robotic techniques / devices, such as those shown in FIG. 1 and described in detail herein. Robotic medical solutions may provide relatively greater precision, control, and / or better eye-hand coordination for certain instruments compared to procedures using only human hands. For example, robot-assisted percutaneous access to the kidney through some procedures may advantageously enable a urologist to perform both direct-entry endoscopic renal access and percutaneous renal access. While some embodiments of the present disclosure are presented in the context of catheters, nephroscopes, ureteroscopes, and / or the human renal anatomy, it should be understood that the principles disclosed herein may be practiced with any type of endoscopic and / or percutaneous procedure.

[0087] In one exemplary percutaneous procedure, medical system 100 may be used to remove a kidney stone from patient 140 through a percutaneous access pathway. For example, physician 160 may interact with control system 150 (e.g., via I / O device(s) 156) to cause robotic system 110 to advance and / or navigate scope 120 from the urethra, through the bladder, up the ureter, and into the kidney where the stone is located. Control system 150 may provide information about scope 120, such as real-time images acquired using scope 120, via display(s) 152 to assist physician 160 in navigating scope 120. In an embodiment, scope 120 may be driven from a first-person perspective (e.g., from the perspective of scope 120). Once scope 120 reaches the site of the kidney stone (e.g., within the calyx of the kidney), scope 120 may be used to designate / tag a target location for catheter 130 to percutaneously access the kidney. To minimize damage to the kidney and / or surrounding anatomical structures, physician 160 may designate a particular papilla as a target location for percutaneous entry into the kidney with catheter 130. However, other target locations may also be designated or determined.

[0088] The physician 160 also interacts with the control system 150 to cause the robotic system 110 to advance and / or navigate the catheter 130 through the percutaneous access pathway to a target location designated by the scope 120. In some embodiments, a needle or another medical instrument is inserted into the patient 140 to create the percutaneous access pathway. The control system 150 may provide information about the catheter 130 via the display(s) 152 to assist the physician 160 in navigating the catheter 130. For example, the interface(s) 154 may provide image data from the perspective of the scope 120. The image data may depict the catheter 130 (e.g., when the catheter 130 is within the field of view of an imaging device of the scope 120). In an example, the catheter 130 may be driven from a third-person perspective (e.g., from the perspective of the scope 120).

[0089] Once the scope 120 and / or catheter 130 are positioned at the target location, the physician 160 may use the scope 120 to break up the kidney stone and / or use the catheter 130 to remove kidney stone fragments from the patient 140. For example, the scope 120 may deploy a tool (e.g., a laser, cutting instrument, etc.) to fragment the kidney stone into pieces, and the catheter 130 may aspirate the pieces out of the kidney through a percutaneous access pathway. In embodiments, the catheter 130 and / or scope 120 may provide irrigation and / or suction to facilitate kidney stone removal. For example, the catheter 130 may be coupled to an irrigation and / or suction system.

[0090] The medical system 100 may provide various benefits, such as providing guidance to assist physicians in performing procedures (e.g., instrument tracking, instrument navigation, instrument calibration, etc.), allowing physicians to perform procedures from an ergonomic position without requiring awkward arm movements and / or positions, allowing a single physician to perform procedures using one or more medical instruments, avoiding radiation exposure (e.g., associated with fluoroscopy techniques), allowing procedures to be performed in a single surgical setting, providing continuous suction to more efficiently remove objects (e.g., removing kidney stones), etc. For example, the medical system 100 may provide guidance information to assist physicians in accessing target anatomical features using various medical instruments while minimizing bleeding and / or damage to anatomical structures (e.g., critical organs, blood vessels, etc.). Additionally, the medical system 100 may provide non-radiation-based navigation and / or localization techniques to reduce physician and patient radiation exposure and / or reduce the amount of equipment in the operating room. Additionally, medical system 100 may provide distributed functionality between at least control system 150 and robotic system 110, which may be independently mobile. Such distribution of functionality and / or mobility may allow control system 150 and / or robotic system 110 to be placed in a location that is optimal for a particular medical procedure, thereby maximizing the working area around the patient and / or providing an optimized location for the physician to perform the procedure.

[0091] Although various techniques and systems are discussed as being implemented in a robotically-assisted procedure (e.g., a procedure that at least partially uses medical system 100), these techniques and systems may be implemented in other procedures, such as fully robotic medical procedures, human-only procedures (e.g., no robotic system is included), etc. For example, medical system 100 may be used to perform a procedure (e.g., a fully robotic procedure) without a physician holding / manipulating medical instruments. That is, each medical instrument used during the procedure may be held / controlled by a component of medical system 100, such as robotic arm(s) 112 of robotic system 110.

[0092] Additionally, although many of the techniques and systems are discussed in the context of a scope 120 meeting with a catheter 130, the techniques and systems may be applicable to other types of medical instruments, such as any type of medical instrument that may meet with another medical instrument at a treatment site or elsewhere, such as a laparoscope or other type of procedure.

[0093] Example Scope FIG. 2 illustrates an exemplary scope 202 (e.g., endoscope, ureteroscope, etc.) disposed in a portion of a patient's urinary system, according to one or more embodiments. Scope 202 may represent scope 120 of FIG. 1 and / or any other scope discussed herein. Scope 202 may be used in a ureteroscopy procedure to examine and / or treat abnormalities in a human ureter. For example, a ureteroscopy procedure may be performed to treat and / or remove kidney stones within kidney 204. However, scope 202 can be used in other types of procedures. As discussed above, ureteroscopy procedures and / or other types of procedures may be performed at least in part manually and / or at least in part using robotic technology, such as with medical system 100 shown in FIG. 1 .

[0094] The scope 202 may include imaging device(s) 206 configured to acquire image data, such as image data representing a patient's internal anatomical structures. The imaging device 206 may include a camera, such as an optical camera and / or another imaging device. The imaging device 206 may include an optical fiber, a fiber array, and / or a lens. One or more optical components of the imaging device 206 move with the tip 208 of the scope 202, such that movement of the tip 208 of the scope 202 results in changes in the image acquired by the imaging device 206. In some embodiments, the scope 202 may house wires and / or optical fibers for transmitting signals to / from the optical assembly and the distal end 208 of the scope 202. Additionally, the scope 202 may be further configured to house optical fibers for carrying light from a proximally located light source, such as a light emitting diode, to the distal end 208 of the scope. The distal end 208 of the scope 202 may include a port for a light source to illuminate the anatomical space when the imaging device 206 is in use.

[0095] The scope 202 may also include a working channel 210 for deploying medical instrument(s) (e.g., lithotriptor, basket device, forceps, laser, etc.), irrigation, and / or suction to a working area at the distal end 208 of the scope 202. In some embodiments, the working channel 210 is offset to one side of the scope 202, as shown in FIG. 2. In other examples, the working channel 210 is positioned in the center of the scope 202 or at another location.

[0096] The scope, in some cases, may comprise a rigid or flexible tube and / or may be sized to pass through an outer sheath, catheter, introducer, or other luminal device. However, the scope 202 can, in some instances, be used without such devices. In some embodiments, the scope 202 may include telescopic components, such as an inner leader portion and an outer sheath portion, which can be manipulated to telescopically extend the scope.

[0097] The scope 202 may be configured to be articulated, such as with respect to at least the distal end 208 of the scope 202. For example, the scope 202 may be configured to move in various degrees of freedom (DOF), such as 3-DOF (e.g., x-, y-, and z-movement), 4-DOF (e.g., x-, y-, z-, and roll-movement), or 6-DOF (e.g., x-, y-, z-, pitch-, yaw-, and roll-movement), which may be coupled through the bending properties of the scope 202. By way of example, the tip 208 may be deflected on a yaw axis 212, a pitch axis 214, and / or a roll axis 216 (also referred to as the "longitudinal axis 216" or "z-axis 216"). The tip 208 or body 218 of the scope 202 may be extended or translated along the longitudinal axis 216, the x-axis 220, or the y-axis 222. In embodiments in which the scope 202 is equipped with position sensors, the position sensors may provide position information such as 3-DOF position information (e.g., x, y, and z coordinates), 5-DOF position information (e.g., x, y, and z coordinates and pitch and yaw angles), 6-DOF position information (e.g., x, y, and z coordinates and pitch, yaw, and roll angles), etc.

[0098] For robotic implementations, the robotic arm of the robotic system may be configured / configurable to manipulate the scope 202 using one or more elongated movement members. The elongated movement members may include one or more pull wires (e.g., pull wires or push wires), cables, fibers, and / or flexible shafts. For example, the robotic arm may be configured to actuate multiple pull wires (not shown) coupled to the scope 202 to deflect the tip 208 of the scope 202. The pull wires may include any suitable or desirable material, such as metallic and non-metallic materials, such as stainless steel, Kevlar, tungsten, carbon fiber, and the like. In some embodiments, the scope 202 is configured to exhibit non-linear behavior in response to forces applied by the elongated movement members. The non-linear behavior may be based on the stiffness and compressibility of the scope 202 and variability in slack or stiffness between different elongated movement members.

[0099] In some embodiments, the scope 202 includes sensors (sometimes referred to as “position sensors”) configured to generate and / or transmit sensor data to another device. The sensor data (sometimes referred to as “sensor position data”) may indicate the position and / or orientation of the medical instrument 202 (e.g., its distal end 208) and / or may be used to determine / estimate the position / or orientation of the medical instrument 202. For example, the sensors may provide sensor data to a control system, which then uses it to determine the position and / or orientation of the scope 202. The sensors may be positioned at the distal end 208 and / or elsewhere on the scope 202. In some embodiments, the sensors may include electromagnetic (EM) sensors having a coil of conductive material or other forms / embodiments of antennas. FIG. 2 shows an EM field generator 224 configured to propagate an EM field 226 that is detected by the EM sensor on the scope 202. The magnetic field 226 may induce a small current in the coil of the EM position sensor, which may be analyzed to determine the distance and / or angle / orientation between the EM sensor and the EM field generator 224. Alternatively, or in addition, the scope 202 may include other types of sensors, such as shape-sensing fibers, accelerometer(s), gyroscope(s), satellite-based positioning sensor(s) (e.g., Global Positioning System (GPS) sensors), radio frequency transceiver(s), etc.

[0100] The scope 202 may be controllable in any suitable or desirable manner, either based on user input or automatically. The controls 228, 230 (also referred to as "I / O devices") provide examples that may be used to receive user input. In an example, the control 228 is located on the proximal handle of the scope 202. Additionally, in an example, the control 230 is implemented as a controller, such as in the example of FIG. 2. In some embodiments, the control 228 and / or the control 230 are used in the context of robotics, such as the medical system 100 of FIG. 1. For example, the control 230 may receive input from a user and provide input signals to control a robotic system connected to the scope 202. While the controls 228, 230 are shown as handheld controllers, input from a user may be received using any type of I / O device, such as a touchscreen / pad, a mouse, a keyboard, a microphone, etc.

[0101] Exemplary Catheters 3 is a diagram illustrating a catheter 302 disposed in a patient's kidney, in accordance with one or more embodiments of the present disclosure. Catheter 302 may represent catheter 130 of FIG. 1 and / or any other catheter described herein. For example, catheter 302 may be used in a ureteroscopic procedure to treat and / or remove kidney stones within kidney 304. However, catheter 302 may also be used in other types of procedures. As discussed above, ureteroscopic procedures and / or other types of procedures may be performed at least in part manually and / or at least in part using robotic technology, such as with medical system 100 shown in FIG. 1.

[0102] In some embodiments, the catheter 302 is configured to provide irrigation and / or suction to the anatomical site. For example, the catheter 302 may include a lumen 306 for implementing an aspiration outflow channel, such as for removing one or more kidney stone fragments, and / or an irrigation inflow channel for providing fluids from the kidney 304. In some embodiments, the catheter 302 is implemented with another medical instrument(s) 308 for providing irrigation / suction to the anatomical site. For example, the catheter 302 may include rigid or flexible tube(s) sized to pass through the medical instrument 308, which may include an outer sheath, an introducer, a nephroscope, or another luminal device. A channel 310 may be formed in the space between the outer wall of the catheter 302 and the inner wall / sheath of the medical instrument 308. The channel 310 may provide suction (and / or, in some cases, irrigation). With the catheter 302 positioned within the medical device 308, the shaft(s) / sheath(s) of the catheter 302 and medical device 308 may be generally concentric. The catheter 302 and medical device 308 may have a generally circular cross-sectional shape over at least a portion thereof. The catheter 302 and / or medical device 308 may provide both suction and irrigation, although irrigation and suction may or may not be provided via the same device(s). For example, a scope may provide irrigation, and the catheter 302 / medical device 308 may provide suction. In some embodiments, the catheter 302 is configured to access an anatomical site via a percutaneous access pathway and / or provide irrigation / suction to an anatomical site.

[0103] The catheter 302 may include one or more markings 312 (sometimes referred to as “one or more orientation markings 312”) on the tip 314 of the catheter 302. The one or more markings 312 may be used to calibrate the catheter 302 or otherwise visualize the orientation of the tip 314 of the catheter 302. The one or more markings 312 may include deformation(s) (e.g., indentations, holes, notches, flats, etc.), coloring (e.g., one side of the tip is colored a first color and the other side a different color, colored Roman numerals, etc.), image(s) (e.g., numbers, letters, shapes, or other images), etc. In the example of FIG. 3 , the one or more markings 312 are implemented in the form of an indentation of the Roman numeral I on one side of the tip 314 and the Roman numeral II on the opposite side of the tip 314. In examples, one or more markings 312 may be implemented on both the outer diameter and inner diameter of the tip 314. This may assist in visually identifying the orientation of the one or more markings 312 for various orientations / positions of the tip 314. In some embodiments, the recessed markings may be filled with a substance to provide a relatively smooth surface on the tip 314. In some embodiments, the tip 314 may include a particular shape for implementing the one or more markings 312, such as a circular cross-section with a flat portion. Additionally, in some embodiments, the one or more markings 312 are implemented in a particular manner that is more easily detectable by image processing techniques, such as a pattern, image (e.g., a QR code), etc. While the one or more markings 312 are implemented on the outer edge / outer diameter of the catheter 302, one or more markings 312 may additionally or alternatively be implemented on the inner edge / inner diameter of the tip 314 of the catheter 302. This can provide a smooth outer edge for the catheter 302, which in some cases can be advantageous for navigating the catheter 302 and / or for avoiding damage to the patient's anatomy. Although the one or more markings 312 are illustrated on the tip 314 of the catheter 302, the one or more markings 312 can be located elsewhere, such as on the body 316 of the catheter 302.

[0104] The catheter 302 may also include one or more markings 318 (sometimes referred to as “one or more depth markings 318”) on the body of the catheter 302. The one or more markings 318 may be used to determine how deeply the catheter 302 has been inserted into the anatomy. For example, the catheter 302 may implement multiple markings 318 located at different distances from the tip 314 of the catheter 302. If a physician / control system can see / detect a first marking located a certain distance relative to the tip 314 of the catheter 302, the physician / control system may determine that the catheter 302 has been inserted at least that certain distance into the patient. The one or more markings 318 may be implemented in a manner similar to the one or more markings 312 described above, such as with deformation(s) (e.g., dents, holes, notches, flats, etc.), coloring, image(s) (e.g., numbers, letters, shapes, or other images), etc. 3, markings 318 are shown with colored markings extending around the circumference of body 316 of catheter 302. In embodiments, one or more markings 318 (and / or one or more markings 312) may be detected / visible using fluoroscopy, an ultrasound camera, or the like.

[0105] The catheter 302 may be configured to be articulated, such as with respect to at least the distal end 314 of the catheter 302. For example, the catheter 302 may be configured to move in various degrees of freedom (DOF), such as 3-DOF (e.g., x-, y-, and z-movement), 4-DOF (e.g., x-, y-, z-, and roll-movement), or 6-DOF (e.g., x-, y-, z-, pitch-, yaw-, and roll-movement), which may be coupled through bending properties of the catheter 302. By way of example, the tip 314 may be deflected on a yaw axis 320, a pitch axis 322, and / or a roll axis 324 (also referred to as the "longitudinal axis 324" or the "z-axis 324"). In some embodiments, the catheter 302 is configured to move in two independent degrees of freedom (DOF) (e.g., using yaw and pitch pull wires) and / or the catheter 302 is not configured to roll. However, the catheter 302 may be configured to allow roll and / or other types of movement in some cases. The tip 314 or body 316 of the catheter 302 may be extended or translated along a longitudinal axis 324, an x-axis 326, or a y-axis 328. In embodiments in which the catheter 302 is equipped with position sensors, the position sensors may provide position information such as 3-DOF position information (e.g., x, y, and z coordinates), 5-DOF position information (e.g., x, y, and z coordinates and pitch and yaw angles), 6-DOF position information (e.g., x, y, and z coordinates and pitch, yaw, and roll angles), etc.

[0106] For robotic implementations, the robotic arm of the robotic system may be configured / configurable to manipulate the catheter 302 using one or more elongated motion members. The elongated motion members may include one or more pull wires (e.g., pull wires or push wires), cables, fibers, and / or flexible shafts. For example, the robotic arm may be configured to actuate multiple pull wires (not shown) coupled to the catheter 302 to deflect the tip 314 of the catheter 302. The pull wires may include any suitable or desirable material, such as metallic and non-metallic materials, such as stainless steel, Kevlar, tungsten, carbon fiber, and the like. In some embodiments, the catheter 302 is configured to exhibit non-linear behavior in response to forces applied by the elongated motion members. The non-linear behavior may be based on the stiffness and compressibility of the catheter 302 and slack or stiffness variability between different elongated motion members.

[0107] In some embodiments, the tip 314 of the catheter 302 is implemented with a material that avoids degradation in certain contexts, such as catastrophic degradation. For example, the tip 314 may be implemented with stainless steel (or other types of steel), titanium, tungsten, and / or other materials (which may have a relatively high melting point) that can generally maintain their structure even when a laser beam from a scope unintentionally and / or frequently contacts the tip 314 of the catheter 302. However, the tip 314 and / or any other portions of the catheter 302 may be implemented with other materials.

[0108] Although some embodiments are discussed in the context of the catheter 302 being implemented without a position sensor, in other embodiments, the catheter 302 includes a position sensor configured to generate and / or transmit sensor data to another device. The sensor data may indicate the position and / or orientation of the catheter 302 (e.g., its distal end 314) and / or may be used to determine / estimate the position / orientation of the catheter 302. The sensor may be positioned at the distal end 314 of the catheter 302 and / or at another location. In some embodiments, the position sensor may include an electromagnetic (EM) sensor having a coil of conductive material or other forms / embodiments of an antenna. Alternatively, or in addition, the position sensor may include other types of sensors, such as shape-sensing fiber, accelerometer(s), gyroscope(s), satellite-based positioning sensor(s) (e.g., Global Positioning System (GPS) sensors), radio frequency transceiver(s), etc.

[0109] 3, in some embodiments, the catheter 302 may include imaging device(s) configured to acquire image data, such as image data representative of the patient's internal anatomy. For example, the catheter 302 may include an imaging device located on the tip 314 of the catheter 302. In some cases, image data from the imaging device may be used to drive the catheter 302 from the perspective of the catheter 302.

[0110] Catheter 302 may be controllable in any suitable or desirable manner, either based on user input or automatically. Controllers 330, 332 (also referred to as "I / O devices") provide examples in which they may be used to receive user input. In an example, controller 330 is located on the proximal handle of catheter 302. Additionally, in an example, controller 332 is implemented as a controller, such as in the example of FIG. 2. In some embodiments, controller 330 and / or controller 332 are used in the context of robotics, such as medical system 100 of FIG. 1. For example, controller 332 may receive input from a user and provide input signals to control a robotic system connected to catheter 302. While controllers 330, 332 are shown as handheld controllers, user input may be received using any type of I / O device, such as a touchscreen / pad, mouse, keyboard, microphone, etc.

[0111] Exemplary Control Modes 4-6 illustrate example implementations of control / drive modes for driving a medical instrument from the perspective of another medical instrument, according to one or more embodiments. These control modes are described in the context of a physician 402 using an input / output device 404 to drive a catheter 406 within a kidney 408 from the perspective of a scope 410 to remove a kidney stone 412. A user interface 414 can be presented to assist the physician 402 in driving the catheter 406. As shown, the user interface 414 may present image data 416 from the perspective of the scope 410 (e.g., image data acquired by the scope 410). In these examples, the physician 402 may select a control mode to implement for various orientations / positions of the catheter 406 relative to the scope 410. However, the control mode may be selected in various manners, as described herein.

[0112] 4-6 illustrate direct and / or inverted control modes for navigating the catheter 406 for various orientations of the catheter 406 relative to the scope 410. While the inverted control mode is often used when the catheter 406 faces the scope 410 and the direct control mode may often be used when the catheter 406 and scope 410 face the same direction, the inverted and / or direct control modes may be implemented in any context, such as other orientations of the instruments relative to each other and / or relative to the kidney stone 412. Additionally, in some embodiments, multiple drive modes may be implemented during the same procedure, such as by the physician 402 switching between drive modes as the catheter 406 / scope 410 are repositioned during the procedure.

[0113] These examples illustrate that the catheter 406 can enter the kidney 408 from a variety of positions relative to the kidney stone 412. For example, in FIGS. 4-1 and 4-2 and 6-1 and 6-2, the catheter 406 enters the kidney 408 behind the kidney stone 412 relative to the scope 410. In FIGS. 5-1 and 5-2, the catheter 406 accesses the kidney stone 412 from substantially the same orientation as the scope 410. However, the catheter 406 may access the kidney stone 412 from a variety of other positions / paths relative to the kidney 408 and / or the scope 410. Additionally, the scope 410 may be positioned in a variety of other positions.

[0114] 4-1 and 4-2 illustrate actuation of the catheter 406 in direct control mode (also called "parallel mode") when the catheter 406 is directly facing the scope 410. As such, icon 418 indicates that the direct control mode is selected. In this example, the physician 402 selects a directional control on the I / O device 404 that is associated with a right direction for the I / O device 404. With the direct control mode in place, the catheter 406 can be controlled in a corresponding manner relative to the catheter 406. That is, the catheter 406 moves to the right relative to the coordinate system / control frame of the catheter 406. As shown in FIG. 4-2, the user interface 414 indicates that the catheter 406 is moving to the left because the catheter 406 is directly facing the scope 410.

[0115] FIGS. 5-1 and 5-2 illustrate driving the catheter 406 in direct control mode when the catheter 406 and the scope 410 are pointing in substantially the same direction. In this example, the physician 402 again selects a direction control on the I / O device 404 associated with the right direction relative to the I / O device 404. With direct control mode engaged, the catheter 406 moves to the right relative to the coordinate system / control frame of the catheter 406. As shown in FIG. 5-2, the user interface 414 indicates that the catheter 406 is moving to the right because the catheter 406 is generally pointing in the same direction. In embodiments, direct control mode may be implemented more frequently in the context of FIGS. 5-1 and 5-2, where the catheter 406 and the scope 410 are substantially facing the kidney stone 412 from the same area, because it may provide a more user-friendly view of the movement of the catheter 406 via the user interface 414.

[0116] 6-1 and 6-2 illustrate driving the catheter 406 in an inverted control mode (also called a "mirror mode") when the catheter 406 is directly facing the scope 410. An icon 420 indicates that the inverted control mode is selected. In this example, the physician 402 selects a direction control on the I / O device 404 associated with a left direction relative to the I / O device 404. With the inverted control mode enabled, the catheter 406 can be controlled in an inverted manner relative to the catheter 406. That is, the catheter 406 moves to the left relative to the coordinate system / control frame of the catheter 406. To do this, the horizontal component of the movement direction of the coordinate system / control frame can be inverted, as described in more detail below. As shown in FIG. 6-2, the user interface 414 indicates that the catheter 406 is moving to the left because the catheter 406 is directly facing the scope 410. In an embodiment, the inverted control mode may be implemented more frequently in the context of Figures 6-1 and 6-2, where the catheter 406 and scope 410 are substantially directly facing each other, because this may provide a more user-friendly view of the movement of the catheter 406 via the user interface 414.

[0117] Exemplary Instrument Drive Interface 7 is a diagram illustrating an example interface 702 for controlling / navigating a medical instrument, according to one or more embodiments. For example, interface 702 may provide information to assist a physician in driving a scope from the scope's perspective, driving a catheter 704 from the scope's perspective, and / or using another medical instrument. As described below, interface 702 may include one or more interface elements (e.g., icons) that can be selected through a touchscreen, controller, mouse, trackpad, or another type of I / O device.

[0118] As shown, interface 702 may present image data 706, which may be generally from the perspective of the scope. That is, image data 706 may depict at least a portion of the field of view of an imaging device on the scope. In some embodiments, image data 706 is presented as an original image view (as described above), while in other embodiments, image data 706 is presented as a rotated image view. Although interface 702 generally presents image data from the perspective of the scope, interface 702 may additionally or alternatively present image data from the perspective of another medical instrument. For example, if catheter 704 is being driven from the perspective of catheter 704, interface 702 may present image data from the perspective of catheter 704.

[0119] The interface 702 may include interface elements 708, 710, 712 for selecting a medical instrument to control. The interface element 708 allows for control of the scope, the interface element 710 allows for control of a medical instrument (also referred to as a "tool") associated with the scope, and the interface element 712 allows for control of the catheter 704. The medical instruments associated with the scope may include a laser, a cutting instrument, a lithotriptor, a basket device, forceps, etc. As discussed herein, such medical instruments may be deployed through the working channel of the scope to break up a kidney stone 714 for removal by the catheter 704.

[0120] Interface 702 may include interface element 716 that enables a direct drive / control mode and interface element 718 that enables an inverted drive / control mode. For example, when driving catheter 704 from the scope's point of view (i.e., interface element 712 is selected) and direct control mode is enabled (i.e., interface element 716 is selected), catheter 704 may be driven in a corresponding manner relative to the point of view of catheter 704. In contrast, when driving catheter 704 from the scope's point of view (i.e., interface element 712 is selected) and inverted control mode is enabled (i.e., interface element 718 is selected), catheter 704 may be driven in an inverted manner relative to the point of view of catheter 704.

[0121] The interface 702 may also include other information to assist the user in controlling the medical instrument. For example, articulation bars 720, 722 may be presented around the image data 706 to allow the user to view the amount of articulation associated with the selected medical instrument. The top / bottom articulation bar 722 may indicate the amount of vertical articulation (e.g., how far the medical instrument has been moved vertically). For example, the top / bottom articulation bar 722 may be positioned above or below the image data 706 and / or may expand / contract in length to indicate vertical articulation of the medical instrument. In the example of FIG. 7 , the top / bottom articulation bar 722 is positioned below the image data 706 to indicate that the catheter 704 is articulating downward. The right / left articulation bar 720 may indicate the amount of horizontal articulation (e.g., how far the medical instrument has been moved horizontally). For example, the right / left articulation bar 720 may be positioned to the right or left of the image data 706 and / or may expand / contract in length to indicate horizontal articulation of the medical instrument. 7 , right / left articulation bar 720 is positioned to the left of image data 706 to indicate that catheter 704 is articulated to the right in mirror mode. Thus, when catheter 704 is being controlled, articulation bars 720, 722 can indicate the amount of movement of catheter 704, such as how far catheter 704 has articulated right, left, up, or down within interface 704 relative to the longitudinal axis of catheter 704. In some cases, when the articulation bar is shown at the shortest allowable length (and / or without the articulation bar), this can indicate that catheter 704 is aligned with the longitudinal axis of catheter 704 (e.g., no articulation).

[0122] In the example of FIG. 7 , the articulation bars 720, 722 are substantially centered on the image data 706 and expand / contract in length as the associated medical instrument articulates. For example, the articulation bar 722 along the bottom edge of the image data 706 can increase / decrease in length while maintaining alignment with a vertical axis through the center of the image data 706 as the medical instrument is navigated up / down within the interface 702. Similarly, the articulation bars 720 along the side edges of the image data 706 can increase / decrease in length while maintaining alignment with a horizontal axis through the center of the image data 706 as the medical instrument is navigated right / left within the interface 702. However, in other examples, the articulation bars 720, 722 can be manipulated in other ways to indicate articulation of the medical instrument, such as moving vertically / horizontally relative to the interface 702 without changing in length.

[0123] In some embodiments, one or more of the articulation bars 720, 722 may be updated based on calibration of the control scheme. For example, if the articulation bars 720, 722 initially indicate that the catheter is articulated in one direction and the control scheme for the catheter is adjusted, the articulation bars 720, 722 may be updated to indicate that the catheter is articulated in a different direction that accurately reflects the actual orientation of the catheter relative to the scope. In contrast, in some embodiments, the orientation of the image data 706 may be maintained after calibration of the control scheme. However, in some cases, the orientation of the image data 706 may be updated to reflect the updated control scheme.

[0124] The interface 702 may also present a progress bar 724 to indicate the position of the medical instrument currently being controlled relative to a target location or another landmark. For example, the progress bar 724 may indicate the proximity of the scope to a target location tagged / designated by the scope, such as a papilla designated as an entry point for the catheter 704 to enter the kidney. In some cases, a full fill of the progress bar 724 may indicate that the scope is at the target location. In other cases, the progress bar 724 may include markings along the bar to indicate the location of the target location. Additionally or alternatively, the progress bar 724 may indicate other progress information, such as the proximity of the scope to the catheter 704 / kidney stone 714, the proximity of the catheter 704 to the scope / target location / kidney stone 714, or the proximity of a tool to the catheter 704 / target location / kidney stone 714. In some embodiments, the progress bar 724 may include information about the selected medical instrument (e.g., the medical instrument currently being actuated).

[0125] 7, in some embodiments, interface 702 may present image data acquired by an external imaging device, such as one or more X-ray images, CT images, etc. Such image data may depict catheter 704, a scope, another medical instrument, the patient's internal anatomy, etc. In some examples, the image data may be acquired as part of a fluoroscopy procedure.

[0126] Exemplary Flow Diagram: Equipment Control Process 8 is an example flow diagram of a process 800 for controlling a medical instrument from the perspective of another medical instrument, according to one or more embodiments. The various operations associated with process 800 may be performed by control circuitry implemented in any or combination of the devices / systems discussed herein, such as control system 150, robotic system 110, table 170, scope 120, catheter 130, and / or another device of FIG. 1.

[0127] At block 802, process 800 may include receiving image data from a first instrument. For example, the control circuitry may receive image data from a first instrument configured to access an anatomical site via a first access pathway. The image data may represent the anatomical site and / or a second instrument, the second instrument configured to access the anatomical site via the second access pathway. In some embodiments, the first instrument is a scope and / or the second instrument is a catheter.

[0128] At block 804, process 800 may include causing a visual representation of the image data to be displayed. For example, the control circuitry may generate a visual representation of the image data and / or user interface data representing a user interface. The control circuitry may cause the user interface and / or the visual representation to be displayed based on the user interface data.

[0129] At block 806, process 800 may include configuring the control system to control the first instrument or the second instrument. For example, the control circuit may receive an input signal from an input device indicating to switch control from the second instrument to the first instrument, or vice versa. Based on such input signal, the control circuit may configure the control system to control the first instrument or the second instrument. Thus, in some embodiments, the same input device may be used to control the first instrument and the second instrument.

[0130] At block 808, process 800 may include receiving a directional input signal from an input device. For example, a user may provide input via the input device, which may include multiple input controls, each associated with a particular direction relative to the input device. The input device may generate a directional input signal based on the input and send the directional input signal to the control circuit. The directional input signal may be associated with a direction relative to the input device. The direction may be associated with a horizontal / vertical component.

[0131] At block 810, process 800 may include determining an orientation and / or position of the first instrument relative to the second instrument. For example, the control circuitry may perform one or more location techniques to track the orientation / position of the first instrument and the orientation / position of the second instrument. The control circuitry may use such information to determine the orientation / position of the first instrument relative to the second instrument.

[0132] At block 812, process 800 may include determining a control mode for the first / second appliance. For example, the control circuitry may determine a direct control mode associated with controlling the appliance in a direct manner relative to the received input and / or an inverted control mode associated with controlling the appliance in an inverted manner relative to the received input. In one embodiment, the control circuitry may receive an input signal indicating a control mode from among a plurality of control modes. The input signal may be received from an input device or another I / O device. In another embodiment, the control circuitry may automatically determine a control mode from among a plurality of control modes based on an orientation / position of the first appliance relative to the second appliance. For example, the control circuitry may select a predetermined control mode associated with a particular orientation of the plurality of appliances relative to each other.

[0133] At block 812, process 800 may include controlling the movement of the first instrument / second instrument. For example, the control circuitry may generate a control signal based on the control mode and / or the directional input signal. The control circuitry may send a control signal to the robotic system to cause the robotic system to manipulate the first instrument / second instrument, resulting in movement of the first instrument / second instrument at the anatomical site. For example, if a directional input signal associated with a first direction relative to the input device is received and an inverted control mode is determined, the second instrument may be controlled to move in a second direction relative to a frame of reference of the second instrument. The second direction may be associated with horizontal / vertical components having an opposite sign to the horizontal / vertical components associated with the first direction. In contrast, if a directional input signal associated with the first direction relative to the input device is received and a direct control mode is determined, the second instrument may be controlled to move in a second direction relative to a frame of reference of the second instrument. The second direction may be associated with horizontal / vertical components having the same sign as the horizontal / vertical components associated with the first direction. The reference system may include a control frame and / or a coordinate system.

[0134] In some embodiments, the control circuitry may control the insertion / retraction of the first / second instrument based on the determined control mode. For example, if a directional input signal associated with insertion is received and a reverse control mode is determined, the control circuitry may control the second instrument to be retracted. Conversely, if a directional input signal associated with retraction is received and a reverse control mode is determined, the control circuitry may control the second instrument to be inserted.

[0135] In some embodiments, one or more of blocks 802-814 may be repeated any number of times to control the first instrument and / or the second instrument in one or more control modes based on input received via the input device.

[0136] Exemplary Coordinate Systems / Control Frames and Control Implementations 9-1 and 9-2 illustrate exemplary implementations of driving a medical instrument from a first-person perspective for different control modes relative to the medical instrument's coordinate system, according to one or more embodiments. In these figures, a catheter 902 is shown substantially from the perspective of the catheter 902, with a coordinate system 904 associated with the tip of the catheter 902. While various conventions for coordinate systems can be used, for ease of illustration, the description herein often refers to the "forward" direction (e.g., insertion / retraction) as corresponding to positive z, the "right" direction as corresponding to positive x, and the "up" direction as corresponding to positive y. The z vector can extend along the longitudinal axis of the medical instrument. For ease of illustration, the z vector of the coordinate system is not shown in FIGS. 9-1 and 9-2. Although described in the context of a coordinate system, the coordinate system 904 can, in some cases, represent a control frame.

[0137] Catheter 902 may be controlled, such as by selecting a direction control on I / O device 906, based on input received via I / O device 906, where the input is associated with a direction / vector 908 relative to I / O device 906, which has a negative x value (horizontal component) and a positive y value (vertical component) relative to I / O device 906. The horizontal / vertical components may indicate direction and / or magnitude.

[0138] As shown, catheter 902 may move in different ways (relative to coordinate system 904) in different control modes. In direct control mode (the mode shown in FIG. 9-1), catheter 902 is controlled to move in direction / vector 910 having a negative x-value and a positive y-value relative to coordinate system 904. In contrast, for the same input, in inverted control mode (the mode shown in FIG. 9-2), catheter 902 is controlled to move in direction / vector 912 having a positive x-value and a positive y-value relative to coordinate system 904. Here, direction / vector 912 has the same magnitude (or a multiple) of direction 908, but has an x-value with the opposite sign of the x-value of direction / vector 908. Thus, the horizontal component may be inverted in the inverted control mode.

[0139] In some embodiments, such as in third-person driving or otherwise, a control reference frame is implemented to facilitate movement of the medical instrument. FIGS. 10-1 and 10-2 illustrate exemplary implementations of driving a medical instrument from a third-person perspective for different control modes relative to the control frame of the medical instrument, according to one or more embodiments. In these figures, the catheter 902 from FIGS. 9-1 and 9-2 is oriented substantially directly opposite the scope 1002. For ease of illustration, the catheter 902 is shown as having only a tip portion, rather than a body portion. FIGS. 10-1 and 10-2 show the catheter 902 and scope 1002 positioned in a manner that might occur during a medical procedure.

[0140] 10-1 and 10-2, the catheter 902 is controlled from the perspective of the scope 1002 with respect to a control frame 1004 (also referred to as the "control reference frame 1004") associated with the catheter 902. The control frame 1004 may include an abstracted coordinate system / set of vectors used to control the catheter 902, such as an x-vector, a y-vector, and a z-vector. For ease of illustration, the z-vector of the control frame 1004 is not shown in FIGS. 10-1 and 10-2. The control frame 1004 may be correlated to the orientation of image data displayed via a user interface to drive the catheter 902. For example, the control frame 1004 may represent the orientation of image data acquired from the scope 1002 as displayed within the interface (e.g., the positive y-axis of the control frame 1004 is aligned with the y-axis of the interface). That is, the image data displayed through the interface depicts the catheter 902 in the orientation shown in FIGS. 10-1 and 10-2. The control frame 1004 can be aligned or offset by an angle with respect to the coordinate system of the catheter 902 (not shown in FIGS. 10-1 and 10-2). As mentioned above, the coordinate system can be fixed relative to the instrument (e.g., fixed to markings on the catheter 902), while the control frame can change (e.g., rotate as the scope 1002 rolls and / or the orientation of the image data changes in the interface).

[0141] 10-1 and 10-2 also show a coordinate system 1006 of the scope 1002 associated with the tip of the scope 1002. The coordinate system 1006 can be relative to an imaging device located at the tip of the scope 1002.

[0142] Catheter 902 may be controlled to move relative to control frame 1004 in different manners for different control modes. In the example of FIGS. 10-1 and 10-2, an input associated with a direction / vector 908 relative to I / O device 906 having a negative x value (horizontal component) and a positive y value (vertical component) is received via I / O device 906. In direct control mode (the mode shown in FIG. 10-1), catheter 902 is controlled to move in direction / vector 1008 having a negative x value and a positive y value relative to coordinate system 1004. In contrast, for the same input, in inverse control mode (the mode shown in FIG. 10-2), catheter 902 is controlled to move in direction / vector 1010 having a positive x value and a positive y value relative to control frame 1004. Here, direction 1010 is associated with an x ​​value that has the opposite sign to the x value of direction 908 (and the opposite sign to the x value of direction 1008 from FIG. 10-1). Thus, the horizontal component can be inverted in the case of the inverted control mode.

[0143] In some embodiments, a vector 1008 may be determined relative to the control frame 1004 based on the vector 908 to determine how to control the catheter 902 relative to the control frame 1004. For example, the direction / magnitude of the vector 1008 may be determined based on the direction / magnitude of the vector 908. In cases where an inverted control mode is implemented, the direction of the vector 1008 may be determined by inverting the horizontal / vertical components (e.g., switching the sign of the x-component of the vector 908). In some embodiments, the vector 908 is indirectly (or in some cases directly) mapped / transformed into the coordinate system and / or control frame of the catheter 902 using one or more algorithms.

[0144] As described above, input received via the I / O device 906 for the catheter 902 may drive the articulation of the catheter 902, such as the velocity of the articulation of the catheter 902. For example, if input is received via an input component (e.g., a joystick) on the I / O device 906, the catheter 902 is articulated. Then, if the input component is left untouched, the catheter 902 may maintain its current position / orientation (e.g., remain in its current articulation position). Thus, the input on the I / O device 906 can cause the catheter 902 to be controlled by adding / subtracting to the existing articulation of the catheter 902.

[0145] Although the invert control modes of the examples of Figures 9-1, 9-2, 10-1, and 10-2 are described in the context of inverting horizontal components (e.g., x values), the invert control mode can alternatively or additionally change vertical components (e.g., change the sign associated with the y value) and / or change components associated with the insertion / retreat (e.g., change the sign associated with the z value).

[0146] Exemplary Calibration Interface 11-12 show an exemplary interface 1102 for calibrating a control scheme for a medical instrument, according to one or more embodiments. For example, interface 1102 may provide information for adjusting orientation information regarding the distal end of a scope relative to the distal end of a catheter 1104. In some embodiments, when a user notices that the movement of the medical instrument does not perfectly correlate with input provided via an input device, the user can navigate along interface 1102. For example, if a user is driving catheter 1104 from the perspective of the scope and notices that catheter 1104 moves to the right in the instrument driving interface (e.g., interface 702 of FIG. 7 ) when an input of up is provided via the input device, the user can navigate along interface 1102 to calibrate catheter 1104 and / or the scope. However, interface 1102 may be accessed at other times and / or in different ways.

[0147] As shown, interface 1102 may present image data 1106 from the perspective of a scope within the vicinity of catheter 1104. That is, image data 1106 may depict at least a portion of the field of view of an imaging device on the scope. Image data 1106 may be presented in an original image view or a rotated image view. Although interface 1102 generally presents image data from the perspective of the scope, interface 1102 may additionally or alternatively present image data from the perspective of another medical instrument.

[0148] Interface 1102 may also present interface elements 1108, 1110, 1112 for selecting a medical device to calibrate / control. Interface element 1108 allows for calibration / control of a scope, interface element 1110 allows for calibration / control of a medical instrument (also called a "tool") associated with the scope, and interface element 1112 allows for calibration / control of catheter 1104. In the example of FIG. 11 , interface element 1112 is selected so that catheter 1104 may be calibrated. For example, a user may calibrate a control scheme / control reference frame associated with catheter 1104. However, interface element 1108 / interface element 1110 may alternatively be selected to calibrate a control scheme.

[0149] To calibrate the control scheme / control frame of the catheter 1102, the interface 1102 may provide an alignment indicator 1116 that represents the orientation of the tip of the catheter 1104. For example, the alignment indicator 1116 may represent the coordinate system of the catheter 1104 relative to the coordinate system of the scope. The alignment indicator 1116 may include a ring 1116(A) and one or more marking indicators 1116(B) that represent the estimated orientation, relative to the scope, of one or more markings 1118 located on the catheter 1104. For example, a control system (not shown) may attempt to track the roll of the scope and the roll of the catheter 1104 relative to each other. Based on such information, the control system may present a marking indicator 1116(B) to indicate the estimated roll of the tip of the catheter 1104 relative to the estimated roll of the tip of the scope. Marking indicator 1116 (B) may be positioned around ring 1116 (A) to indicate the probable orientation of marking 1118 on the tip of catheter 1104 .

[0150] If desired, the user can provide input to adjust the orientation of the alignment indicator 1116 to more closely match the orientation of the marking 1118 on the tip of the catheter 1104. For example, as shown in FIG. 11 , the orientation of the marking indicator 1116(B) is not aligned with the marking 1118 on the catheter 1104. Therefore, the user can provide input to rotate the alignment indicator 1116 so that the marking indicator 1116(B) is more aligned with the orientation of the marking 1118 displayed in the image data 1106, as shown in FIG. 12 . The user can rotate the alignment indicator 1116 by any angle to a position that the user believes is aligned with the marking 1118 on the catheter 1104.

[0151] A user may provide input via any type of input device to rotate the alignment indicator 1116. For example, a user may provide input via a right directional control on a controller (or touchscreen, etc.) that may be associated with the right directional interface element 1120 to rotate the alignment indicator 1116 clockwise. Additionally, a user may provide input via a left directional control on a controller (or touchscreen, etc.) that may be associated with the left directional interface element 1122 to rotate the alignment indicator 1116 counterclockwise. However, other modes of input and / or rotation may also be implemented. When rotated, a user may provide input indicating that the marking indicator 1116(B) is aligned with the marking 1118. Adjustments to the alignment indicator 1116 may be used to update / calibrate the control scheme / control reference frame for the catheter 1104.

[0152] The orientation of the marking indicator 1116(B) is generally initially displayed based on the estimated orientation of the marking 1118 on the catheter 1104 (as shown in FIG. 11 ), although the marking indicator 1116(B) can also be displayed in another orientation. For example, the marking indicator 1116(B) may initially be displayed in a predetermined orientation, such as a predetermined position around the ring 1116(A). Similar to what was described above, the user can adjust the marking indicator 1116(B) to match the orientation of the marking 1118 on the catheter 1104. The control system can then use the specified orientation of the catheter 1104 relative to the scope to adjust / calibrate the control scheme / control reference frame of the catheter 1104.

[0153] Additionally, although alignment indicator 1116 is implemented in the context of Figures 11-12, other visual elements may additionally or alternatively be implemented. For example, a slider may be presented to allow the user to adjust the orientation of catheter 1104 relative to the scope.

[0154] Exemplary Flow Diagram: Instrument Calibration Process 13 is an example flow diagram of a process 1300 for calibrating a control scheme / control reference frame for a medical instrument, according to one or more embodiments. The various operations associated with process 1300 may be performed by control circuitry implemented in any or combination of the devices / systems discussed herein, such as control system 150, robotic system 110, table 170, scope 120, catheter 130, and / or another device of FIG.

[0155] At block 1302, process 1300 may include receiving image data representing an anatomical region. For example, a first instrument may be configured to access the anatomical region via a first access path, and a second instrument may be configured to access the anatomical region via a second access path. The second instrument may generate image data and transmit the image data to the control circuitry. The control circuitry may receive the image data from the second instrument. The image data may represent the anatomical region and / or the first instrument. In some embodiments, the first instrument is a catheter and / or the second instrument is a scope.

[0156] At block 1304, process 1300 may include executing one or more of blocks 1306-1322. For example, control circuitry may execute one or more of blocks 1306-1322 in parallel, serially, etc.

[0157] At block 1306, process 1300 may include causing the image data to be displayed. For example, the control circuitry may cause a graphical representation of the image data (received at block 1302) to be displayed via a user interface.

[0158] At block 1308, process 1300 may include displaying an alignment indicator. For example, the control circuitry may cause the alignment indicator to be displayed via the interface, the alignment indicator representing the orientation / first coordinate system of the first instrument (e.g., relative to the second instrument). In some embodiments, the alignment indicator includes a ring and / or one or more marking indicators representing the orientation / first coordinate system of the first instrument. In some embodiments, the alignment indicator represents an estimated roll of the distal end of the first instrument.

[0159] At block 1310, process 1300 may include receiving an input indicating an orientation of the first instrument, such as a roll of the distal end of the first instrument. For example, the control circuitry may receive an input including an adjustment to the alignment indicator. The input may include rotating the alignment indicator to orient one or more marking indicators of the alignment indicator to one or more markings on the distal end of the first instrument. Alternatively or additionally, the control circuitry may receive other types of input indicating the orientation of the first instrument, such as text / voice input indicating the degrees / angles of one or more markings on the tip of the first instrument, displayed via an interface.

[0160] At block 1312, process 1300 may include causing the image data to be displayed. For example, the control circuitry may cause a graphical representation of the image data (received at block 1302) to be displayed via the interface.

[0161] At block 1314, process 1300 may include displaying instructions. The instructions may request that the user perform a particular action. In one example, the instructions indicate selecting a particular directional control on the input device (e.g., text that reads "Select right" may be presented via the interface). In another example, the instructions indicate moving the first instrument in a particular direction relative to the interface (e.g., text that reads "Move catheter right" may be displayed via the interface).

[0162] At block 1316, process 1300 may include receiving a directional input signal. In one example where a user is requested to select a particular directional control on an input device, the control circuitry may receive a directional input signal indicative of the input received via the particular directional control. In another example where a user is requested to move a first instrument in a particular direction relative to the interface, the control circuitry may receive a directional input signal to move the first instrument in the particular direction relative to the interface. Here, the user may provide multiple inputs in an attempt to move the first instrument in the particular direction relative to the interface.

[0163] At block 1318, the process 1300 may control the first instrument to move. For example, the control circuitry may move the first instrument based at least in part on the directional input signal.

[0164] In some embodiments, such as when a user is requested to select a particular directional control on an input device, process 1300 may include receiving an input indicating a direction of movement at block 1320. For example, the control circuit may receive an input indicating a direction in which the first instrument was moved within the interface in response to selecting the particular directional control on the input device. Here, a user may specify the direction in which the first instrument was moved by providing an input via an input device, such as an interface, controller, or the like.

[0165] At block 1322, process 1300 may include performing one or more image processing techniques using the image data received at block 1302. For example, the control circuitry may perform one or more image processing techniques to identify the first instrument depicted in the image data. Such techniques may identify one or more features of the first instrument, such as the tip of the first instrument, one or more markings on the first instrument, etc.

[0166] At block 1324, process 1300 may include determining an orientation of the first instrument and / or an orientation of the second instrument. For example, the control circuitry may determine a first coordinate system associated with the first instrument and / or a second coordinate system associated with the second instrument based at least in part on one or more of blocks 1306-1322. The coordinate systems may indicate / represent a roll of the distal end of the instrument. In some embodiments, the control circuitry may generate roll data indicating the roll of the distal end of the first instrument relative to the roll of the distal end of the second instrument. For example, the roll data may indicate an orientation of the coordinate system associated with the first instrument relative to the coordinate system associated with the second instrument.

[0167] At block 1326, process 1300 may include calibrating the control scheme for the first instrument. For example, the control scheme may include and / or be represented using a control coordinate system for the first instrument. The control circuitry may identify a difference between a first coordinate system associated with the first instrument and a second coordinate system associated with the second instrument and update the control coordinate system associated with the first instrument based on the difference.

[0168] At block 1328, process 1300 may include receiving an input signal from an input device. For example, the control circuit may receive a directional input signal from the input device indicating a direction of movement of the first instrument relative to the input device.

[0169] At block 1330, process 1300 may include controlling movement of the first instrument based on the control scheme and the input signal. For example, the control circuitry may use the control scheme calibrated at block 1326 to control movement of the first instrument based on the input signal received at block 1328.

[0170] Exemplary Control Frame Calibration FIGS. 14-1 through 14-4 illustrate an exemplary implementation of calibrating a control frame for a medical instrument, according to one or more embodiments. In these figures, a catheter 1402 is positioned proximate to a scope 1404 to facilitate a medical procedure. Here, the catheter 1402 is oriented substantially directly opposite the scope 1404, and the catheter 1402 has its body portion omitted to show only its tip portion. FIGS. 14-1 through 14-4 illustrate various coordinate / control frames for the catheter 1402 and / or the scope 1404. While various conventions can be used, descriptions often refer to the "forward" direction as corresponding to positive z, the "right" direction as corresponding to positive x, and the "up" direction as corresponding to positive y. For ease of illustration, the z vectors of the coordinate / control frames are not shown in FIGS. 14-1 through 14-4.

[0171] FIG. 14-1 shows an estimated orientation of a catheter 1402 relative to a scope 1404. In particular, FIG. 14-1 shows an estimated coordinate system 1406 associated with the catheter 1402 relative to a coordinate system 1408 associated with the scope 1404. Meanwhile, FIG. 14-2 shows the actual orientation of the catheter 1402 relative to the scope 1404, i.e., the actual coordinate system 1410 of the catheter 1402 relative to the coordinate system 1408 of the scope 1404. In this example, the estimated orientation of the catheter 1402 relative to the scope 1404 (as shown in FIG. 14-1) contains some amount of error with respect to the actual orientation of the catheter 1404 relative to the scope 1404 (as shown in FIG. 14-2). For example, the coordinate systems 1406 and 1410 are offset from each other. As mentioned above, such errors may result from undetected roll of the scope 1404, manipulation of the proximal end of the scope 1404 that does not fully propagate to the distal end of the scope 1404, or other unexplained roll of the scope 1404.

[0172] FIG. 14-1 also shows a control frame 1412 for the catheter 1402, which is set based on the estimated orientation of the catheter 1402 relative to the scope 1404. The control frame 1412 may be implemented to control the movement of the catheter 1402 from the perspective of the scope 1404. The control frame 1412 is represented / defined by an offset relative to the estimated coordinate system 1406 of the catheter 1402. However, when the control frame 1412 is implemented with the actual orientation of the catheter 1402, the control frame 1412 appears as shown in FIG. 14-3. As shown, the offset of the control frame 1412 relative to the estimated coordinate system 1406 of the catheter 1402 (FIG. 14-2) is the same as the offset of the control frame 1412 relative to the actual coordinate system 1410 of the catheter 1406 (FIG. 14-3).

[0173] Because control frame 1412 is set based on an estimated orientation of catheter 1402 relative to scope 1404, and this estimated orientation has errors, control frame 1412 inaccurately represents a reference frame for controlling catheter 1402, as shown in FIG. 14-3. For example, if a directional control input to move catheter 1402 straight up is received from an input device, catheter 1402 is controlled to move up relative to control frame 1410 (i.e., in the positive direction along the y-axis of control frame 1412). In the context of FIG. 14-3, which shows the actual orientation of catheter 1402 relative to scope 1404, catheter 1402 moves to the right and up, rather than straight up. Therefore, catheter 1402 is controlled to move inaccurately relative to scope 1404.

[0174] To address discrepancies between the estimated and actual orientation of the catheter 1402 relative to the scope 1404, the control frame 1412 for the catheter 1402 can be calibrated. For example, the control frame 1412 may be adjusted to accurately reflect the orientation of the catheter 1402 relative to the scope 1404, as shown in FIG. 14-4 . In some embodiments, the estimated coordinate system 1406 of the catheter 1402 and the coordinate system 1408 of the scope 1404 can be moved to be coplanar. A first offset / difference between the coordinate system 1406 and the coordinate system 1408 (e.g., an offset between the x / y axes of the coordinate system 1406 and the x / y axes of the coordinate system 1408) can then be determined. In addition, the determined coordinate system of the catheter 1402 (e.g., determined through user adjustments to alignment indicators on an interface, determined through image processing, etc.) and the coordinate system 1408 of the scope 1404 can be moved to be coplanar. A second offset / difference may be determined between the coordinate system determined for the catheter 1402 and the coordinate system 1408 for the scope 1404. The difference between the first offset and the second offset may be determined and used to update the control frame 1412. For example, if there is a 15 degree difference between the first offset and the second offset (meaning, e.g., that the estimated orientation of the catheter 1402 relative to the scope 1404 is inaccurate by 15 degrees), the control frame 1412 may be adjusted by 15 degrees or by some amount based on 15 degrees.

[0175] Exemplary Procedures Using the Medical System 15-18 are top views of the medical system 100 of FIG. 1 arranged to perform a percutaneous procedure, according to one or more embodiments. In these examples, the medical system 100 is positioned in an operating room to remove a kidney stone from a patient 140 with the aid of a scope 120 (e.g., a ureteroscope) and a catheter 130. In many embodiments of such procedures, the patient 140 is positioned in a modified supine position, with the patient 140 tilted slightly to the side, to access the back or side of the patient 140, as illustrated in FIG. 1. However, the patient 140 may be positioned differently, such as supine, prone, etc. Also, for ease of illustration, the imaging device 180 (including the C-arm) has been removed.

[0176] 15-18 illustrate the use of medical system 100 to perform a percutaneous procedure to remove kidney stones from patient 140, medical system 100 may be used to remove kidney stones in other ways and / or to perform other procedures. Additionally, patient 140 may be positioned in other positions as desired for the procedure. Various actions are described in FIGS. 15-18 and throughout this disclosure as being performed by physician 160. It should be understood that these actions may be performed directly by physician 160, a user under the direction of the physician, another user (e.g., a technician), a combination thereof, and / or any other user.

[0177] Renal anatomy, as illustrated at least in part in Figures 15-18, is described herein for reference with respect to certain medical procedures related to embodiments of the present concepts. The kidneys generally comprise two bean-shaped organs located on either side of the retroperitoneal cavity. In adult humans, the kidneys are generally approximately 11 cm in length. The kidneys receive blood from paired renal arteries, which exit into paired renal veins. Each kidney is attached to a ureter, which is a tube that carries excreted urine from the kidney to the bladder. The bladder is attached to the urethra.

[0178] The kidneys typically lie relatively high within the abdominal cavity, at a slightly oblique angle in a retroperitoneal position. Intraperitoneal asymmetry caused by the position of the liver typically causes the right kidney to be slightly lower and smaller than the left, and slightly more centrally located than the left kidney. Above each kidney is an adrenal gland, which is partially protected by the 11th and 12th ribs. Each kidney, along with its adrenal gland, is surrounded by two layers of fat: perirenal fat, located between the renal fascia and the renal capsule, and pararenal fat, located above the renal fascia.

[0179] The kidneys are responsible for regulating the volume of various body fluid compartments, fluid osmolality, acid-base balance, various electrolyte concentrations, and the removal of toxins. The kidneys provide a filtration function by secreting certain substances and reabsorbing others. Examples of substances secreted in urine are hydrogen, ammonium, potassium, and uric acid. In addition, the kidneys also perform various other functions, such as hormone synthesis and others.

[0180] The concave area on the kidney's concave border is the renal hilum, where the renal artery enters the kidney and the renal vein and ureter exit. The kidney is surrounded by a tough fibrous tissue, the renal capsule, which is itself surrounded by perirenal fat, renal fascia, and pararenal fat. The anterior (front) surface of these tissues is the peritoneum, while the posterior (back) surface is the transversalis fascia.

[0181] The functional matrix, or parenchyma, of the kidney is divided into two major structures: the outer renal cortex and the inner renal medulla. These structures comprise multiple cone-shaped renal lobes, each of which contains a renal cortex surrounding a portion of the medulla called a renal pyramid. Between the pyramids are processes of the cortex called renal columns. The nephron, the urine-producing functional structure of the kidney, spans the cortex and medulla. The initial filtering portion of the nephron is the renal corpuscle, located in the cortex. This is followed by renal tubules, which pass from the cortex deep into the medullary pyramid. The medullary ray, part of the renal cortex, is a collection of tubules that drain into a single collecting duct.

[0182] The tip, or papilla, of each pyramid drains urine into a minor calyx, which drains into a major calyx, which drains into the renal pelvis, where it passes into the ureter. At the hilum, the ureter and renal vein leave the kidney, and the renal artery enters. Hilar fat and lymphatic tissue, along with lymph nodes, surround these structures. The hilar fat is adjacent to a fat-filled cavity called the renal sinus. The renal sinus collectively contains the renal pelvis and calyx, separating these structures from the renal medullary tissue.

[0183] 15-18 illustrate various features of the anatomy of a patient 140. For example, the patient 140 includes a kidney 1502 fluidly connected to a bladder 1504 via a ureter 1506, and a urethra 1508 fluidly connected to the bladder 1504. As shown in the close-up depiction of the kidney 1502(A), the kidney 1502(A) includes a renal calyx (including a renal calyx 1510), a renal papilla (also referred to as "papilla 1512," including a "renal papilla 1512"), and a renal pyramid (including a renal pyramid 1514). In these examples, a kidney stone 1516 is located proximate to the papilla 1512. However, the kidney stone 1516 may be located elsewhere within the kidney 1502(A), etc.

[0184] As shown in FIG. 15 , to remove a kidney stone 1516 in an exemplary percutaneous procedure, a physician 160 may position the robotic system 110 at the side / foot of a table 170 to begin delivery of a scope 120 (not illustrated in FIG. 15 ) to a patient 140. Specifically, the robotic system 110 may be positioned at the side of the table 170 within close proximity of the patient's 140's feet and aligned for direct, linear access to the patient's 140's urethra 1508. In an embodiment, the patient's 140 hip joint is used as a reference point for positioning the robotic system 110. Once positioned, one or more of the robotic arms 112, such as robotic arms 112(B) and 112(C), can extend outward to reach between the patient's 140's legs. For example, the robotic arm 112(B) may be controlled to extend to the urethra 1508, as shown in FIG. 15 , to provide linear access thereto. In this example, the physician 160 inserts a medical instrument 1518 at least partially into the urethra 1508 along this direct, linear access path (sometimes referred to as a "virtual rail"). The medical instrument 1518 may include a luminal device configured to receive the scope 120, thereby assisting in inserting the scope 120 into the anatomy of the patient 140. By aligning the robotic arm 112(B) with the urethra 1508 of the patient 140 and / or by using the medical instrument 1518, friction and / or force on sensitive anatomy in the area may be reduced. Although the medical instrument 1518 is illustrated in FIG. 15 , in some embodiments, the medical instrument 1518 is not used (e.g., the scope 120 may be inserted directly into the urethra 1508).

[0185] The physician 160 can also position the robotic arm 112(A) near the treatment site for the procedure. For example, the robotic arm 112(A) can be positioned at the incision site and / or within close proximity of the kidney 310 of the patient 140. The robotic arm 112(A) can be connected to an EM field generator 1520 to assist in tracking the location of the scope 120 and / or other instruments during the procedure. While the robotic arm 112(A) is positioned relatively close to the patient 140, in some embodiments, the robotic arm 112(A) is positioned elsewhere and / or the EM field generator 1520 is integrated into the table 170 (which can allow the robotic arm 112(A) to be in a docked position). In this example, at this point in the procedure, the robotic arm 112(A) remains in a docked position, as shown in FIG. 15 . However, robotic arm 112(C) may, in some embodiments, be used to perform any of the above-discussed functions of robotic arms 112(A) and / or 112(C).

[0186] Once the robotic system 110 is properly positioned and / or the medical instrument 1518 is at least partially inserted into the urethra 1518, the scope 120 may be inserted into the patient 140 robotically, manually, or a combination thereof, as shown in FIG. 16 . For example, the physician 160 may connect the scope 120 to the robotic arm 112(C) and / or position the scope 120 at least partially within the medical instrument 1518 and / or the patient 140. The scope 120 may be connected to the robotic arm 112(C) at any time, such as before or during the procedure (e.g., after positioning the robotic system 110). The physician 160 may then interact with the control system 150, such as the I / O device(s) 156, to navigate the scope 120 within the patient 140. For example, physician 160 may provide input via I / O device(s) 156 to control robotic arm 112(C) to navigate scope 120 through urethra 1508, bladder 1504, ureter 1506(A), and to kidney 1502(A).

[0187] In some embodiments, the control system 150 may present an interface (not shown) via the display(s) 152 to show real-time images acquired by the scope 120 to assist the physician 160 in controlling the scope 120. The physician 160 may navigate the scope 120 to locate a kidney stone 1516. In some embodiments, the control system 150 may use localization techniques to determine the position and / or orientation of the scope 120, which is viewed by the physician 160 via the display(s) 152 to assist in controlling the scope 120. Additionally, in some embodiments, other types of information, such as x-ray images of the internal anatomy of the patient 140, may be presented through the display(s) 152 to assist the physician 160 in controlling the scope 120.

[0188] Upon locating the kidney stone 1516, the physician 160 may identify a location where the needle 17022 will enter the kidney 1502(A) for eventual removal of the kidney stone 1516. For example, to minimize bleeding and / or avoid striking blood vessels or other undesirable anatomical structures of the kidney 1502(A) and / or anatomical structures surrounding the kidney 1502(A), the physician 160 may seek to align the needle 17022 with the axis of the calyx (e.g., seek to reach the calyx head-on through the center of the calyx). To do so, the physician 160 may identify the papilla as the target location. In this example, the physician 160 uses the scope 120 to locate the papilla 1512 near the kidney stone 1516 and designate the papilla 1512 as the target location. In some embodiments that designate the nipple 1512 as the target location, the physician 160 can navigate the scope 120 to contact the nipple 1512, and the control system 150 can use localization techniques to determine the location of the scope 120 (e.g., the location of the end of the scope 120), and the control system 150 can associate the location of the scope 120 with the target location. In other embodiments, the physician 160 can navigate the scope 120 to within a certain distance of the nipple 1512 (e.g., resting in front of the nipple 1512) and provide input indicating that the target location is within the field of view of the scope 120. The control system 150 can perform image analysis and / or other localization techniques to determine the location of the target location. In yet other embodiments, the scope 120 can deliver a reference point for marking the nipple 1512 as the target location.

[0189] As shown in FIG. 17 , physician 160 (and / or robotic system 110) can proceed with the procedure by positioning needle 1702 for insertion into the target location. In some embodiments, physician 160 can place needle 1702 into patient 140 at the incision site using their best judgment, such as based on knowledge of the patient's 140 anatomy, experience performing the procedure previously, analysis of CT / X-ray images, or other preoperative information about the patient 140. Additionally, in some embodiments, control system 150 can provide information regarding where to place needle 1702 into patient 140. Physician 160 can attempt to avoid critical anatomical structures of patient 140, such as the lungs, pleura, colon, paraspinal muscles, ribs, and intercostal nerves. In some examples, control system 150 can use CT / X-ray / ultrasound images to provide information regarding where to place needle 1702 into patient 140. In some embodiments, control system 150 can present information to assist physician 160 in inserting needle 1702. For example, the control system 150 may display an instrument alignment element indicating the orientation of the needle 1702 relative to the target trajectory to assist the physician 160 in properly orienting the needle 1702 (i.e., the target trajectory). Additionally, the control system 150 may display progress information indicating the proximity of the needle 1702 to the target location.

[0190] 18 , once the needle 1702 reaches the target location, the physician 160 and / or the robotic system 110 can insert another medical instrument into the path formed by the needle 1702. In this example, the EM field generator 1520 on the robotic arm 112(A) is replaced with a catheter 130 configured to be inserted into a percutaneous access pathway. The physician 160 can interact with the control system 150 (e.g., I / O device(s) 156) to navigate the catheter 130 through the percutaneous access pathway to the target location. The control system 150 can provide information via the display(s) 152, such as any of the interfaces discussed herein, to assist the physician 160 in navigating the catheter 130. In an example, the catheter 130 can be driven from a third-person perspective (e.g., from the perspective of the scope 120). In some embodiments, the medical system 100 may facilitate one or more control / drive modes and / or calibration techniques to assist the physician 160 in driving and / or calibrating the catheter 130, the scope 120, and / or another medical instrument.

[0191] Once the scope 120 and / or catheter 130 are in place, the physician 160 can use the scope 120 to break up the kidney stone 1516 and / or the catheter 130 to extract fragments of the kidney stone 1516 from the patient 140. For example, the scope 120 can deploy a tool (e.g., a laser, cutting instrument, etc.) to fragment the kidney stone into pieces, and the catheter 130 can suction the pieces out of the kidney 1502(A) through a percutaneous access pathway. The physician 160 can optionally switch control of the scope 120 and the catheter 130. For example, the physician 160 can use the same I / O device(s) 156 to control the catheter 130 and the scope 120. In an embodiment, the I / O device(s) 156 include buttons to switch control of the medical instruments.

[0192] In some embodiments, the catheter 130 and / or scope 120 may provide irrigation and / or suction to facilitate removal of the kidney stone 1516. For example, the catheter 130 and / or scope 120 may connect to an irrigation / suction system 1802. The irrigation / suction system 1802 may be configured to hold and / or control fluid from one or more fluid bags / containers. For example, an irrigation line 1804 may be coupled to one or more of the bags / containers and to an irrigation port on the catheter 130. Irrigation fluid may be provided to the target anatomical structure via the irrigation line 1804 and the catheter 130. The irrigation / suction system 1802 may include certain electronic components, such as a display, a flow control mechanism, and / or certain associated control circuitry. In some examples, the irrigation / suction system 1802 is implemented as a fluid management cart. In an embodiment, the irrigation / aspiration system 1802 is configured to interface with other components of the medical system 100, such as the control system 150, the robotic system 110, and / or other components.

[0193] In some embodiments, such as the example described above with reference to FIG. 18 , the EM field generator 1520 is detached from the robotic arm 112(A) while the catheter 130 is being driven. During such a phase, one or more EM-based localization techniques may not be performed to determine the position / orientation of the scope 120. Thus, the medical system 100 may sometimes lose track of the position / orientation of the scope 120 and / or another medical instrument. Therefore, one or more of the techniques discussed herein may be implemented to calibrate the scope 120 and / or to calibrate a control scheme for controlling the catheter 130 from the perspective of the scope 120.

[0194] While particular robotic arms of robotic system 110 are illustrated as performing particular functions in the context of FIGS. 15-18 , any of robotic arms 112 may be used to perform those functions. Furthermore, any additional robotic arms and / or systems may be used to perform the procedure. Furthermore, robotic system 110 may be used to perform other portions of the procedure. In some embodiments, the percutaneous procedure may be performed completely or partially using medical system 100 (e.g., with or without the assistance of physician 160).

[0195] Exemplary Robotic System 1 is an exemplary detailed diagram of the robotic system of FIG. 1 , according to one or more embodiments. As shown, the robotic system 110 may include control circuitry 1902, communication interface(s) 1904 (e.g., configured to communicate with one or more components / devices), power unit(s) 1906 (e.g., configured to provide / manage power to the components of the robotic system 110), I / O components 1908 (e.g., display(s) 116 and / or other I / O devices / controls 1910), one or more robotic arms 112, actuators / hardware 1912, and / or movement component(s) 1914 (e.g., wheels). In some embodiments, the robotic system 110 may comprise a housing / enclosure configured and / or dimensioned to house or contain at least a portion of one or more of the components of the robotic system 110. In this example, the robotic system 110 is illustrated as a cart-based system that is mobile using one or more wheels 1914. In some cases, after reaching the proper position, one or more wheels 1914 may be locked using wheel locks to hold the robotic system 110 in place. However, the robotic system 110 may be implemented as a fixed system, integrated into another system / device, etc.

[0196] In some embodiments, one or more components of the robotic system 110 may be electrically and / or communicatively coupled using certain connection circuits / devices / features, which may or may not be part of the control circuit 1902. For example, the connection feature(s) may include one or more printed circuit boards configured to facilitate the mounting and / or interconnection of at least some of the various components / circuits of the robotic system 110. While certain components of the robotic system 110 are illustrated in FIG. 18 , it should be understood that additional components not shown may be included in embodiments consistent with the present disclosure. Furthermore, in some embodiments, some of the illustrated components may be omitted.

[0197] The support structure 114 may include a base 1916, an elongated column 1918, and / or a console 1920 atop the column 1918. The column 1918 may include one or more arm supports 1922 (also referred to as "carriages") to support the deployment of one or more robotic arms 112 (three are shown in FIG. 2 ). The arm supports 1922 may include individually configurable arm mounts that rotate along a vertical axis to adjust the base of the robotic arms 112 for better positioning relative to the patient. The arm supports 1922 also include a column interface 1924 that allows the arm supports 1922 to translate vertically along the column 1916. In some embodiments, the column interface 1922 may connect to the column 1918 through slots, such as slots 1926, positioned on either side of the column 1918 to guide the vertical translation of the arm supports 1922. The slot 1926 contains a vertical translation interface for positioning and holding the arm support 1922 at various vertical heights relative to the base 1916. The vertical translation of the arm support 1922 allows the robotic system 110 to adjust the reach of the robotic arm 112 to meet various table heights, patient sizes, and / or physician preferences. Similarly, the individually configurable arm mounts on the arm support 1922 can allow the base 1916 of the robotic arm 1916 to be angled in various configurations.

[0198] The robotic arm 112 may generally include a robotic arm base 1928 and an end effector 1930 separated by a series of linkages 1932 connected by a series of joints 1934, with each joint including one or more independent actuators 1912. Each actuator may include an independently controllable motor. Each independently controllable joint 1934 may provide or represent an independent degree of freedom available to the robotic arm. In some embodiments, each of the arms 112 has seven joints, thus providing seven degrees of freedom, including "redundant" degrees of freedom. The redundant degrees of freedom allow the robotic arms 112 to position their respective end effectors 1930 at specific positions, orientations, and / or trajectories in space using different linkage positions and joint angles. This allows the robotic system 110 to position and orient a medical instrument from a desired point in space while also allowing the physician to move the arm joints to clinically advantageous positions away from the patient to create better access while avoiding arm collisions.

[0199] Each end effector 1930 may include an instrument device manipulator (IDM), which may be attached using a mechanism changer interface (MCI). In some embodiments, the IDM may be removed and replaced with a different type of IDM; for example, a first type of IDM may operate an endoscope, while a second type of IDM may operate a laparoscope. The MCI may include connectors for transmitting air pressure, power, electrical signals, and / or optical signals from the robotic arm 112 to the IDM. The IDM may be configured to manipulate medical instruments (e.g., surgical tools / instruments) using techniques including, for example, direct drive, harmonic drive, gear drive, belt and pulley drive, magnetic drive, etc.

[0200] The robotic system base 1916 can balance the weight of the column 1918, arm support 1920, and arm 112 on the floor. Thus, the robotic system base 1916 can house heavier components such as electronics, motors, and power supplies, as well as components that selectively allow movement or immobilize the robotic system 110. In an embodiment such as that shown in FIG. 19 , the robotic system base 1916 includes casters 1914 in the form of wheels that allow the robotic system to be easily moved around a room before a procedure. Once in the appropriate position, the casters 1914 can be locked using wheel locks to hold the cart 110 in place during the procedure.

[0201] When the console 1920 is positioned at the top of the column 1918, it allows the physician / user to view both pre-operative and / or intra-operative data on both the display screen 116 (or a dual-purpose device, such as a touchscreen) and the user interface for receiving user input. Potential pre-operative data may include pre-operative planning, navigation, and / or mapping data derived from a pre-operative computed tomography (CT) scan, and / or notes from a pre-operative patient interview. Intra-operative data may include optical information provided by tools, sensor and coordinate information from sensors, and vital patient statistics such as respiration, heart rate, and / or pulse. The console 1920 may be positioned and tilted to allow the physician to access the console 1920 from the side of the column 1918 opposite the arm support 1922. From this position, the physician may view the console 1920, the robotic arm 112, and / or the patient while operating the console 1920 from behind the robotic system 110. As shown, the console 1920 may also include a handle 1936 to assist in manipulating and stabilizing the robotic system 110.

[0202] Exemplary Control System 20 is an exemplary detailed diagram of control system 150 of FIG. 1 , according to one or more embodiments. As illustrated, control system 150 can include one or more of the following components, devices, modules, and / or units (referred to herein as “components”), either separately / individually and / or in combination / collectively: control circuitry 2002, data storage / memory 2004, one or more communication interfaces 2006, one or more power supply units 2008, one or more input / output (I / O) components 2010, and / or mobilization members 2012 (e.g., casters or other types of wheels). In some embodiments, control system 150 can comprise a housing / enclosure configured and / or dimensioned to house or contain at least a portion of one or more of the components of control system 150. In this example, control system 150 is illustrated as a cart-based system that is movable using one or more wheels 2012. In some cases, after reaching the proper position, one or more wheels 2012 may be locked using wheel locks to hold control system 150 in place. However, control system 150 may be implemented as a locked system, integrated into another system / device, etc.

[0203] While certain components of control system 150 are illustrated in FIG. 20 , it should be understood that additional components not shown may be included in embodiments consistent with the present disclosure. Furthermore, in some embodiments, some of the illustrated components may be omitted. While control circuit 2002 is illustrated as a separate component in the diagram of FIG. 20 , it should be understood that any or all of the remaining components of control system 150 may be at least partially embodied in control circuit 2002. That is, control circuit 2002 may include various devices (active and / or passive), semiconductor materials, and / or regions, layers, areas, and / or portions thereof, conductors, leads, vias, connections, etc., and one or more of the other components of control system 150 and / or portion(s) thereof may be at least partially formed and / or embodied by such circuit components / devices.

[0204] The various components of control system 150 may be electrically and / or communicatively coupled using certain connection circuits / devices / features, which may or may not be part of control circuit 2002. For example, connection feature(s) may include one or more printed circuit boards configured to facilitate mounting and / or interconnection of at least some of the various components / circuits of control system 150. In some embodiments, two or more of control circuit 2002, data storage / memory 2004, communication interface(s) 1206, power supply unit(s) 1208, and / or input / output (I / O) component(s) 1210 may be electrically and / or communicatively coupled to one another.

[0205] As illustrated, the memory 2004 may include a location determination component 2014, a user interface component 2016, an instrument drive component 2018, and a calibration component 2020 configured to facilitate various functionalities discussed herein. In some embodiments, the location determination component 2014, the user interface component 2016, the instrument drive component 2018, and / or the calibration component 2020 may include one or more instructions executable by the control circuit 2002 to perform one or more operations. While many embodiments are discussed in the context of the components 2014-2020 including one or more instructions executable by the control circuit 2002, any of the components 2014-2020 may be implemented at least in part as one or more hardware logic components, such as one or more application specific integrated circuits (ASICs), one or more field programmable gate arrays (FPGAs), one or more advanced standard products (ASSPs), one or more complex programmable logic devices (CPLDs), etc. Additionally, although components 2014-2020 are illustrated as being included within control system 150, any of components 2014-2020 may be implemented at least partially within another device / system, such as robotic system 110, table 170, or another device / system. Similarly, any of the other components of control system 150 may be implemented at least partially within another device / system.

[0206] The localization component 2014 may be configured to perform one or more localization techniques to determine and / or track the position and / or orientation of an object, such as a medical instrument. For example, the localization component 2014 may process input data (e.g., sensor data from a medical instrument, model data about a patient's anatomy, patient position data, pre-operative data, robotic commands, and / or kinematic data, etc.) to generate position / orientation data 2022 for one or more medical instruments. In an example, the position / orientation data 2022 may indicate the location and / or orientation of one or more medical instruments relative to a frame of reference. The frame of reference may be relative to the patient's anatomy, a known object (e.g., an EM field generator, a robotic arm, another medical instrument, etc.), a coordinate system / space, a control frame, etc. In some implementations, the position / orientation data 1220 may indicate the location and / or orientation of the distal end (and / or, in some cases, the proximal end) of the medical instrument.

[0207] In some embodiments, the localization component 2014 may process preoperative data to determine the position and / or orientation of an object. The preoperative data (sometimes referred to as “mapping data”) may be generated by performing a computed tomography (CT) scan, such as a low-dose CT scan. The preoperative CT images from the scan may be reconstructed into a three-dimensional image, which is visualized, for example, as cutaway “slices” of the patient's internal anatomy. When analyzed as a whole, an image-based model may be generated of the anatomical cavities, spaces, and structures of the patient's anatomy, such as the patient's pulmonary network, renal anatomy, etc. Centerline geometries may be determined and approximated from the CT images to develop a three-dimensional volume of the patient's anatomy, referred to as model data (also referred to as “preoperative model data” when generated using only preoperative CT scans). A network topology model may also be derived from the CT images.

[0208] Additionally, in some embodiments, the localization component 2014 may perform vision-based techniques to determine the position and / or orientation of an object. For example, a medical instrument may be equipped with a camera, a distance sensor (sometimes referred to as a “depth sensor”), a radar device, etc. to provide sensor data in the form of visual data. The localization component 2014 may process the visual data to facilitate vision-based location tracking of the medical instrument. For example, pre-operative model data may be used in conjunction with the visual data to enable computer vision-based tracking of the medical instrument (e.g., an endoscope). In an example implementation, using the pre-operative model data, the control system 150 may generate a library of predicted endoscopic images based on the expected path of travel of the scope, with each image linked to a location within the model. During surgery, this library may be referenced by the control system 150 to compare real-time images acquired by the scope (e.g., a camera at the distal end of the endoscope) and / or other visual data with images in the image library to assist with localization.

[0209] Additionally, in some embodiments, other types of vision-based techniques may be implemented to determine the position and / or orientation of an object. For example, the localization component 2014 may use feature tracking to determine the movement of an image sensor (e.g., a camera or other sensor) such that the medical instrument is associated with the image sensor. In some cases, the localization component 2014 may identify circular geometries in the preoperative model data that correspond to anatomical lumens and track changes in those geometries to determine which anatomical lumens were selected and the relative rotational and / or translational movement of the medical instrument. The use of topology maps may also further enhance vision-based algorithms or techniques. Additionally, the localization component 2014 may use optical flow, another computer vision-based technique, to analyze the displacement and / or translation of image pixels in a video sequence in the vision data to infer camera movement. Examples of optical flow techniques may include motion detection, object segmentation calculations, brightness, motion-compensated coding, stereo disparity measurement, etc. By comparing multiple frames over multiple iterations, the localization component 2014 can determine the movement and location of the image sensor (and therefore the endoscope).

[0210] Further, in some embodiments, the localization component 2014 may use electromagnetic tracking to determine the position and / or orientation of an object. For example, the localization component 2014 may use real-time EM tracking to determine the real-time location of a medical instrument in a coordinate system / space (and / or relative to another medical instrument), which may be registered to the patient's anatomy, and this location may be represented by a pre-operative or other model. In EM tracking, an EM sensor (or tracker) including one or more sensor coils may be embedded in one or more locations and / or orientations within the medical instrument (e.g., a scope, needle, etc.). The EM sensor may measure variations in an EM field created by one or more static EM field generators positioned at known locations. The location information detected by the EM sensor may be stored as EM data. The localization component 2014 may process the EM data to determine the position and / or orientation of an object, such as a medical instrument. The EM field generator (or transmitter) may be placed near (e.g., within a predetermined distance from) the patient to create a low-intensity magnetic field that the EM sensor can detect. The magnetic field can induce small currents in the sensor coil of the EM sensor, which can be analyzed to determine the distance and angle between the EM sensor and the EM field generator. These distances and orientations can be "recorded" intraoperatively on the patient's anatomy (e.g., a pre-operative model) to determine a geometric transformation that aligns a single location in a coordinate system with a position on the pre-operative model of the patient's anatomy. Once recorded, EM sensors (e.g., implanted EM trackers) at one or more locations on the medical instrument (e.g., the distal tip of an endoscope, a needle, etc.) can indicate the position and / or orientation of the medical instrument through the patient's anatomy in real time.

[0211] Additionally or alternatively, in some embodiments, the localization component 2014 may use robotic command and / or kinematic data to determine the position and / or orientation of an object. The robotic command and / or kinematic data may indicate the position / orientation (e.g., pitch, yaw, etc.) of a robotic arm resulting from articulation commands, such as commands used during pre-operative calibration and / or during a procedure. In an example embodiment, the localization component 2014 may use data indicating the position / orientation of the robotic arm to determine the position / orientation of a medical instrument attached to the robotic arm. For example, based on the position / orientation of the robotic arm attached to a catheter, commands sent to control the catheter, and / or characteristic(s) of the catheter (e.g., catheter length, catheter capacity, etc.), the localization component 2014 may determine / estimate the position / orientation of the catheter. Further, in an example embodiment, the localization component 2014 may use the robotic command data to determine how far the medical instrument has been inserted / retracted into a patient, etc., based on commands to control the medical instrument, markings on the medical instrument indicating distance, etc. In some intraoperative embodiments, calibration measurements may be used in combination with known insertion depth information to estimate the position and / or orientation of the medical instrument. Alternatively or additionally, these calculations may be analyzed in combination with EM, visual, and / or topological modeling to estimate the position and / or orientation of the medical instrument.

[0212] Additionally, in some embodiments, the localization component 2014 may use other types of data to determine the position and / or orientation of the object. For example, the localization component 2014 may analyze sensor data from shape-sensing fibers (e.g., which can provide shape data regarding the location / shape of the medical instrument), accelerometers, gyroscopes, satellite-based positioning sensors (e.g., Global Positioning System (GPS)), radio frequency transceivers, etc. embedded in the medical instrument. Such data can suggest the position and / or orientation of the medical instrument.

[0213] In some embodiments, the localization component 2014 may use a combination of input data. For example, the localization component 2014 may use a probabilistic approach in which confidence weights are assigned to positions / orientations determined from multiple forms of input data. To illustrate, if the EM data is unreliable (such as in the case of EM interference), the EM data may be associated with a relatively low confidence value, and other forms of input data, such as visual data, robot commands, kinematic data, etc., may be relied upon.

[0214] The user interface component 2016 may be configured to facilitate one or more user interfaces (also referred to as "one or more graphical user interfaces (GUIs)"). For example, the user interface component 2016 may generate user interface data representing one or more of the interfaces described herein, such as interface 702 of FIG. 7 and interface 1102 of FIGS. 11-12. The user interface component 2016 may present one or more visualizations or other information to assist in calibrating the actuation and / or control scheme of the medical instrument. In an example, the user interface component 2016 may generate a visual representation of image data acquired by the scope. The user interface component 2016 may provide one or more user interface data or other data to display information on the display 156 and / or another display(s).

[0215] The instrument drive mechanism 2018 may be configured to drive the medical instrument. For example, the instrument drive component 2018 may be configured to process directional input signals from the I / O device(s) 156, process position or / orientation data 2022 related to the medical instrument, generate control signals, transmit the control signals to the robotic system 110 to control the movement of the instrument(s) connected to the robotic system 110, etc. In some embodiments, the instrument drive component 2018 may facilitate driving of the medical instrument from the perspective of another medical instrument. Additionally, in some embodiments, the instrument drive component 2018 may facilitate one or more drive / control modes, such as a direct control mode, an inverted control mode, etc., to assist the physician in driving the medical instrument.

[0216] The calibration component 2020 may be configured to calibrate the control scheme / control reference frame of the medical instrument. For example, the calibration component 2020 may determine the orientation of a first instrument relative to a second instrument, such as the orientation of the distal end of a catheter relative to the distal end of a scope. In some embodiments, the user interface component 2016 may provide a user interface with image data depicting the first instrument from the perspective of the second instrument and one or more interface elements to enable a physician to identify the orientation of the catheter. The calibration component 2020 may process input from the physician to identify the orientation of multiple instruments relative to one another. Furthermore, in some embodiments, the calibration component 2020 may analyze image data and / or other sensor data from the first instrument / second instrument to identify the orientation of the instruments relative to one another. Furthermore, in some embodiments, other techniques may be used to identify the orientation of the instruments relative to one another. Based on the orientation of the instruments, the calibration component 2020 adjusts the control scheme associated with the control of the first instrument / second instrument.

[0217] Although not shown in FIG. 20 , in some embodiments, the data storage 2004 may include a targeting component configured to determine the position of a target location within a human anatomy and / or coordinate space / system. The target location may represent a point / set of points within the human anatomy and / or coordinate space / system. For example, the targeting component may identify one or more points of the target location within a coordinate system, identify coordinates of the one or more points (e.g., X, Y, Z coordinates for each point), and associate the coordinates with the target location. In some embodiments, the targeting component may use the position and / or orientation of the medical instrument to determine the position of the target location. For example, a scope may be navigated to touch or be within proximity (e.g., resting in front of) the target location. The localization component 2014 may use localization techniques to determine the position of the scope (e.g., the location of the end of the scope) and / or the position of an object within the field of view of the scope. The targeting component may associate the position of the scope (e.g., the coordinates of the scope) with the target location. Additionally or alternatively, in some embodiments, the scope can deliver a reference point to mark the target location, and the position of the reference point can be determined.

[0218] The target location may represent fixed or movable point(s) within the human anatomy and / or coordinate space / system. For example, if the nipple is initially designated as the target location, the coordinates of the target location may be determined and updated as the procedure progresses and the nipple moves (e.g., due to insertion of a medical instrument). Here, the location of the scope (which may be within proximity of the nipple) may be tracked over time and used to update the coordinates of the target location. In some embodiments, the targeting component may estimate / predict the position of the target location. Here, the target location may be represented by a predicted position. For example, the targeting component may use an algorithm that predicts the coordinates of the target location as the human anatomy moves. The predicted coordinates may be used to determine the target trajectory.

[0219] The one or more communication interfaces 2006 may be configured to communicate with one or more devices / sensors / systems. For example, the one or more communication interfaces 2006 may transmit / receive data wirelessly and / or wired over a network. Networks according to embodiments of the present disclosure may include local area networks (LANs), wide area networks (WANs) (e.g., the Internet), personal area networks (PANs), body area networks (BANs), etc. In some embodiments, the one or more communication interfaces 2006 may implement wireless technologies such as Bluetooth, Wi-Fi, near field communication (NFC), etc.

[0220] One or more power supply units 2008 may be configured to manage power for the control system 150 (and / or, in some cases, the robotic system 110). In some embodiments, the one or more power supply units 2008 (and / or any other power supply unit) may include one or more batteries, such as lithium-based batteries, lead-acid batteries, alkaline batteries, and / or another type of battery. That is, the one or more power supply units 2008 may comprise one or more devices and / or circuits configured to provide a power source and / or provide power management functionality. Further, in some embodiments, the one or more power supply units 2008 include a mains power connector configured to couple to an alternating current (AC) or direct current (DC) mains power source.

[0221] The one or more I / O components 2010 may include various components for receiving input, such as interfacing with a user, and / or providing output. The one or more I / O components 2010 may be configured to receive touch, speech, gestures, or any other type of input. In embodiments, the one or more I / O components 2010 may be used to provide input for device / system control, such as control of the robotic system 110, navigation of a scope or other medical instrument attached to the robotic system 110, control of the table 170, control of the fluoroscopy device 190, etc. As shown, the one or more I / O components 2010 may include one or more displays 152 (sometimes referred to as “one or more display devices 152”) configured to display data. The one or more displays 152 may include one or more liquid-crystal displays (LCDs), light-emitting diode (LED) displays, organic LED displays, plasma displays, electronic paper displays, and / or any other type(s) of technology. In some embodiments, the one or more displays 152 include one or more touchscreens configured to receive input and / or display data. Further, the one or more I / O components 2010 may include one or more I / O devices / controllers 156, which may include a touchscreen, a touchpad, a controller, a mouse, a keyboard, a wearable device (e.g., an optical head-mounted display), a virtual or augmented reality device (e.g., a head-mounted display), etc. Additionally, the one or more I / O components 2010 may include one or more speakers 2024 configured to output sound based on an audio signal and / or one or more microphones 2026 configured to receive sound and generate an audio signal. In some embodiments, the one or more I / O components 2010 include or are implemented as a console.

[0222] 20 , control system 150 may include and / or control other components, such as one or more pumps, flow meters, valve controls, and / or fluid access components, to provide controlled irrigation and / or aspiration capabilities to a medical instrument (e.g., a scope), devices that may be deployed through the medical instrument, etc. In some embodiments, irrigation and aspiration capabilities may be delivered directly to the medical instrument through separate cable(s). Additionally, control system 150 may include voltage and / or surge protectors designed to provide filtered and / or protected power to another device, such as robotic system 110, thereby avoiding the placement of power transformers and other auxiliary power components within robotic system 110 and making robotic system 110 smaller and more mobile.

[0223] Control system 150 may also include support equipment for sensors deployed throughout medical system 100. For example, control system 150 may include optoelectronics for detecting, receiving, and / or processing data received from optical sensors and / or cameras. Such optoelectronics may be used to generate real-time images for display on any number of devices / systems included within control system 150. Similarly, control system 150 may also include electronic subsystems for receiving and / or processing signals received from deployed electromagnetic (EM) sensors. In some embodiments, control system 150 may also be used to house and position EM field generators for detection by EM sensors in or on the medical instrument.

[0224] In some embodiments, control system 150 may be coupled to robotic system 110, table 170, medical instruments, etc. via one or more cables or connections (not shown). In some implementations, support functionality from control system 150 can be provided via a single cable, which may simplify and reduce clutter in the operating room. In other implementations, certain functionality may be combined with separate wiring and connections. For example, power may be provided through a single power cable, while support for control, optics, fluidics, and / or navigation may be provided through separate cables.

[0225] The term “control circuitry” (e.g., control circuitry 1902, control circuitry 2002, and / or any other control circuitry) is used herein according to its broad ordinary meaning and may refer to any collection of one or more processors, processing circuits, processing modules / units, chips, dies (e.g., semiconductor dies including one or more active and / or passive devices and / or connection circuits), microprocessors, microcontrollers, digital signal processors, microcomputers, central processing units, graphic processing units, field programmable gate arrays, programmable logic devices, state machines (e.g., hardware state machines), logic circuits, analog circuits, digital circuits, and / or any devices that manipulate signals (analog and / or digital) based on hard-coding of circuit and / or operational instructions. The control circuitry may further include one or more storage devices, which may be embodied in a single memory device, multiple memory devices, and / or embedded circuitry of a device. Such data storage devices may include read-only memory, random access memory, volatile memory, non-volatile memory, static memory, dynamic memory, flash memory, cache memory, data storage registers, and / or any device that stores digital information. It should be noted that in embodiments in which the control circuitry comprises a hardware state machine (and / or implements a software state machine) and comprises analog, digital, and / or logic circuits, the data storage device(s) / register(s) storing any associated operating instructions may be embedded within or external to the circuitry comprising the state machine, analog, digital, and / or logic circuits.

[0226] The term "memory" is used herein according to its broad, ordinary meaning and can refer to any suitable or desirable type of computer-readable medium. For example, a computer-readable medium can include one or more volatile, non-volatile, removable, and / or removable data storage devices implemented using any technology, layout, and / or data structure(s) / protocol, and containing any suitable or desirable computer-readable instructions, data structures, program modules, or other types of data.

[0227] One or more computer-readable media that may be implemented in accordance with embodiments of the present disclosure include, but are not limited to, phase-change memory, static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage device, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device, or any other non-transitory medium that may be used to store information for access by a computing device. As used in certain contexts herein, computer-readable medium may generally not include communication media such as modulated data signals and carrier waves. As such, computer-readable medium should be understood to generally refer to non-transitory media.

[0228] Example I / O Devices 21A and 21B are exemplary detailed diagrams of a controller 2102, according to one or more embodiments. In an example, I / O device(s) 156 of control system 150 and / or another I / O device discussed herein are implemented as controller 2102. However, I / O device(s) 156 may be implemented as other types of devices. FIGS. 21A and 21B show perspective and side views, respectively, of controller 2102, according to certain embodiments.

[0229] The controller 2102 may receive / facilitate axial movement inputs, such as via one or more joysticks 2104, 2106 and / or one or more directional pads 2108. For example, a user may manipulate one or more joysticks 2104, 2106 (and / or possibly one or more directional pads 2108) to provide directional inputs for controlling the medical instrument. In some embodiments, the joysticks 2104, 2106 provide analog inputs, while the directional pad 2108 provides digital inputs. However, any of the joysticks 2104, 2106 and / or directional pad 2108 may provide analog inputs and / or digital inputs. In examples, inputs received via one or more directional pads 2108 may be used to control a user interface, while inputs received via one or more joysticks 2104, 2106 may be used to control movement of the medical instrument. The controller 2102 may further include a plurality of buttons 2110 for providing additional control inputs. 21B, the controller 2102 includes four buttons on the side of the controller: R1 2112, R2 2114, L1 2116, and L2 2118. Other embodiments may include a different number of buttons and / or a different layout. In some embodiments, the controller 2102 may be a game-type console controller (and / or similar to a game-type console controller) repurposed to work with the control system 150. For example, the controller game firmware may be overwritten with medical device firmware, and / or an input device manager may be installed in a component of the medical system 100 (e.g., the control system 150) to translate inputs from the controller 2102 into inputs understandable by the robotic system 110.

[0230] The controller 2102 may be implemented to receive inputs for controlling / actuating the medical instrument. For example, the joystick 2104 may receive directional inputs indicating a direction to move the medical instrument (e.g., right, left, diagonal, up, down, insert, retract, etc.). Illustratively, as described above, a user may tilt the joystick 2106 left or right to move the catheter / scope left or right relative to the control frame (which may depend on the control mode). In another example, a user may push / tilt the joystick 2104 forward / backward relative to FIG. 21A to insert / retract the catheter / scope (depending on the control mode). While certain controls are described as mapping to specific functions, the controller 2102 may be configured in a variety of other ways. In some embodiments, the controller 2102 may be customized with a user interface that allows for assigning functions to specific controls on the controller 2102.

[0231] In some embodiments, the controller 2102 may implement controls (e.g., one or more of the controls 2104-2118 and / or other controls) to facilitate switching between different medical instruments. For example, a user may select one of the buttons 2110 to switch from driving the scope to driving the catheter. Additionally, the controller 2102 may implement controls to switch between control / drive modes of the medical instrument(s), such as direct control mode, inverted control mode, etc. Additionally, the controller 2102 may implement controls to navigate to specific interfaces, such as a drive interface, a calibration interface, etc.

[0232] Additional Embodiments Depending on the embodiment, certain acts, events, or functions of any of the processes or algorithms described herein may be performed in a different order, added, merged, or omitted entirely, and thus, in a particular embodiment, not all described acts or events may be required to practice a process.

[0233] In particular, hypothetical language used herein, such as "can," "could," "might," "may," "eg," and the like, is intended in its ordinary sense unless specifically stated otherwise or understood otherwise within the context in which it is used, and is generally intended to convey that certain embodiments include certain features, elements, and / or steps, while other embodiments do not. Thus, such hypothetical language is generally not intended to imply that features, elements, and / or steps are required in any way for one or more embodiments, or that one or more embodiments necessarily include logic for determining whether those features, elements, and / or steps are included in or performed in any particular embodiment, with or without author input or prompting. Terms such as "comprising," "including," "having," and the like, are used in their ordinary sense and are used inclusively in a non-limiting manner and do not exclude additional elements, features, acts, operations, etc. Also, when the term "or" is used, for example, to connect a list of elements, the term "or" is used in its inclusive sense (and not its exclusive sense) to mean one, some, or all of the listed elements. Unless specifically stated otherwise, connective language such as the phrase "at least one of X, Y, and Z" is understood in the context as it is commonly used to convey that an item, term, element, etc. can be either X, Y, or Z. Thus, such connective language is generally not intended to imply that a particular embodiment requires that at least one of X, at least one of Y, and at least one of Z, respectively, be present.

[0234] In the foregoing description of the embodiments, it should be understood that various features are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects. However, this method of disclosure should not be interpreted as reflecting an intention that any claim requires more features than are expressly recited in that claim. Moreover, any component, feature, or step illustrated and / or described in a particular embodiment(s) herein may be applied to or used in conjunction with any other embodiment. Moreover, no component, feature, step, or group of components, features, or steps is necessary or essential for each embodiment. Accordingly, it is intended that the scope of the disclosure disclosed herein and claimed below, and the scope of the invention(s) claimed below, should not be limited by the particular embodiments described above, but should be determined solely by a fair reading of the following claims.

[0235] It should be understood that certain ordinal terms (e.g., "first" or "second") may be provided for ease of reference and do not necessarily imply any physical characteristics or ordering. Thus, as used herein, ordinal terms (e.g., "first," "second," "third," etc.) used to modify elements such as structures, components, operations, etc., do not necessarily indicate a priority or order of the element relative to any other elements, but rather may generally distinguish the element from other elements having a similar or identical name (apart from the use of the ordinal terminology). Furthermore, as used herein, the indefinite articles ("a" and "an") may indicate "one or more" rather than "one." Furthermore, an action performed "based on" a condition or event may also be performed based on one or more other conditions or events not expressly recited.

[0236] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the exemplary embodiments belong. It is further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning consistent with their meaning in the context of the relevant art, and should not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0237] Spatially relative terms such as "outside," "inside," "upper," "lower," "below," "upper," "vertical," "horizontal," and similar terms may be used herein for ease of description to describe the relationship between one element or component and another element or component as illustrated in the drawings. It should be understood that spatially relative terms are intended to encompass different orientations of the device in use or operation in addition to the orientation depicted in the drawings. For example, if a device shown in the drawings were inverted, a device positioned "below" or "under" another device would be disposed "above" the other device. Thus, the exemplary term "lower" can include both lower and upper positions. Devices may also be oriented in other directions, and thus spatially relative terms may be interpreted differently depending on the orientation.

[0238] Unless otherwise specified, comparative and / or quantitative terms such as "less," "more," "greater than," etc. are intended to encompass the notion of equality. For example, "less" can mean "less than" in the strict mathematical sense, but also "less than or equal to."

[0239] [Embodiment] (1) A health care system: a first instrument configured to access the anatomical site via a first access pathway; a second instrument including an imaging component configured to provide image data representative of the anatomical location and the first instrument, the second instrument configured to access the anatomical location via a second access path; and One or more computer-readable media storing executable instructions that, when executed by control circuitry, cause the control circuitry to: Identifying a difference between a first coordinate system associated with the first instrument and a second coordinate system associated with the second instrument; and updating a control reference frame associated with the first instrument based at least in part on the difference. (2) A medical system as described in embodiment 1, wherein the first coordinate system indicates the roll of the distal end of the first instrument and the second coordinate system indicates the roll of the distal end of the second instrument. (3) the one or more computer-readable media further store executable instructions that, when executed by the control circuitry, cause the control circuitry to: displaying an alignment indicator, the alignment indicator representing the first coordinate system; receiving input including an adjustment to the alignment indicator; 2. The medical system of claim 1, wherein the difference between the first coordinate system and the second coordinate system is determined based at least in part on the adjustment to the alignment indicator. (4) The medical system of embodiment 3, wherein the first instrument includes one or more markings on the first instrument, and the alignment indicator represents the orientation of the one or more markings. (5) The medical system of embodiment 4, wherein the one or more computer-readable media further store executable instructions that, when executed by the control circuit, cause the control circuit to display a user interface, the user interface including a graphical representation of the image data and the alignment indicator, the alignment indicator including a ring and one or more marking indicators representing the orientation of the one or more markings.

[0240] (6) further comprising a robotic manipulator coupled to the first instrument and configured to control movement of the first instrument; The medical system of embodiment 1, wherein the one or more computer-readable media further store executable instructions that, when executed by the control circuit, cause the control circuit to determine the first coordinate system based on at least one of the position or orientation of the robot manipulator. (7) The one or more computer-readable media further store executable instructions that, when executed by the control circuitry, cause the control circuitry to: receiving a directional input signal from an input device, the directional input signal indicating a direction of movement of the first implement; A medical system as described in embodiment 1, which controls the movement of the first instrument based at least in part on the directional input signal and the control reference coordinate system. (8) A method comprising: a control circuit receiving image data from a first instrument positioned at a target anatomical site, the image data representing at least a portion of a second instrument; displaying a pictorial representation of the image data; and receiving an input indicating an orientation of the second instrument in the graphical representation; the control circuitry calibrating a control scheme for the second instrument based at least in part on the input. (9) receiving a directional input indicating a direction of movement of the second instrument; 9. The method of claim 8, further comprising: controlling movement of the second instrument based at least in part on the directional input and the control scheme. (10) calibrating the control scheme determining a roll of the distal end of the first instrument relative to the second instrument based at least in part on the input; adjusting a control reference coordinate system used to control the second instrument based at least in part on the roll of the distal end of the first instrument relative to the second instrument.

[0241] (11) The method of embodiment 10, wherein determining the roll of the first instrument relative to the second instrument includes determining a roll of a distal end of the first instrument relative to a distal end of the second instrument. (12) The method further includes displaying an alignment indicator representing the orientation of the second instrument; 9. The method of claim 8, wherein receiving the input includes receiving an adjustment to the alignment indicator. (13) The method of embodiment 12, wherein the second instrument includes one or more markings on a distal end of the second instrument, and the alignment indicator represents an orientation of the one or more markings. (14) The method of embodiment 8, wherein the first instrument is an endoscope and the second instrument is a catheter. (15) One or more non-transitory computer-readable media storing computer-executable instructions that, when executed by control circuitry, cause the control circuitry to: receiving image data from a direct access device positioned at a target anatomical location, the image data representing at least a portion of the percutaneous access device positioned at the target anatomical location; generating roll data indicative of a roll of the distal end of the direct access device relative to the percutaneous access device based at least in part on the image data; and controlling movement of the percutaneous access device based at least in part on the roll data.

[0242] (16) The one or more non-transitory computer-readable media described in embodiment 15, wherein the direct access device includes an endoscope and the percutaneous access device includes a catheter. (17) The one or more non-transitory computer-readable media described in embodiment 15, wherein the roll data indicates an orientation of a coordinate system associated with the direct access device relative to a coordinate system associated with the percutaneous access device. (18) One or more non-transitory computer-readable media according to embodiment 15, wherein the image data represents a distal end of the percutaneous access device including one or more markings. (19) The operation is displaying a pictorial representation of the image data; and displaying an alignment indicator, the alignment indicator representing an estimated orientation of the percutaneous access device; and receiving input including an adjustment to the alignment indicator; 16. The one or more non-transitory computer-readable media of claim 15, wherein the roll data is generated based at least in part on the adjustment to the alignment indicator. (20) The operation is One or more non-transitory computer-readable media as described in embodiment 19, further comprising determining the estimated orientation of the percutaneous access device based on at least one of a position or orientation of a robotic manipulator configured to control the percutaneous access device.

[0243] (21) The operation is receiving a directional input signal from an input device, the directional input signal indicating a direction of movement of the percutaneous access device; One or more non-transitory computer-readable media as described in embodiment 15, wherein controlling the movement of the percutaneous access device includes generating a control signal for controlling the movement of the percutaneous access device based at least in part on the roll data. (22) The operation is performing one or more image processing techniques using the image data; and determining an orientation of the percutaneous access device relative to the direct access device based at least in part on the one or more image processing techniques; 16. The one or more non-transitory computer-readable media of embodiment 15, wherein the roll data is generated based at least in part on the orientation of the percutaneous access device relative to the direct access device. (23) The operation is displaying a pictorial representation of the image data via a user interface; displaying an instruction via the user interface, the instruction indicating selecting a particular directional control on an input device; and receiving a directional input signal from the input device, the directional input signal being associated with the particular directional control; receiving an input indicating a direction in which the percutaneous access device has been moved relative to the user interface; 16. The one or more non-transitory computer-readable media of claim 15, wherein the role data is generated based at least in part on the input. (24) The operation is displaying a pictorial representation of the image data via a user interface; displaying instructions via the user interface, the instructions indicating to move the percutaneous access device in a particular direction relative to the user interface; and receiving a directional input signal from the input device; controlling the percutaneous access device to move based at least in part on the directional input signal; One or more non-transitory computer-readable media as described in embodiment 15, wherein the roll data is generated based at least in part on the directional input signal. (25) A system comprising: a first robotic manipulator configured to manipulate the direct access instrument; a second robotic manipulator configured to manipulate the percutaneous access device; a control circuit communicatively coupled to the first robotic manipulator and the second robotic manipulator, the control circuit comprising: receiving image data from the direct access device, the image data representing at least a portion of the percutaneous access device; displaying a pictorial representation of the image data; and receiving an input indicating an orientation of the percutaneous access device in the graphical representation; calibrating a control scheme for the percutaneous access device based at least in part on the input.

[0244] (26) The direct access device configured to access a target anatomical site via a natural lumen of a patient; The system of embodiment 25 further comprises a percutaneous access device configured to access the target anatomical site via a percutaneous access pathway within the patient, the percutaneous access device including one or more markings at a distal end. (27) The method further comprises: a display configured to display the image representation and an alignment indicator including one or more marking indicators representing an orientation of the one or more markings; 27. The system of claim 26, wherein the input includes an adjustment to the alignment indicator. (28) The system of embodiment 26, wherein the direct access device comprises an endoscope and the percutaneous access device comprises a catheter. (29) The calibrating of the control scheme comprises: determining a roll of the distal end of the direct access device relative to the percutaneous access device based at least in part on the input; and adjusting a control reference coordinate system used to control the percutaneous access device based at least in part on the roll of the distal end of the direct access device relative to the percutaneous access device. (30) The operation is receiving a first directional input signal indicating a direction of movement of the percutaneous access device; 26. The system of claim 25, further comprising: controlling movement of the percutaneous access device based at least in part on the first directional input signal and the control scheme.

Claims

1. 1. A healthcare system comprising: a first instrument configured to access the anatomical site via a first access pathway; a second instrument including an imaging component configured to provide image data representative of the anatomical location and the first instrument, the second instrument configured to access the anatomical location via a second access path; and One or more computer-readable media storing executable instructions that, when executed by control circuitry, cause the control circuitry to: Identifying a difference between a first coordinate system associated with the first instrument and a second coordinate system associated with the second instrument; updating a control reference frame associated with the first instrument based at least in part on the difference; and displaying an alignment indicator, the alignment indicator representing the first coordinate system; and receiving input including adjustments to the alignment indicators; The difference between the first coordinate system and the second coordinate system is determined based at least in part on the adjustment to the alignment indicator.

2. The medical system of claim 1 , wherein the first coordinate system represents a roll of a distal end of the first instrument and the second coordinate system represents a roll of a distal end of the second instrument.

3. The medical system of claim 1 , wherein the first instrument includes one or more markings thereon, and the alignment indicator represents an orientation of the one or more markings.

4. 4. The medical system of claim 3, wherein the one or more computer-readable media further store executable instructions that, when executed by the control circuitry, cause the control circuitry to display a user interface, the user interface including a graphical representation of the image data and the alignment indicator, the alignment indicator including a ring and one or more marking indicators representing the orientation of the one or more markings.

5. a robotic manipulator coupled to the first instrument and configured to control movement of the first instrument; 2. The medical system of claim 1, wherein the one or more computer-readable media further store executable instructions that, when executed by the control circuitry, cause the control circuitry to determine the first coordinate system based on at least one of a position or an orientation of the robotic manipulator.

Images