Methods and systems for mapping and navigation

Abstract

Certain aspects relate to systems and techniques for mapping and / or navigating an internal region of a body using a robotically controllable medical instrument. The instrument may include a position sensor that provides position information as the instrument navigates within the internal region. Visual indicia derived from the position information may be superimposed on a reference image of the internal region. The visual indicia may characterize historical positions of the instrument. The instrument may include an imaging device. Images of the internal region captured with the imaging device may be linked to the location within the internal region where the image was captured.

Description

Technical Field

[0001] (Priority) This application claims priority to U.S. Provisional Patent Application No. 62 / 637,048, filed on March 1, 2018, which is incorporated herein by reference.

[0002] (Field of the Invention) The present disclosure generally relates to mapping and / or navigation of internal regions of the body, and more specifically to methods and systems for mapping and / or navigation of internal regions of the body using robotically controllable medical devices.

Background Art

[0003] Medical procedures such as endoscopy (e.g., ureteroscopy) may involve accessing and visualizing internal regions of a patient's body (e.g., the kidneys) for diagnostic and / or therapeutic purposes.

[0004] Ureteroscopy is a medical procedure commonly used for the treatment of kidney stones. During the procedure, a thin, flexible tubular tool or instrument known as a ureteroscope can be inserted into the urethra, through the bladder and ureter, and into the kidney.

[0005] In certain procedures, a robotically controllable medical system can be used to control the insertion and / or operation of the instrument. A robotically controllable medical system can include a robotic arm or other instrument positioning device having a manipulator assembly used to control the positioning of the instrument during the procedure.

Summary of the Invention

Means for Solving the Problems

[0006] Embodiments of the present disclosure relate to systems and techniques for mapping and / or navigating internal regions of the body using robot-controllable medical instruments. The systems and techniques may be configured to generate visual markers indicating the instrument's historical (e.g., previous) positions as the instrument navigates within the internal region. The visual markers may be derived from positional information received from position sensors. The visual markers may be superimposed on a reference image of the internal region. The visual markers may form a map that can represent the instrument's previous positions. The map may be used to visualize the anatomical structure of the internal region and may be used by a physician to navigate the internal region. In some embodiments, the visual markers include points or traces indicating the instrument's path as it navigates the internal region. In some embodiments, the visual markers may include a mesh structure representing the anatomical structure of the internal region.

[0007] The method and technique may also be configured to receive image data from an image sensor positioned on the instrument. The image data may include still images or videos. The method and technique may link the image data with location data so that images captured at a particular location can be played back and displayed to the user. In some embodiments, images are played back and displayed when the user selects a location within an internal area, or when the instrument is positioned at a location where the linked image exists. Furthermore, the method and technique may be configured to allow a physician to tag specific features or locations of interest.

[0008] Accordingly, one embodiment relates to a method for mapping an internal region of the body. The method includes displaying a reference image of the internal region of the body, moving an instrument within the internal region of the body, the instrument including at least one position sensor, receiving position information from at least one position sensor, the position information including a plurality of position datasets, each position dataset indicating the position of the instrument during its movement, and characterizing the historical position of the instrument during its movement within the internal region of the body by superimposing visual markers derived from at least a subset of the position datasets onto the reference image.

[0009] In some embodiments, the method may include one or more of the following features in any combination: (a) the reference image includes an image captured during retrograde pyelonephrography; (b) the reference image includes a fluoroscopic or ultrasound image; (c) the reference image is captured during computed tomography (CT) or magnetic resonance imaging (MRI) procedure; (d) the reference image is captured during surgery; (e) the reference image is captured before surgery; (f) instrument image data is received from an imaging device positioned on the instrument (the instrument image data includes multiple images captured by the imaging device while the instrument is moving), and for at least a subset of the multiple images, each image in the subset is linked to a position dataset indicating the location where the image was captured; (g) the linked images are stored for use in future procedures; (h) the user input for position selection is received, and the linked image corresponding to the user input is displayed; (i) the current position of the instrument is determined using a position sensor, and the linked image corresponding to the determined current position is displayed; (j) the instrument image data is automatically captured; (k) the instrument image data The data is captured when a user command is received, (l) tags the location or features of an object within the internal region, (m) tagging includes receiving user input, (n) tagging includes automatically detecting the location or features of an object, (o) overlays the tagged location or features of an object onto a reference image, (p) connects visual markers to characterize the historical path of the instrument's movement, (q) the visual markers include a mesh, (r) the mesh indicates the anatomical structure of the internal region, (s) the mesh The method is derived from a subset of positional datasets and image data received from an imaging device on the instrument, (t) the subset of positional datasets is superimposed on a reference image at a duration frequency, (u) the subset of positional datasets is superimposed on a reference image at a positional frequency, (v) visual markers are displayed during surgery, (w) visual markers are memorized for use in future medical procedures, (x) the internal region of the body includes the kidney, the method involves moving the instrument into the renal calyces, the entrance to the renal calyces, the pole of the kidney, the stones within the kidney,and further comprising tagging at least one of the regions of transitional cell carcinoma, and / or (y) adjusting the position determined by the position sensor to account for physiological dynamics and superimposing the adjusted position onto a reference image.

[0010] In another embodiment, a non-temporary computer-readable storage medium storing instructions is disclosed. When executed, the instructions can cause the device's processor to at least: move an instrument within the internal region of a body; receive positional information from at least one position sensor of the instrument during the movement of the instrument, the positional information comprising a plurality of positional datasets, each positional dataset indicating the position of the instrument; display an image of the internal region of the body; and characterize the historical position of the instrument during its movement within the internal region of the body by superimposing visual markers derived from at least a subset of the positional datasets onto the image.

[0011] In some embodiments, the non-temporary computer-readable storage medium may include one or more of the following features in any combination: (a) The reference image includes an image captured during retrograde pyelonephrography; (b) The reference image includes a fluoroscopic or ultrasound image; (c) The reference image is captured during computed tomography (CT) or magnetic resonance imaging (MRI) procedure; (d) When the command is executed, it causes the processor to further capture a reference image during surgery; (e) The reference image is captured before surgery; (f) When the command is executed, it causes the processor to receive instrument image data from an imaging device positioned on the instrument, the instrument image data including a plurality of images captured by the imaging device during instrument movement, and to further link at least a subset of the plurality of images to a location dataset indicating the location where the image was captured; (g) When the command is executed, it causes the processor to further store the linked images for use in future procedures; (h) When the command is executed, it causes the processor to receive user input for location selection and user input (i) When the command is executed, the processor further performs the following: (i) Display the linked image corresponding to the current position of the instrument using a position sensor, and (j) Display the linked image corresponding to the determined current position, (k) Display image data is automatically captured, (l) Display image data is captured when a user command is received, (m) When the command is executed, the processor further performs the following: Tag the location or feature of an object within the internal region, (n) Tagging includes receiving user input, (o) When the command is executed, the processor further performs the following: Overlay the tagged location or feature of an object onto a reference image, (p) When the command is executed, the processor further performs the following: Connect visual markers to characterize the historical path of the instrument's movement, (q) Visual markers include a mesh, (r) The mesh indicates the anatomical structure of the internal region, (s) The mesh is(t) a subset of the positional dataset is superimposed on a reference image at a duration frequency, (u) a subset of the positional dataset is superimposed on a reference image at a positional frequency, (v) the command, when executed, causes the processor to further display visual markers during surgery, (w) the command, when executed, causes the processor to further store the visual markers for use in future medical procedures, (x) if the internal region of the body includes the kidney, the command, when executed, causes the processor to further move the instrument into the renal calyces and tag at least one of the entrance to the renal calyces, the pole of the kidney, stones within the kidney, and / or areas of transitional cell carcinoma, and / or (y) the command, when executed, causes the processor to further adjust the position determined by the position sensor to take physiological dynamics into account and superimpose the adjusted position onto the reference image.

[0012] In another embodiment, a robotic surgical system is disclosed. The robotic surgical system may include an instrument having an elongated body and at least one position sensor disposed on the elongated body; at least one computer-readable memory storing executable instructions; and one or more processors communicating with the at least one computer-readable memory, which are configured to execute instructions and cause the system to at least: move the instrument within the internal region of the body; receive position information from at least one position sensor while the instrument is moving; display an image of the internal region of the body; and characterize the historical position of the instrument as it moves within the internal region of the body by superimposing visual markers derived from at least a subset of a position dataset onto the image.

[0013] In some embodiments, the system may include one or more of the following features in any combination: (a) The position sensor includes an electromagnet sensor, (b) The position sensor includes a shape-detecting fiber, (c) The position sensor is positioned on the distal end of an elongated body, (d) The instrument includes an endoscope, (e) The instrument includes a ureteroscope, (f) The elongated body is articulated to control the posture of the instrument, (g) An instrument positioning device connected to the instrument, the instrument positioning device is configured to operate the instrument, (h) The instrument positioning device includes a robotic arm, (i) The reference image includes an image captured during retrograde pyelography, (j) The reference image includes a fluoroscopic image or an ultrasound image, (k) The reference image is captured during computed tomography (CT) or magnetic resonance imaging (MRI) procedure, (l) When the command is executed, it causes one or more processors to further capture a reference image during surgery, (m) The reference image is captured before surgery, (n) When the command is executed, it causes one or more processors to receive instrument image data from an imaging device positioned on the instrument, the instrument image data being captured during movement of the instrument (o) The instruction, when executed, causes one or more processors to store the linked images for use in future procedures, including multiple images captured by the imaging device, and for at least a subset of the multiple images, to link each image in the subset to a location dataset indicating the location where the image was captured, (p) The instruction, when executed, causes a processor to receive user input for location selection and display the linked image corresponding to the user input, (q) The instruction, when executed, causes one or more processors to determine the current location of the instrument using a location sensor and display the linked image corresponding to the determined current location, (r) Instrument image data is captured automatically, (s) Instrument image data is captured when a user command is received, (t) The instruction, when executed, causes one or more processors to tag the location or feature of an object within an internal region, (u) Tagging is(v) Tagging, which includes receiving user input, includes automatically detecting the location or features of an object, (w) when executed, the instruction causes one or more processors to further overlay the tagged location or features of an object onto a reference image, (x) when executed, the instruction causes one or more processors to further connect visual markers to characterize the historical path of the instrument's movement, (y) the visual markers include a mesh, (z) the mesh represents the anatomical structure of an internal region, (aa) the mesh is derived from a subset of a location dataset and image data received from an imaging device on the instrument, (bb) the subset of the location dataset is overlaid on the reference image at a duration frequency, (cc) the subset of the location dataset is overlaid on the reference image at a position frequency The (dd) command, when executed, causes one or more processors to further display visual markers during surgery, the (ee) command, when executed, causes one or more processors to further store the visual markers for use during future medical procedures, the (ff) command, when executed, causes one or more processors to further move an instrument into the renal calyces and tag at least one of the following: the entrance to the renal calyces, the pole of the kidney, a stone within the kidney, and / or a region of transitional cell carcinoma, and / or the (gg) command, when executed, causes one or more processors to further adjust the position determined by the position sensor to take physiological dynamics into account and to overlay the adjusted position onto a reference image.

[0014] In another embodiment, a non-temporary computer-readable storage medium storing instructions is described. When executed, the instructions may cause the device's processor to: move an instrument within the internal region of a body; receive position information from at least one position sensor of the instrument during the movement of the instrument, the position information comprising a plurality of position datasets, each position dataset indicating the position of the instrument during the movement of the instrument; receive image data from an imaging device of the instrument within the internal region during the movement of the instrument, the image data comprising one or more images captured by the imaging device at one or more locations within the internal region; link at least a subset of one or more images to at least a subset of the position datasets based on the location where each image was captured, as determined by the position sensor; determine user input, including a position selection; and display one of the linked images corresponding to the position selection.

[0015] In some embodiments, a non-temporary computer-readable storage medium may include one or more of the following features in any combination: (a) determining a user command includes receiving a user command; (b) image data includes still images; (c) image data includes video; (d) a subset of location datasets is selected by duration frequency; (e) a subset of location datasets is selected by location frequency; (f) location information includes information indicating the orientation of an instrument; (g) image data is captured automatically; (h) image data is captured when a user command is received; (i) an instruction, when executed, causes the processor to receive user input associated with the current location and to link the user input to a linked image corresponding to the current location; and / or (j) an instruction, when executed, causes the processor to further detect features of a target within the image of the image data.

[0016] In another embodiment, a robotic surgical system for navigating an internal region of the body is described. The system may include an instrument comprising an elongated body, at least one position sensor positioned on the elongated body, and an imaging device positioned on the elongated body; at least one computer-readable memory storing executable instructions; and one or more processors, which communicate with the at least one computer-readable memory and execute instructions to cause the system to at least: move the instrument within the internal region of the body; receive position information from at least one position sensor during the movement of the instrument, the position information comprising a plurality of position datasets, each position dataset indicating the position of the instrument during its movement; receive image data from the imaging device, the image data comprising one or more images captured by the imaging device during the movement of the instrument at one or more locations within the internal region; and link at least a subset of one or more images to at least a subset of the position datasets based on the location where each image is captured, as determined by the position sensor.

[0017] In some embodiments, the system may include one or more of the following features in any combination: (a) the position sensor includes an EM sensor; (b) the position sensor includes a shape-sensing fiber; (c) the position sensor is positioned on the distal end of an elongated body; (d) the instrument includes an endoscope; (e) the instrument includes a ureteroscope; (f) the elongated body is articulated to control the posture of the instrument; (g) an instrument positioning device connected to the instrument, the instrument positioning device being configured to manipulate the instrument; and / or (h) the instrument positioning device including a robotic arm.

[0018] In another embodiment, a method for navigating an internal region of a body is disclosed. This method may include moving an instrument within an internal region of a body, the instrument comprising at least one position sensor and at least one imaging device; receiving position information from at least one position sensor of the instrument, the position information comprising a plurality of position datasets, each position dataset indicating the position of the instrument as it moves; receiving image data from the imaging device of the instrument, the image data comprising one or more images captured by the imaging device at one or more locations within the internal region; linking at least a subset of the one or more images to at least a subset of the position datasets based on the location where each image was captured, as determined by the position sensor; determining the current position of the instrument using at least one position sensor, the current position corresponding to a current position dataset among the plurality of position datasets; and displaying images linked to the current position dataset on a user display. [Brief explanation of the drawing]

[0019] The disclosed embodiments will be described below in conjunction with the attached drawings, and the disclosed embodiments will be illustrative but not limited to them, and similar designations will indicate similar elements. [Figure 1] This document illustrates an embodiment of a cart-based robotic system positioned for diagnostic and / or therapeutic bronchoscopy procedures. [Figure 2] Further aspects of the robot system shown in Figure 1 are depicted. [Figure 3] Figure 1 shows an embodiment of the robotic system deployed for ureteroscopy. [Figure 4] Figure 1 shows an embodiment of a robotic system deployed for vascular procedures. [Figure 5] This document illustrates an embodiment of a table-based robotic system positioned for bronchoscopy procedures. [Figure 6] An alternative diagram of the robot system shown in Figure 5 is provided. [Figure 7] An exemplary system configured to store a robotic arm (s). [Figure 8] Embodiments of a table - based robotic system configured for ureteroscopy procedures are shown. [Figure 9] Embodiments of a table - based robotic system configured for laparoscopic procedures are shown. [Figure 10] Embodiments of the table - based robotic system of FIGS. 5 - 9 with pitch or tilt adjustment are shown. [Figure 11] Provides a detailed illustration of the interface between the table and the column of the table - based robotic system of FIGS. 5 - 10. [Figure 12] An exemplary instrument driver is shown. [Figure 13] An exemplary medical instrument with a pair of instrument drivers is shown. [Figure 14] Alternative designs of an instrument driver and an instrument are shown where the axis of the drive unit is parallel to the axis of the elongated shaft of the instrument. [Figure 15] A block diagram showing a position - specifying system for estimating the position of one or more elements of the robotic system of FIGS. 1 - 10, such as the position of the instruments of FIGS. 13 and 14, according to an exemplary embodiment, is shown. [Figure 16A] A first exemplary fluoroscopic image showing an instrument for navigating within the kidney. [Figure 16B] A second exemplary fluoroscopic image showing the instrument of FIG. 16A for navigating within the kidney. [Figure 16C] A third exemplary fluoroscopic image showing the instrument of FIG. 16A for navigating within the kidney. [Figure 17] An exemplary display of visual markers showing position data received from a position sensor of an instrument for navigating within the kidney is shown. [Figure 18] A display of the visual markers of FIG. 17 superimposed on a reference image of the kidney is shown. [Figure 19]This demonstrates that images captured by imaging devices of medical instruments navigating within a patient's internal region can be linked to the location of the medical instrument from which the image was captured. [Figure 20A] This flowchart shows an exemplary method for mapping internal regions of the body. [Figure 20B] This flowchart shows another exemplary method for mapping internal regions of the body. [Figure 21A] This is a flowchart illustrating exemplary methods for navigating internal regions of the body. [Figure 21B] This flowchart shows another exemplary method for navigating internal regions of the body. [Figure 22] Block diagram showing a specific component of one embodiment of a system for mapping and / or navigating internal regions of the body. [Modes for carrying out the invention]

[0020] 1.Overview Aspects of this disclosure may be integrated into a robot-controllable medical system capable of performing a variety of medical procedures, including both minimally invasive procedures such as laparoscopy and non-invasive procedures such as endoscopy. Among endoscopic procedures, the system may be capable of performing bronchoscopy, ureteroscopy, gastroscopy, and others.

[0021] In addition to performing a wide range of procedures, the system can offer additional benefits to assist physicians, such as enhanced imaging and guidance. Furthermore, the system can provide physicians with the ability to perform procedures from an ergonomic position without requiring cumbersome arm movements and positions. Moreover, the system can provide physicians with improved ease of use, as one or more of the system's instruments can be controlled by a single user.

[0022] Hereinafter, various embodiments will be described in conjunction with the drawings for illustrative purposes. Many other embodiments of the disclosed concepts are possible, and various advantages may be achieved in the disclosed embodiments. Headings are included herein for reference and to help locate the various sections. These headings are not intended to limit the scope of the concepts described therein. Such concepts may be applicable throughout this specification.

[0023] A. Robot System - Cart Robot-controlled medical systems can be configured in various ways depending on the specific procedure. Figure 1 shows an embodiment of a cart-based robot-controlled system 10 positioned for a diagnostic and / or therapeutic bronchoscopy procedure. During a bronchoscopy, the system 10 may include a cart 11 having one or more robotic arms 12 for delivering medical instruments, such as a maneuverable endoscope 13 which may be a procedure-specific bronchoscope for bronchoscopy, to a natural orifice access point for delivering the diagnostic and / or therapeutic instrument (i.e., the patient's mouth positioned on a table in this embodiment). As shown, the cart 11 may be positioned close to the patient's upper torso to provide access to the access point. Similarly, the robotic arms 12 may be actuated to position the bronchoscope relative to the access point. The arrangement in Figure 1 may also be used when performing gastrointestinal (GI) procedures using a gastroscopy, a specialized endoscope for GI procedures. Figure 2 draws an exemplary embodiment of the cart in more detail.

[0024] Continuing to refer to Figure 1, once the cart 11 is properly positioned, the robotic arm 12 can insert the maneuverable endoscope 13 into the patient robotically, manually, or a combination thereof. As shown, the maneuverable endoscope 13 may include at least two nesting parts, such as an inner leader portion and an outer sheath portion, each portion coupled to a separate instrument driver from a set of instrument drivers 28, each instrument driver coupled to the distal end of an individual robotic arm. This linear arrangement of the instrument drivers 28, which facilitates the alignment of the leader portion coaxially with the sheath portion, creates a “virtual rail” 29 that can be repositioned in space by manipulating one or more robotic arms 12 at different angles and / or positions. The virtual rail described herein is shown in the figures using dashed lines, and therefore the dashed lines do not indicate any physical structure of the system. Translation of the instrument drivers 28 along the virtual rail 29 causes the inner leader portion to nest with the outer sheath portion or moves the endoscope 13 forward or backward from the patient. The angle of the virtual rail 29 may be adjusted, translated, and pivoted based on clinical use or physician preference. For example, in bronchoscopy, the angle and position of the virtual rail 29 as shown represent a compromise that provides physician access to the endoscope 13 while minimizing friction caused by bending the endoscope 13 into the patient's mouth.

[0025] The endoscope 13 may be directed downwards through the patient's trachea and lungs after insertion using precise commands from the robotic system until it reaches the target destination or surgical site. To enhance navigation through the patient's lung network and / or reach the desired target, the endoscope 13 may be manipulated to extend the inner leader portion in a nested manner from the outer sheath portion to obtain increased articulation and a larger bending radius. The use of a separate instrument driver 28 also allows the leader portion and the sheath portion to be driven independently of each other.

[0026] For example, the endoscope 13 may be directed to deliver a biopsy needle to a target, such as a lesion or nodule in the patient's lung. The needle may be deployed down the working channel along the length of the endoscope to obtain a tissue sample to be analyzed by a pathologist. Depending on the pathological results, additional tools may be deployed down the working channel of the endoscope for further biopsies. After identifying the nodule as malignant, the endoscope 13 may endoscopically deliver instruments to excise the potentially cancerous tissue. In some cases, diagnostic and therapeutic procedures may need to be delivered in separate procedures. In these situations, the endoscope 13 may also be used to deliver a criterion to “mark” the location of the target nodule. In other examples, diagnostic and therapeutic procedures may be delivered during the same procedure.

[0027] System 10 may also include a movable tower 30 connected to the cart 11 via support cables, which can provide support for control, electronics, fluid mechanics, optics, sensors, and / or power to the cart 11. Placing such functions within the tower 30 allows for a smaller form factor cart 11 that can be easily adjusted and / or rearranged by the operating physician and their staff. Additionally, the separation of functions between the cart / table and the support tower 30 reduces clutter in the operating room and facilitates improved clinical workflow. The cart 11 may be positioned close to the patient, while the tower 30 may be housed in a remote location so as not to interfere during procedures.

[0028] In supporting the robotic system described above, the tower 30 may include components of a computer-based control system that store computer program instructions in a non-temporary computer-readable storage medium, such as a persistent magnetic memory drive or a solid-state drive. Execution of these instructions may occur within the tower 30, or the cart 11 may control the entire system or subsystem. For example, when executed by the processor of the computer system, the instructions may cause components of the robotic system to actuate associated carriages and arm mounts, actuate a robotic arm, or control a medical device. For example, in response to receiving a control signal, motors in the joints of the robotic arm may position the arm in a particular posture.

[0029] Tower 30 may also include a pump, flow meter, valve control, and / or fluid access to provide controlled irrigation and suction capabilities to the system which can be deployed through the endoscope 13. These components may also be controlled using the computer system of Tower 30. In some embodiments, the irrigation and suction capabilities may be delivered directly to the endoscope 13 through separate cables.

[0030] Tower 30 may include voltage and surge protectors designed to provide filtered and protected power to cart 11, thereby avoiding the placement of power transformers and other auxiliary power components within cart 11, resulting in a smaller and more portable cart 11.

[0031] Tower 30 may also include support equipment for sensors positioned throughout the robotic system 10. For example, Tower 30 may include optoelectronic equipment for detecting, receiving, and processing data received from optical sensors or cameras through the robotic system 10. In combination with a control system, such optoelectronic equipment may be used to generate real-time images for display in any number of consoles deployed throughout the system, including within Tower 30. Similarly, Tower 30 may also include electronic subsystems for receiving and processing signals received from deployed electromagnetic (EM) sensors. Tower 30 may also be used to house and position an EM field generator for detection by EM sensors within or on a medical device.

[0032] Tower 30 may also include console 31, in addition to other consoles available in the rest of the system, such as a console mounted on top of a cart. Console 31 may include a user interface and display screen, such as a touchscreen, for the physician operator. Consoles within System 10 are generally designed to provide both pre-operative and real-time information of the procedure, as well as robot control, and navigation and positioning information for the endoscope 13. When Console 31 is not the only console available to the physician, it may be used by a second operator, such as a nurse, to monitor the patient's health or life and the operation of the system, and to provide procedure-specific data, such as navigation and positioning information.

[0033] The tower 30 may be coupled to the cart 11 and endoscope 13 via one or more cables or connections (not shown). In some embodiments, support functions from the tower 30 are provided to the cart 11 through a single cable, which can simplify and organize the operating room. In other embodiments, specific functions may be coupled with separate cabling and connections. For example, power may be supplied to the cart through a single power cable, while support for control, optics, fluid mechanics, and / or navigation may be provided through separate cables.

[0034] Figure 2 provides a detailed diagram of an embodiment of a cart from the cart-based robot-controllable system shown in Figure 1. The cart 11 generally includes an elongated support structure 14 (often referred to as the “column”), a cart base 15, and a console 16 located at the top of the column 14. The column 14 may include one or more carriages, such as carriages 17 (alternatively referred to as “arm supports”), for supporting the deployment of one or more robot arms 12 (three are shown in Figure 2). The carriage 17 may include individually configurable arm mounts that rotate along a vertical axis to adjust the base of the robot arm 12 for better positioning relative to the patient. The carriage 17 also includes a carriage interface 19 that allows the carriage 17 to translate vertically along the column 14.

[0035] The carriage interface 19 is connected to the column 14 through slots such as slots 20 located on both sides of the column 14 to guide the vertical translation of the carriage 17. The slots 20 include a vertical translation interface for positioning and holding the carriage at various vertical heights relative to the cart base 15. The vertical translation of the carriage 17 allows the cart 11 to adjust the reach of the robotic arm 12 to meet various table heights, patient sizes, and physician preferences. Similarly, individually configurable arm mounts on the carriage 17 allow the robotic arm base 21 of the robotic arm 12 to be angled in various configurations.

[0036] In some embodiments, the slot 20 may be further fitted with a slot cover that is coplanar and parallel to the slot surface to prevent contamination and fluid from entering the internal chamber of the column 14 and the vertical translation interface as the carriage 17 translates vertically. The slot cover may be deployed through a pair of spring spools located near the vertical top and bottom of the slot 20. The cover extends as the carriage 17 translates vertically up and down, and is coiled within the spools until it retracts from its coiled state. The spring load of the spools provides a force to retract the cover into the spool as the carriage 17 translates toward the spool, while also maintaining a seal when the carriage 17 translates away from the spool. The cover may be connected to the carriage 17 using, for example, a bracket in the carriage interface 19 to ensure proper extension and retraction of the cover as the carriage 17 translates.

[0037] Column 14 may internally include mechanisms such as gears and motors designed to use vertically aligned main screws to mechanically translate the carriage 17 in response to control signals generated in response to user input, such as input from a console 16.

[0038] The robotic arm 12 may include a robotic arm base 21 and end effectors 22 separated by a series of couplings 23 connected by a series of joints 24, each joint including an independent actuator, and each actuator including an independently controllable motor. Each independently controllable joint represents an independent degree of freedom available to the robotic arm. Each arm 12 has seven joints and therefore provides seven degrees of freedom. The number of joints results in a number of degrees of freedom, enabling "redundant" degrees of freedom. Redundant degrees of freedom allow the robotic arm 12 to position their respective end effectors 22 in a specific position, orientation, and trajectory in space using different coupling positions and joint angles. This allows the system to position and direct medical instruments from a desired point in space, while enabling the physician to move the arm joint to a clinically advantageous position away from the patient, creating greater access while avoiding arm collisions.

[0039] The cart base 15 balances the weight of the column 14, carriage 17, and arm 12 on the floor. Therefore, the cart base 15 accommodates heavier components such as electronics, motors, power supplies, and components that enable either movement and / or fixing of the cart. For example, the cart base 15 includes casters 25 with rotatable wheels, allowing the cart to move easily around the room before treatment. After reaching a suitable position, the casters 25 may be locked using wheel locks to hold the cart 11 in place during treatment.

[0040] Positioned at the vertical end of column 14, console 16 provides physician users with both preoperative and intraoperative data, enabling both a user interface and a display screen (or a dual-purpose device such as a touchscreen 26) for receiving user input. Potential preoperative data on touchscreen 26 may include preoperative planning, navigation and mapping data derived from preoperative computed tomography (CT) scans, and / or notes from preoperative patient interviews. Intraoperative data on the display may include optical information provided by tools, sensor and coordinate information from sensors, and vital patient statistics such as respiration, heart rate, and / or pulse. Console 16 may be positioned and tilted to allow physicians to access the console from the side of column 14 opposite the carriage 17. From this position, the physician can view console 16, robotic arm 12, and patient while operating console 16 from behind cart 11. As shown in the figure, console 16 also includes a handle 27 to assist in operating and stabilizing cart 11.

[0041] Figure 3 shows an embodiment of a robot-controllable system 10 configured for ureteroscopy. In a ureteroscopy procedure, a cart 11 may be positioned to deliver a ureteroscope 32, a procedure-specific endoscope designed to traverse the patient's urethra and ureters, to the patient's lower abdominal region. In ureteroscopy, it is sometimes desirable for the ureteroscope 32 to be directly aligned with the patient's urethra to reduce friction and force on sensitive anatomical structures in the region. As shown, the cart 11 may be aligned with the legs of a table to allow a robotic arm 12 to position the ureteroscope 32 for direct linear access to the patient's urethra. From the legs of the table, the robotic arm 12 may insert the ureteroscope 32 along a virtual rail 33 directly into the patient's lower abdomen through the urethra.

[0042] Using a similar control technique as in bronchoscopy, after insertion into the urethra, the ureteroscope 32 may be navigated to the bladder, ureters, and / or kidneys for diagnostic and / or therapeutic purposes. For example, the ureteroscope 32 can be directed to the ureters and kidneys, and a growing kidney stone can be broken up using a laser or ultrasonic lithotripter deployed below the working channel of the ureteroscope 32. After the lithotripsy is complete, the resulting stone fragments may be removed using a basket deployed below the ureteroscope 32.

[0043] Figure 4 shows an embodiment of a robot-controllable system similarly positioned for vascular procedures. In a vascular procedure, the system 10 may be configured such that a cart 11 can deliver a medical instrument 34, such as a maneuverable catheter, to an access point in the femoral artery within the patient's leg. The femoral artery offers both a larger diameter for navigation and a relatively indirect and non-tortuotic route to the patient's heart, thereby simplifying navigation. As in a ureteroscopy, the cart 11 may be positioned toward the patient's leg and lower abdomen, allowing a robotic arm 12 to provide a virtual rail 35 with direct linear access to the femoral artery access point in the patient's thigh / lumbar region. After insertion into the artery, the medical instrument 34 may be directed and inserted by translating the instrument driver 28. Alternatively, the cart may be positioned around the patient's upper abdomen to reach alternative vascular access points, such as the carotid and brachial arteries near the shoulder and wrist.

[0044] B. Robot System - Table Embodiments of a robot-controllable medical system may also incorporate a patient table. Incorporating a table reduces the amount of capital equipment in the operating room by eliminating carts and allows for greater access to the patient. Figure 5 shows one embodiment of such a robot-controllable system arranged for a bronchoscopy procedure. The system 36 includes a support structure or column 37 for supporting a platform 38 (illustrated as “table” or “bed”) on the floor. Similar to cart-based systems, the end effector of the robotic arm 39 of the system 36 includes an instrument driver 42 designed to manipulate elongated medical instruments, such as a bronchoscope 40 in Figure 5, through or along a virtual rail 41 formed from a linear alignment of the instrument driver 42. In practice, a C-arm for providing fluoroscopic imaging may be positioned above the patient’s upper abdominal region by placing the radiator and detector around the table 38.

[0045] Figure 6 provides an alternative diagram of the system 36 without a patient and medical equipment for illustrative purposes. As shown, the column 37 may include one or more carriages 43, illustrated as a ring shape, within the system 36 on which one or more robotic arms 39 may be based. The carriages 43 may translate along a vertical column interface 44 along the length of the column 37 to provide different bandage points where the robotic arms 39 can be positioned to reach the patient. The carriages 43 may rotate around the column 37 using mechanical motors located within the column 37, allowing the robotic arms 39 to have access to multiple sides of the table 38, such as both sides of the patient. In embodiments with multiple carriages, the carriages may be individually positioned on the column and may translate and / or rotate independently of each other. The carriages 43 do not need to surround the column 37, or even be circular, but the illustrated ring shape facilitates the rotation of the carriages 43 around the column 37 while maintaining structural balance. The rotation and translation of carriage 43 allows the system to align medical instruments such as endoscopes and laparoscopes to different access points on the patient.

[0046] The arm 39 may be mounted on the carriage via a set of arm mounts 45, which include a series of joints that can individually rotate and / or extend in a nesting manner to provide additional configurability for the robot arm 39. In addition, the arm mounts 45 may be positioned on the carriage 43 such that, when the carriage 43 is rotated appropriately, the arm mounts 45 can be positioned on one side of the table 38 (as shown in Figure 6), on both sides of the table 38 (as shown in Figure 9), or on adjacent sides of the table 38 (not shown).

[0047] Column 37 structurally provides support for the table 38 and a path for the vertical translation of the carriage. Internally, column 37 may be equipped with a main screw for guiding the vertical translation of the carriage and a motor for mechanizing the translation of the carriage based on the main screw. Column 37 may also transmit power and control signals to the carriage 43 and the robotic arm 39 mounted thereon.

[0048] The table base 46 performs a similar function to the cart base 15 of the cart 11 shown in Figure 2, and accommodates heavier components to balance the table / bed 38, column 37, carriage 43, and robot arm 39. The table base 46 may also incorporate rigid casters to provide stability during treatment. Casters extending from the bottom of the table base 46 extend in opposite directions on both sides of the base 46 and may be retracted when it is necessary to move the system 36.

[0049] Continuing with Figure 6, the system 36 may also include a tower (not shown) that divides the functions of the system 36 between the table and the tower, thereby reducing the form factor and bulk of the table. Similar to the embodiments disclosed previously, the tower may provide the table with various support functions such as processing, computing, and control capabilities, power, fluid optics, and / or optical and sensor processing. The tower may also be movable so as to be positioned away from the patient in order to improve physician access and keep the operating room tidy. In addition, placing components within the tower allows for more storage space within the table base for potential storage of robotic arms. The tower may also include a console that provides both a user interface for user input such as a keyboard and / or pendant, and a display screen (or touchscreen) for preoperative and intraoperative information such as real-time imaging, navigation, and tracking information.

[0050] In some embodiments, the table base may house and store a robot arm when not in use. Figure 7 shows a system 47 housing a robot arm in one embodiment of a table-based system. In system 47, the carriage 48 may be translated perpendicular to the base 49, which houses the robot arm 50, arm mount 51, and carriage 48 within the base 49. The base cover 52 may be translated and retracted so as to open to allow the carriage 48, arm mount 51, and arm 50 to unfold around the column 53, and to close to house them for protection when not in use. The base cover 52 may be sealed with a membrane 54 along the edge of its opening to prevent the ingress of dirt and fluid when closed.

[0051] Figure 8 shows one embodiment of a robot-controllable table-based system configured for a ureteroscopy procedure. In a ureteroscopy, the table 38 may include a swivel portion 55 for positioning the patient off-angle from the column 37 and the table base 46. The swivel portion 55 may rotate or swivel around a pivot point (e.g., located below the patient's head) to position the bottom of the swivel portion 55 away from the column 37. For example, swivel of the swivel portion 55 allows a C-arm (not shown) to be positioned above the patient's lower abdomen without competing for space with the column (not shown) below the table 38. By rotating a carriage 35 (not shown) around the column 37, a robotic arm 39 may insert a ureteroscope 56 directly into the inguinal region along a virtual rail 57 to reach the urethra. In a ureteroscopy, stirrups 58 may also be fixed to the swivel portion 55 of the table 38 to support the position of the patient's legs during the procedure and to allow clear access to the patient's inguinal region.

[0052] In laparoscopic procedures, minimally invasive instruments (elongated in shape to adapt to the size of one or more incisions) may be inserted into the patient's anatomical structures through one or more small incisions in the patient's abdominal wall. After the patient's abdominal cavity is expanded, the instrument, often called a laparoscope, may be directed to perform surgical tasks such as grasping, cutting, ablation, and suturing. Figure 9 shows an embodiment of a robot-controllable table-based system configured for laparoscopic procedures. As shown in Figure 9, the carriage 43 of the system 36 rotates and is adjusted vertically, and a pair of robotic arms 39 may be positioned on either side of the table 38 so that the laparoscope 59 can be positioned using arm mounts 45 to pass through minimal bilateral incisions of the patient and reach the patient's abdominal cavity.

[0053] To accommodate laparoscopic procedures, the robot-controllable table system may also tilt the platform to a desired angle. Figure 10 shows an embodiment of a robot-controllable medical system having pitch or tilt adjustment. As shown in Figure 10, the system 36 can adapt to the tilt of the table 38 by positioning one side of the table at a greater distance from the floor than the other side. In addition, the arm mount 45 may rotate to match the tilt so that the arm 39 maintains the same planar relationship as the table 38. To adapt to steep angles, the column 37 may also include a nested portion 60 that allows for vertical extension of the column 37 to prevent the table 38 from contacting the floor or colliding with the base 46.

[0054] Figure 11 provides a detailed illustration of the interface between the table 38 and the column 37. The pitch rotation mechanism 61 may be configured to change the pitch angle of the table 38 relative to the column 37 in multiple degrees of freedom. The pitch rotation mechanism 61 may also be enabled by the arrangement of orthogonal axes 1 and 2 at the column-table interface, each axis being actuated by separate motors 3 and 4 in response to an electrical pitch angle command. Rotation along one screw 5 allows for tilt adjustment along one axis 1, and rotation along the other screw 6 allows for tilt adjustment along the other axis 2.

[0055] For example, pitch adjustment is particularly useful when positioning the table in the Trendelenburg position, i.e., when trying to position the patient's lower abdomen higher off the floor than the patient's lower abdomen for lower abdominal surgery. The Trendelenburg position allows the patient's internal organs to slide towards their upper abdomen by gravity, emptying the abdominal cavity for minimally invasive tools and moving on to lower abdominal surgical procedures such as laparoscopic prostatectomy.

[0056] C. Appliance Drivers and Interfaces The end effector of the system's robotic arm includes (i) an instrument driver (alternatively called an “instrument drive mechanism” or “instrument device manipulator”) incorporating electromechanical means for operating a medical instrument, and (ii) a removable or detachable medical instrument which may lack any electromechanical components such as a motor. This dichotomy may be driven by the need to sterilize medical instruments used in medical procedures, as their complex mechanical assemblies and sensitive electronics make it impossible to adequately sterilize expensive capital equipment. Therefore, medical instruments may be designed to be detached, removed, and replaced from the instrument driver (and thus its system) for individual sterilization or disposal by the physician or physician's staff. In contrast, the instrument driver does not need to be replaced or sterilized and may be covered for protection.

[0057] Figure 12 shows an exemplary instrument driver. Positioned at the distal end of a robotic arm, the instrument driver 62 includes one or more drive units 63 arranged with parallel axes to provide controlled torque to a medical instrument via a drive shaft 64. Each drive unit 63 includes an individual drive shaft 64 for interacting with the instrument, a gearhead 65 for converting the rotation of the motor shaft into a desired torque, a motor 66 for generating the drive torque, an encoder 67 for measuring the speed of the motor shaft and providing feedback to the control circuit, and a control circuit 68 for receiving a control signal and operating the drive unit. Each drive unit 63 is independently controlled and motorized, and the instrument driver 62 may provide multiple (four as shown in Figure 12) independent drive outputs to the medical instrument. During operation, the control circuit 68 receives a control signal, transmits a motor signal to the motor 66, compares the motor speed measured by the encoder 67 to a desired speed, modulates the motor signal to generate the desired torque.

[0058] For procedures requiring a sterile environment, the robotic system may incorporate a drive interface, such as a sterile adapter connected to a sterile drape positioned between the instrument driver and the medical instrument. The primary purpose of the sterile adapter is to maintain physical separation and thus maintain sterility between the drive shaft and the drive input while transmitting angular motion from the instrument driver's drive shaft to the instrument's drive input. Accordingly, an exemplary sterile adapter may consist of a series of rotary input and output sections intended to mate with the instrument driver's drive shaft and a drive input section to the instrument. The sterile drape connected to the sterile adapter is made of a thin, flexible material, such as transparent or translucent plastic, and is designed to cover the instrument driver, the robotic arm, and capital equipment such as a cart (in a cart-based system) or a table (in a table-based system). The use of the drape would allow the capital equipment to be positioned close to the patient while still being located in an area that does not require sterilization (i.e., a non-sterile field). On the other side of the sterile drape, the medical instrument may interface with the patient in an area that requires sterilization (i.e., a sterile field).

[0059] D. Medical devices Figure 13 shows an exemplary medical instrument with a paired instrument driver. Like other instruments designed for use with robotic systems, the medical instrument 70 includes an elongated shaft 71 (or elongated body) and an instrument base 72. The instrument base 72, also called an “instrument handle” due to its design intended for manual interaction by a physician, may generally include a rotatable drive input 73, e.g., a receptacle, pulley, or spool, designed to mate with a drive output 74 extending through a drive interface on the instrument driver 75 at the distal end of a robotic arm 76. When physically connected, latched, and / or coupled, the mated drive input 73 of the instrument base 72 may share a rotational axis with the drive output 74 in the instrument driver 75, allowing for the transmission of torque from the drive output 74 to the drive input 73. In some embodiments, the drive output 74 may include a spline designed to mate with a receptacle on the drive input 73.

[0060] The elongated shaft 71 is designed to be delivered through either an anatomical opening or lumen, such as in endoscopy, or a minimally invasive incision, such as in laparoscopy. The elongated shaft 66 may be flexible (e.g., having properties similar to an endoscope) or rigid (e.g., having properties similar to a laparoscope), or may include a customized combination of both flexible and rigid portions. When designed for laparoscopy, the distal end of a rigid elongated shaft may be connected to an end effector including a jointed wrist formed from a clevis having a pivot axis and a surgical tool, such as a gripper or scissors, which can be actuated based on force from a tendon as the drive input rotates in response to torque received from the drive output 74 of the instrument driver 75. When designed for endoscopy, the distal end of a flexible elongated shaft may include a maneuverable or controllable flex portion that can articulate and flex based on torque received from the drive output 74 of the instrument driver 75.

[0061] Torque from the instrument driver 75 is transmitted downwards on the elongated shaft 71 using tendons within the shaft 71. These individual tendons, such as pull wires, may be individually fixed to individual drive inputs 73 within the instrument handle 72. From the handle 72, the tendons are directed downwards into one or more pull lumens within the elongated shaft 71 and fixed to the distal portion of the elongated shaft 71. In laparoscopy, these tendons may be coupled to distally mounted end effectors such as wrists, grippers, or scissors. Under such a configuration, torque applied to the drive inputs 73 transmits tension to the tendons, thereby acting on the end effectors in some way. In laparoscopy, the tendons can rotate the joints around an axis, thereby moving the end effectors in one direction or another. Alternatively, the tendons may be connected to one or more jaws of a gripper at the distal end of the elongated shaft 71, and the tension from the tendons closes the gripper.

[0062] In endoscopy, tendons may be coupled to flexion or articulation sections located along an elongated shaft 71 (e.g., at the distal end) via adhesive, control rings, or other mechanical fixation. When fixedly attached to the distal end of the flexion section, the torque applied to the drive input section 73 is transmitted downwards to the tendon, causing flexion or articulation of the softer flexion section (sometimes called the articulation section or region). Along the non-flexion section, it may be advantageous to equilibrium the radial forces resulting from tension in the pull wire by arranging individual pull lumens spirally or spirally, directing individual tendons along (or inward) the wall of the endoscope shaft. The angles of the spirals and / or spacing between these may be modified or designed for a particular purpose, with narrower spirals resulting in less shaft compression under load, while smaller amounts of spirals result in greater shaft compression under load, but also exhibit flexion limitations. At the other end of the spectrum, the lumen may be directed parallel to the longitudinal axis of the elongated shaft 71, enabling controlled articular movement in the desired flexible or articulated portion.

[0063] In endoscopic procedures, the elongated shaft 71 houses several components that support robotic procedures. The shaft may include a working channel for deploying surgical tools, irrigation, and / or aspiration to the surgical area at the distal end of the shaft 71. The shaft 71 may also house wires and / or optical fibers to transmit signals to and from an optical assembly at its distal tip, which may include an optical camera. The shaft 71 may also house optical fibers for transporting light from a proximal light source, such as a light-emitting diode, to the distal end of the shaft.

[0064] At the distal end of the instrument 70, the distal tip may also include an opening for a working channel for delivering tools to the surgical site for diagnosis and / or treatment, irrigation, and aspiration. The distal tip may also include a port for a camera, such as a fiberscope or digital camera, to capture images of the internal anatomical space. In connection with this, the distal tip may also include a port for a light source to illuminate the anatomical space when using the camera.

[0065] In the example shown in Figure 13, the drive shaft axis, and therefore the drive input axis, is perpendicular to the axis of the elongated shaft. However, this arrangement complicates the rolling capability of the elongated shaft 71. By allowing the elongated shaft 71 to roll along its axis while the drive input 73 remains stationary, undesirable entanglement of tendons occurs as they extend from the drive input 73 and enter the pull lumen within the elongated shaft 71. Such resulting entanglement of tendons can disrupt any control algorithm intended to predict the movement of the flexible elongated shaft during endoscopic procedures.

[0066] Figure 14 shows an alternative design for an instrument driver and instrument in which the axis of the drive unit is parallel to the axis of the instrument's elongated shaft. As shown, the circular instrument driver 80 includes four drive units, each having a drive output unit 81 aligned parallel to the end of a robot arm 82. The drive units and their respective drive output units 81 are housed in a rotary assembly 83 of the instrument driver 80, which is driven by one of the drive units in the assembly 83. In response to the torque provided by the rotary drive unit, the rotary assembly 83 rotates along a circular bearing that connects the rotary assembly 83 to the non-rotating portion 84 of the instrument driver. Power and control signals may be transmitted from the non-rotating portion 84 of the instrument driver 80 to the rotary assembly 83 through electrical contacts, or maintained through rotation by a brushed slip ring connection (not shown). In other embodiments, the rotary assembly 83 may be integrated with the non-rotatable portion 84 and therefore respond to a separate drive unit that is not parallel to the other drive units. The rotation mechanism 83 enables the instrument driver 80 to rotate the drive unit and their respective drive output units 81 as a single unit around the instrument driver shaft 85.

[0067] Similar to the embodiments disclosed previously, the fixture 86 may include an elongated shaft 88 and a fixture base 87 (shown by a transparent outer skin for illustrative purposes) which includes a plurality of drive inputs 89 (such as receptacles, pulleys, and spools) configured to receive drive output units 81 in the fixture driver 80. Unlike the embodiments disclosed previously, the fixture shaft 88 extends from the center of the fixture base 87, which has an axis substantially parallel to the axes of the drive inputs 89, rather than being perpendicular to the design of Figure 13.

[0068] When coupled to the rotary assembly 83 of the instrument driver 80, the medical instrument 86, including the instrument base 87 and instrument shaft 88, rotates in conjunction with the rotary assembly 83 around the instrument driver shaft 85. Since the instrument shaft 88 is centered on the instrument base 87, the instrument shaft 88 is coaxial with the instrument driver shaft 85 when mounted. Therefore, the rotation of the rotary assembly 83 causes the instrument shaft 88 to rotate around its own longitudinal axis. Furthermore, as the instrument base 87 rotates together with the instrument shaft 88, any tendons connected to the drive input 89 within the instrument base 87 do not become entangled during rotation. Thus, the parallelism of the axes of the drive output 81, drive input 89, and instrument shaft 88 allows for shaft rotation without entanglement of any control tendons.

[0069] E. Navigation and Control Conventional endoscopy may involve the use of fluoroscopy (for example, which may be delivered via a C-arm) and other forms of radiation-based imaging modalities to provide intracavitary guidance to the operating physician. In contrast, the robotic systems envisioned by this disclosure can provide non-radiation-based navigation and localization means to reduce the physician's exposure to radiation and reduce the amount of equipment in the operating room. As used herein, the term “localization” may mean determining and / or monitoring the position of an object in a reference coordinate system. Techniques such as preoperative mapping, computer vision, real-time EM tracking, and robot command data may be used individually or in combination to achieve a radiation-free operating environment. Where radiation-based imaging modalities are still used, preoperative mapping, computer vision, real-time EM tracking, and robot command data may be used individually or in combination to enhance information that can only be obtained by radiation-based imaging modalities.

[0070] Figure 15 is a block diagram illustrating a positioning system 90 for estimating the position of one or more elements of a robotic system, such as the position of a device, according to an exemplary embodiment. The positioning system 90 may be a set of one or more computer devices configured to execute one or more instructions. The computer devices may be embodied by a processor (or more processors) and computer-readable memory within the one or more components described above. For example, but not limited to, the computer devices may be located in the tower 30 shown in Figure 1, a cart shown in Figures 1-4, a bed shown in Figures 5-10, and so on.

[0071] As shown in Figure 15, the positioning system 90 may include a positioning module 95 that processes input data 91-94 to generate position data 96 for the distal end of a medical device. The position data 96 may be data or logic representing the position and / or orientation of the distal end of the device relative to a reference system. The reference system may be the anatomical structure of a patient or a reference system to a known object such as an EM field generator (see the following description of an EM field generator).

[0072] Here, various input data 91-94 are described in more detail. Preoperative mapping can be achieved through the use of low-dose CT scan acquisition. Preoperative CT scans are reconstructed into three-dimensional images that are visualized, for example, as "slices" of notched diagrams of the patient's internal anatomical structures. When analyzed together, image-based models of anatomical cavities, spaces, and structures of the patient's anatomical structures, such as the patient's lung network, can be generated. Techniques such as central line geometry can be determined and approximated from the CT images to generate three-dimensional volumes of the patient's anatomical structures, called preoperative model data 91. The use of central line geometry is described in U.S. Patent Application No. 14 / 523,760, the contents of which are incorporated in their entirety herein. Network topological models may also be derived from CT images and are particularly suitable for bronchoscopy.

[0073] In some embodiments, the instrument may be equipped with a camera to provide vision data 92. A localization module 95 may process the vision data to enable one or more vision-based position tracking. For example, preoperative model data may be used with the vision data 92 to enable computer vision-based tracking of a medical instrument (e.g., an endoscope or instrument advancing through the working channel of the endoscope). For example, using preoperative model data 91, a robotic system may generate a library of endoscopic images predicted from the model based on the expected movement path of the endoscope, with each image linked to a position in the model. Operationally, this library may be referenced by the robotic system to assist in localization by comparing real-time images captured by a camera (e.g., a camera at the distal end of the endoscope) with those in the image library.

[0074] Other computer vision-based tracking techniques use feature tracking to determine camera movement and therefore endoscope. Some features of the localization module 95 may identify circular geometric shapes within preoperative model data 91 corresponding to anatomical lumens, and changes in those geometric shapes may be tracked to determine which anatomical lumen was selected and the relative rotation and / or translational motion of the camera. The use of topological maps may further enhance vision-based algorithms or techniques.

[0075] Optical flow, another computer vision-based technique, may analyze the displacement and translation of image pixels in a video sequence within vision data 92 to infer camera motion. Examples of optical flow techniques include motion detection, object segmentation calculation, luminance, motion compensation coding, and stereoscopic parallax measurement. By comparing multiple frames across multiple iterations, the movement and position of the camera (and therefore the endoscope) can be determined.

[0076] The positioning module 95 can use real-time EM tracking to generate the real-time position of the endoscope within a global coordinate system that can be aligned to the patient's anatomical structure represented by a preoperative model. In EM tracking, EM sensors (or trackers) containing one or more sensor coils embedded in one or more locations and orientations within a medical instrument (e.g., an endoscopic instrument) measure fluctuations in the EM field generated by one or more static EM field generators placed at known locations. The position information detected by the EM sensors is stored as EM data 93. The EM field generators (or transmitters) may be placed close to the patient to generate a low-intensity magnetic field that the embedded sensors can detect. The magnetic field induces a small current in the sensor coils of the EM sensors, which can be analyzed to determine the distance and angle between the EM sensors and the EM field generators. These distances and orientations can be intraoperatively "registered" with the patient's anatomical structure (e.g., preoperative model) to determine the geometric transformation that aligns the position in the preoperative model of the patient's anatomical structure with a single position in the coordinate system. Once registered, EM trackers embedded in one or more locations on a medical device (e.g., the distal tip of an endoscope) can provide a real-time display of the medical device's progression through the patient's anatomical structure.

[0077] Robot command and kinematic data 94 may also be used by a positioning module 95 to provide positioning data 96 for the robotic system. Device pitch and yaw resulting from joint motion commands can be determined during preoperative calibration. Intraoperatively, these calibration measurements can be used in combination with known insertion depth information to estimate the position of the instrument. Alternatively, these calculations can be analyzed in combination with EM, vision, and / or topological modeling to estimate the position of medical instruments in the network.

[0078] As shown in Figure 15, a number of other input data may be used by the positioning module 95. For example, although not shown in Figure 15, a device utilizing shape-sensing fibers may provide shape data that the positioning module 95 can use to determine the position and shape of the device.

[0079] The positioning module 95 may use a combination of input data 91-94(or more). In some cases, such a combination may use a probabilistic approach in which the positioning module 95 assigns confidence weights to the positions determined from each of the input data 91-94. Therefore, if the EM data is not reliable (for example, if EM interference is present), the reliability of the position determined by the EM data 93 may be reduced, and the positioning module 95 may rely more heavily on the vision data 92 and / or the robot command and kinematics data 94.

[0080] As described above, the robotic systems discussed herein may be designed to incorporate one or more combinations of the above technologies. Based on towers, beds, and / or carts, the computer-based control system of the robotic system may store computer program instructions in non-temporary computer-readable storage media such as persistent magnetic memory drives, solid-state drives, etc., which, when executed, cause the system to receive and analyze sensor data and user commands, generate control signals for the entire system, and display navigation and localization data such as the position of instruments in a global coordinate system and anatomical maps.

[0081] 2. Mapping and navigation of internal body regions Embodiments of this disclosure relate to systems and techniques for mapping and / or navigating internal regions of the body using robotically controllable medical instruments. As described in more detail below, the systems and techniques may be configured to generate visual markers indicating the instrument's previous position as the instrument is being navigated within an internal region. The visual markers may be superimposed on a reference image of the internal region and displayed to the user. The visual markers may be derived from positional information received from a position sensor. The position sensor may be positioned on the instrument. The visual markers may form a map representing the instrument's previous position. The map may be used to visualize the anatomical structure of the internal region and may be used by a physician to navigate the instrument within the internal region. For example, a physician may use the map created by the visual markers to navigate to previously visited parts of the internal region or to determine when the instrument is navigating to a new (i.e., not previously visited) part of the internal region.

[0082] The method and technique may also be configured to receive image data from an image sensor positioned on the instrument. The image data may include still images or videos. The method and technique may link the image data with location data so that images captured at a particular location can be played back and displayed to the user. In some embodiments, the images are played back and displayed when the user selects a location or when the instrument is positioned at a location where the linked image exists.

[0110] Furthermore, the method and technique may be configured to allow a physician to tag specific subject features or locations.

[0083] Systems and techniques for mapping and / or navigating internal areas of the body using robot-controllable medical instruments can be used during many medical procedures, such as endoscopic or laparoscopic procedures. These systems and techniques can improve upon the prior art by reducing or eliminating one or more complications or challenges associated with the prior art, as described below. In some embodiments, the techniques described herein can be used in manual medical procedures.

[0084] In many of the examples described herein, systems and techniques for mapping and / or navigating internal regions of the body are described with reference to ureteroscopy procedures for removing kidney stones. However, it will be understood that these systems and techniques may be used in other medical procedures in which medical instruments are navigated within internal regions of a patient's body, such as bronchoscopy and gastroscopy.

[0085] A. Introduction At the beginning and end of ureteroscopy or percutaneous nephrolith removal procedures, physicians often perform a mapping procedure. This may involve navigating a flexible endoscope, such as a ureteroscope or cystoscope, to the first (cranial) cup, visually observing the stone fragments using an imaging device on the endoscope, and then sequentially moving to the next cup until the lowest cup is reached. Physicians use this systematic approach to ensure that all areas of the kidney are observed, despite the wide variation in kidney morphology.

[0086] Several challenges may exist in this mapping procedure. Firstly, such mapping can be time-intensive. For example, each time a fragment is found and processed, the physician must repeat the mapping process and start again from the first cup. Secondly, such mapping can be radiation-intensive. For example, if the endoscope is positioned inside the kidney based solely on the endoscopic view, it is often unclear. Therefore, physicians often take fluoroscopic images, and sometimes selective renal pelvis images, for all cups to confirm the position of the endoscope within the kidney. Thirdly, errors can occur in such mapping. The residual fragment rate suggests that physicians often miss cups during this mapping procedure.

[0087] Some embodiments discussed herein may relate to systems and techniques for mapping and / or navigating internal areas of the body (e.g., kidneys, etc.) using robotically controllable medical instruments (e.g., endoscopes, ureteroscopes, cystoscopes, etc.) that can improve upon the mapping techniques described above and / or reduce or eliminate one or more of the associated challenges. For example, in some embodiments, the mapping and / or navigation systems and techniques discussed herein can increase the physician's confidence that the instrument has navigated through all cups of the kidney while reducing the amount of radiation and the procedure time required for mapping.

[0088] In some embodiments, systems and technologies for mapping and / or navigation may provide physicians with continuous tracking of instrument locations within a map-like context of anatomical structures. Furthermore, in some embodiments, images acquired using an imaging device on an endoscope, and tagging them with the target location, further enhances the ability to inform physicians of the instrument's current location within an anatomical structure.

[0089] For example, a flexible ureteroscope or flexible cystoscope including an EM sensor (or other position sensor) incorporated at its distal tip (or other location on the scope) may be inserted through the working channel of the endoscope used to navigate the kidney. The EM sensor may provide positional data indicating the position of the instrument (e.g., position and / or orientation). As will be described in more detail below, in addition to or instead of the EM sensor, other sensors and methods may be used to determine the position of the instrument (e.g., shape-sensing fiber, impedance tracking, or other forms of localization). Visual markers derived from the positional information may be displayed to the physician. In some embodiments, the visual markers are superimposed on reference images of anatomical structures (e.g., preoperative CT images or intraoperative or preoperative retrograde pelvic images). The positional data may be aligned using, for example, one-point, two-point, or three-point alignment to align the coordinate system of the positional data with the coordinate system of the reference image.

[0090] Alignment of the coordinate system of positional data with the coordinate system of a reference image can be achieved in various ways. For example, a scope may be navigated to one or more features within an anatomical structure that can be recognized both from the scope's viewpoint (e.g., in an image captured by an imaging device on the scope) and / or in a reference image. Positional data from the scope when positioned at the recognized anatomical feature can be used to align the coordinate system of the positional data with the coordinate system of the reference image. In some embodiments, such as the kidney, the scope may be navigated to one or more infundibular segments within the kidney using an imaging device on the scope. Positional data from an EM sensor regarding the location of the infundibular segments can be aligned with the location of the infundibular segments in the reference image to achieve alignment. Other methods for image-based alignment are also possible (i.e., correlating a recognized position in an image captured by an imaging device on the scope with a position in a reference image). As another example of alignment, a reference image of the anatomical structure or another image (e.g., a CT image) may be captured while the scope is positioned within the anatomical structure so that the scope itself is visible in the image. The position of the scope, which is determined via positional data (e.g., from EM data), and the position of the scope within the image can be used to align the coordinate frames of both the positional data and the reference image. Other alignment methods are also possible.

[0091] In some embodiments, after alignment, the position of the instrument may be displayed on a reference image as the instrument moves within the internal region. In some embodiments, the visual markers are displayed as a breadcrumb trail (e.g., data points, traces, or any other preferred visual indicator) showing the instrument's historical (e.g., previous) positions. The displayed visual markers can help the physician confirm that all cups of the kidney have been navigated using the instrument. In some embodiments, as the physician navigates the instrument through the kidney, the displayed visual markers form a map representing the anatomical structure of the kidney. The physician can use this map to revisit previously visited locations within the kidney and / or determine whether the instrument is currently positioned in a new (i.e., not previously visited) cup.

[0092] In some embodiments, the visual markers may include a three-dimensional mesh structure representing the anatomical structure of the kidney. In this way, as the physician navigates the instrument through the kidney, the visual markers can construct a three-dimensional model of the previously visited anatomical structures of the kidney. In some embodiments, the mesh is derived from positional information received from an EM sensor, and optionally from image data received from an imaging device on the instrument.

[0093] Systems and techniques for mapping and / or navigation described herein may enable tagging of locations and / or features of a target while navigating an instrument through an anatomical structure. For example, a physician may "tag" points of a target, such as "cup 1," "supraptic pole," or "large stone," while navigating. In some embodiments, once a point is tagged, its location is automatically saved along with the location where it was captured (for example, as determined by an EM sensor). The tagged points may be displayed on a reference image, and the system may enable the user to add supplementary data, such as labels or written descriptions. In some embodiments, during subsequent navigation (either during the same procedure or a subsequent procedure), the systems and techniques for mapping and / or navigation described herein may determine when the instrument is positioned near a previously tagged point and allow the user to retrieve a saved image for reference or to replay the tagged point based on user selection.

[0094] In addition to the ability to create a three-dimensional map of the kidneys, systems and techniques for mapping and / or navigation can be used to correlate recorded images or videos (e.g., taken with an imaging device on an instrument) with their anatomical positions and orientations within the kidneys. This allows physicians to understand which kidneys they are viewing and / or to replay images from specific locations.

[0095] By overlaying visual markers derived from positional information received from EM sensors onto reference images of anatomical structures, physicians can contextualize the data. This can enable physicians to visualize the location of anatomical structures and instruments within them. This can further enable physicians to identify any areas of the kidney that may have been missed. Tagging can further enhance physicians' ability to understand visual markers, and by replaying images of tagged locations, physicians can visually confirm that the tagged locations are the same as those previously observed. These systems and technologies can reduce procedure time, radiation exposure, and / or the error rate of missed cups.

[0096] Currently, conventional ureteroscopy relies on multiple fluoroscopic images to confirm that the ureteroscope has reached each cup. Typically, the kidney is not filled with contrast during the procedure, which means there are very few anatomical landmarks to confirm that the correct cup has been accessed.

[0097] For example, Figures 16A–16C show three exemplary fluoroscopic images taken during a ureteroscopy procedure. Figure 16A is a first exemplary fluoroscopic image 101a showing an instrument navigating within the kidney 103. As shown in image 101a, the outline of the kidney 103 is visible, but specific internal structures of the kidney 103 are not immediately apparent. The instrument 105 navigating within the kidney 103 is visible in a first position. The first position may be in the first cup of the kidney 103. A physician may take image 101a to confirm that the instrument 105 is positioned in the first cup. Figure 16B is a second exemplary fluoroscopic image 101b showing the instrument 105 in a second position within the kidney 103. The second position may be in the second cup of the kidney 103. A physician may take image 101b to confirm that the instrument 105 is positioned in the second cup. Figure 16C is a third exemplary fluoroscopic image 101c showing the instrument 105 in a third location within the kidney 103. The third location may be a third cup. The physician can take image 101c to confirm that the instrument 105 is positioned within the third cup. As shown in Figures 16A–16C, during conventional ureteroscopy, the physician must take multiple fluoroscopic images at different stages of navigation to determine the position of the instrument within the kidney 103. As mentioned above, this can be disadvantageous as it requires multiple exposures to radiation, is time-consuming, and can be prone to errors.

[0098] Figure 17 shows an exemplary representation of a visual marker 110 that shows, represents, or is derived from position data received from a position sensor of an instrument navigating within the kidney. In the illustrated example, the visual marker 110 of the position data is displayed as a trace. In some embodiments, the trace may include a series of points connected by lines. The trace may indicate the path the instrument travels when navigating within the kidney. As shown, the path may form a map that can represent the anatomical structure of the internal region. The visual marker 110 and / or position data may represent a continuous or nearly continuous position tracking of the instrument tip as it moves through the kidney. The visual marker 110 can create a useful image or map, particularly when superimposed on a reference image of the internal region.

[0099] Figure 18 shows a representation of the visual markers 110 from Figure 17 superimposed on a reference image 115 of the kidney. In this context, the visual markers 110 provide a map of anatomical structures that can be used to aid navigation. Furthermore, physicians can tag specific locations of interest. The tagged information can also be superimposed on the reference image. For example, in the example in Figure 18, seven cups of the kidney are tagged.

[0100] In some embodiments, visual markers 110 representing the anatomical structure of the internal region may be derived. For example, as shown in Figure 18, the contour 117 of the anatomical structure may be derived from positional data and / or the visual markers 110 and superimposed on a reference image. In the illustrated embodiments, the contour 117 is shown as a two-dimensional representation of the anatomical structure, but in some embodiments, the contour 117 may include a three-dimensional mesh representing the anatomical structure. Thus, the contour 117 may provide a model of the internal region that can assist the physician's navigation during the procedure.

[0101] Next, these and other features and advantages of methods and systems for mapping and navigating internal regions of the body will be described in more detail with reference to several specific exemplary methods and systems. The methods and systems described are provided merely as examples and are intended to illustrate the principles of this disclosure. This disclosure should not be limited to the examples described.

[0102] B. Exemplary methods and systems for mapping and navigation of internal body regions. Figure 20A is a flowchart illustrating an exemplary method 200 for mapping internal body regions. Method 200 may be implemented in a specific robotic system, such as the robotic systems shown in Figures 1-15, Figure 22, and others. In some embodiments, one or more computer devices may be configured to perform Method 200. The computer device may be embodied by a processor (or more processors) and computer-readable memory in one or more of the above-mentioned or later components. The computer-readable memory may store instructions that can be executed by the processor(s) to perform Method 200. The instructions may include one or more software modules. For example, but not limited to, the computer device may be located in the tower 30 shown in Figure 1, a cart shown in Figures 1-4, a bed shown in Figures 5-10, and so on.

[0103] Method 200 may be performed, for example, when a medical instrument is being navigated within an internal region of the body during a wide variety of medical procedures (e.g., endoscopic and / or laparoscopic procedures). The internal region may be, for example, an organ, lumen, network of tubules, and / or cavities. In some embodiments, Method 200 may be initiated when the instrument is introduced into the internal region. Method 200 may be triggered automatically (e.g., during initialization or upon detection of an event) or manually (e.g., upon receipt of user input or a command).

[0104] Method 200 begins with block 201, in which a reference image of an internal region of the body is displayed. The reference image may represent or illustrate an internal region of the body. In some embodiments, the reference image may be one or more of the following: for example, an image captured during a retrograde pyelonephrography procedure, a fluoroscopic image, an ultrasound image, an image captured during a computed tomography (CT) procedure, and an image captured during a magnetic resonance imaging (MRI) procedure, or any other type of medical image.

[0105] In some embodiments, reference images may be captured during surgery. For example, reference images may be captured during a medical procedure in which method 200 is implemented. In such cases, method 200 may include the step of capturing reference images. In some embodiments, reference images may be captured before surgery. For example, images of internal regions taken before the procedure (e.g., during a previous procedure or a previous visit to a physician) may be used as reference images.

[0106] Reference images may be displayed to the physician performing the medical procedure. As mentioned above, reference images can provide some indication of the anatomical structure of the internal region. However, in some embodiments, reference images alone may not provide the physician with enough detail to fully understand the anatomical structure. For example, as shown in the exemplary fluoroscopic images in Figures 16A–16C, fluoroscopic images may only provide an outline of the internal region. In Figures 16A–16C, the general shape of the kidney 103 can be seen, but the specific internal anatomical structures and layout of the kidney (e.g., poles, cups, etc.) are not fully visible.

[0107] As will be discussed in more detail below, the reference image can provide a context on which visual markers derived from positional information received from position sensors on the device can be superimposed.

[0108] Method 200 can reduce or completely eliminate the amount of radiation exposure during the procedure. For example, in some embodiments, the patient is exposed to radiation only once during the procedure when a reference image is captured. In contrast, during conventional ureteroscopy, the physician typically takes multiple fluoroscopic images (exposing the patient to multiple doses of radiation) throughout the procedure described above. Figures 16A–16C show exemplary fluoroscopic images, all of which can be captured during a single ureteroscopy. In conventional ureteroscopy, the physician may rely on consecutive fluoroscopic images to determine that each cup has been visited. However, in some embodiments of Method 200, only a single reference image needs to be captured, thus limiting the amount of radiation the patient is exposed to. Furthermore, in some embodiments of Method 200, radiation exposure during the procedure can be completely eliminated, for example, by using a reference image previously captured during a previous procedure or physician's visit.

[0109] In block 203, method 200 involves moving (e.g., navigating) an instrument within an internal region of the body. The instrument may be any other medical instrument described herein, such as endoscope 13 (Figure 1), ureteroscope 32 (Figure 3), instrument 34 (Figure 4), ureteroscope 56 (Figure 8), laparoscope 59 (Figure 9), instrument 70 (Figure 13), or instrument 86 (Figure 14) described above, or instrument 401 (Figure 22) described later. In some embodiments, the instrument may be a robot-controllable instrument controlled by a robot-controllable medical system described throughout this disclosure. In some embodiments, the instrument may be a manually controlled instrument.

[0110] The device may include at least one position sensor. The position sensor may be configured to provide positional information regarding the position of the device (e.g., position and / or orientation). The position sensor may be, for example, an EM sensor as described above. The EM sensor may be configured to provide positional information regarding the position of an EM sensor (e.g., position and / or orientation) within an EM field generated by an EM field generator. In some embodiments, the position sensor may be an EM field generator positioned on the device. The position of the EM field generator may be determined relative to a plurality of EM sensors positioned outside the patient to determine the position of the device. In some embodiments, other types of position sensors may be used. For example, the position sensor may be a shape-sensing fiber (e.g., an optical fiber shape sensor), an impedance tracker, an accelerometer, a gyroscope, an ultrasonic sensor, or any other type of sensor for determining the position of the device.

[0111] One or more of the position sensors may be positioned on the instrument. For example, one or more of the position sensors may be positioned on the distal end of the instrument. In some embodiments, one or more of the position sensors are not positioned on the instrument. For example, the position sensor may be a torque sensor on the proximal end of the instrument or on the motor pack of a robotic arm to which the instrument is attached. Another example of a position sensor may include motion data commanded by a robot-controllable medical system, which can be used to model the estimated orientation and position of the instrument, and / or vision data received from an imaging device, on an image, and to analyze this to determine the movement and position of the instrument.

[0112] In some embodiments of Method 200, the physician controls the movement of an instrument within an internal area, for example, by providing commands to a robot-controllable medical system. The physician can control the movement of the instrument and navigate it within the internal area. For example, during a ureteral nephrolith removal procedure, the physician can navigate the instrument to the stone to be removed, or, as previously stated, the physician can navigate the instrument through each cup of the kidney and visually inspect each cup to confirm that the stone and fragments have been removed. In another example, the physician can navigate the instrument to a target area. For example, the physician can navigate the instrument to a region of transitional cell carcinoma to monitor, biopsy, treat, or excise that region. As will be discussed later, the movement or navigation of the instrument may be assisted by visual markers derived from positional information received from position sensors, superimposed on a reference image.

[0113] In block 205, method 200 involves receiving position information from at least one position sensor. As described above, the position information may indicate the position of the fixture (e.g., position and / or orientation). In some embodiments, the position information may include a 3-degree-of-freedom position of the fixture. For example, the position information may include x, y, and z coordinates representing the position of the fixture in a Cartesian coordinate system. Other coordinate systems (e.g., polarity) may also be used. In some embodiments, the position information may include a higher-degree-of-freedom position of the fixture. For example, the position information may include a 5-degree-of-freedom position, which includes, for example, x, y, and z coordinates of position indicating the orientation of the fixture, as well as indices of the pitch and yaw angles of the fixture. A 6-degree-of-freedom position of the fixture may also include roll information of the fixture.

[0114] In some embodiments, the location information includes multiple location datasets. Each location dataset may indicate the location of the device during its movement (e.g., position and / or orientation). In some embodiments, the location datasets are generated sequentially as the device moves through an internal region. Each location dataset may indicate the location of the device at a specific point in time. Thus, the location datasets may include a log of the device's historical locations.

[0115] In block 207, method 200 involves overlaying visual markers derived from location information onto a reference image to characterize the hierarchical location of an instrument. For example, block 207 may involve overlaying visual markers derived from at least a subset of a location dataset onto a reference image to characterize the hierarchical location of an instrument moving within an internal region of the body. The overlaid visual markers may form a map that can assist a physician in navigating the internal region. The visual markers may represent the path the instrument takes as it navigates the internal region. The visual markers overlaid on the reference image may be displayed to the physician via a user display. Thus, the reference image provides a context for understanding and visualizing the displayed visual markers.

[0116] In some embodiments, the visual markers may include points corresponding to at least a subset of the location dataset (e.g., a breadcrumb trail). These points may be plotted on a reference image to display the historical location of the instrument. The points may be represented by a wide variety of visual markers. For example, the visual markers may be represented as any preferred shape or marker, such as a dot, dash, X, or other shape.

[0117] Various criteria can be used to determine when visual markers are superimposed onto a reference image (i.e., how often visual markers are superimposed). These criteria may include, for example, the distance the instrument has moved (e.g., the distance moved from a previous visual marker), the direction of movement, and the time elapsed since the previous superimposed visual marker. Those skilled in the art will understand that these criteria can be modified to determine how often visual markers are generated and superimposed onto the reference image, as well as the distance between consecutive visual markers.

[0118] In some embodiments, a subset of the location dataset is superimposed on a reference image at a duration frequency. The duration frequency may include the time between superimposed visual markers. For example, visual markers may be superimposed approximately every 0.05 seconds, i.e., at time intervals of approximately 0.05 seconds. In other examples, visual markers may be superimposed at time intervals of approximately 0.1, 0.25, or 0.5 seconds, as well as other durations both shorter and longer than those listed. In some embodiments, the duration frequency may vary. For example, there does not need to be a fixed duration between each superimposed visual marker.

[0119] In some embodiments, a subset of the positional dataset is superimposed on a reference image at positional frequencies. Positional frequencies may include the distances between the superimposed visual markers. For example, visual markers may be superimposed at intervals of approximately 0.05 mm, 0.1 mm, 0.25 mm, 0.5 mm, 1.0 mm, or 2.0 mm, as well as other distances both shorter and longer than those listed. In some embodiments, positional frequencies may vary. For example, there does not need to be a fixed distance between each superimposed visual marker.

[0120] In some embodiments, the visual markers are superimposed substantially continuously on a reference image to form a continuous trace representing the historical position of the instrument (see, for example, Figures 17 and 18). In some embodiments, the superimposed visual markers may be connected at a duration frequency or positional frequency (e.g., as separate points) to show the historical path of the instrument's movement. For example, in some embodiments, lines may be used to connect adjacent superimposed visual markers. In other embodiments, curves may be adapted to the superimposed visual markers.

[0121] As a physician navigates an instrument through an internal region and visual markers are superimposed onto a reference image, these markers may begin to form a map. This map may represent the instrument's historical position. The map may also represent the anatomical structure of the internal region. For example, as a physician navigates in and out of each cup, the path formed by visual markers may represent the structure of each cup. The physician can use this map to determine whether the instrument is currently navigating a new area (i.e., a cup not previously visited) or to return to a previously visited cup.

[0122] In some embodiments, the visual markers superimposed on the reference image may include a mesh representing the anatomical structure of the internal region. For example, in some cases, a tubular structure can be fitted around a positional dataset to develop a three-dimensional mesh or model representing the anatomical structure. The diameter or other shape of the tubular structure may be determined based on the movement of the instrument within the internal region. In some embodiments, the diameter or other shape of the tube is determined based on a positional dataset and images received from an imaging device positioned on the instrument. For example, this method can estimate the diameter of a tube at a given position from an image captured at that position. Based on this, a three-dimensional mesh representing the anatomical structure of the internal region can be developed and displayed as a visual marker superimposed on the reference image.

[0123] In some cases, visual markers may change over time. For example, visual markers may fade over time, with more recently moved areas becoming darker than previously moved areas. As another example, visual markers from different areas of an internal region may be represented by different colors (see, for example, Figures 17 and 18), and these different colors may be represented by different types of dashed lines. For example, different colors may be used to indicate paths through each different cup.

[0124] In some embodiments, data can be associated with visual markers. For example, a physician can create notes containing data associated with specific visual markers. Users can enter data via a command console, and the data can be viewed via a display.

[0125] Visual markers superimposed on a reference image may be displayed to the user during surgery (procedure). These visual markers may be stored or saved for use in future medical procedures.

[0126] In some embodiments, method 200 may include one or more additional steps. For example, method 200 may include one or more steps of method 300 (Figure 21A) and / or method 320 (Figure 21B) described below.

[0127] Method 200 may also act to tag a location or feature of a subject. For example, Method 200 may include receiving and displaying a current image from an imaging device positioned on an instrument. A physician viewing the image may determine that the image contains a location or feature of a subject. For example, the physician may determine that the image contains the entrance to the cup, the pole of the kidney, an area of ​​transitional cell carcinoma, a nephrolith, a stone fragment, etc. The physician may tag this location. The physician may input data associated with the tagged location. The tagged location and / or input data may be overlaid on a reference image (see, for example, Figure 18 showing seven tagged kidney cups).

[0128] In some embodiments, Method 200 may act to automatically detect and tag features of an object. For example, images received from an imaging device on an instrument may be processed and analyzed to determine whether they contain any location or feature of the instrument. For example, an automated process may detect stones, stone fragments, or entrances to cups in the image and automatically tag these locations. As previously mentioned, the tagged locations and / or associated data may be overlaid on a reference image.

[0129] In some embodiments, method 200 may act to adjust the position determined by the position sensor to take physiological dynamics into account and to superimpose the adjusted position onto a reference image. For example, a motion sensor may be positioned above the patient. The motion sensor may detect the patient's movement (e.g., breathing) and adjust the position determined by the position sensor to take the patient's movement into account.

[0130] Figure 20B is a flowchart illustrating another exemplary method 220 for mapping internal body regions. Method 220 may be implemented in certain robotic systems, such as the robotic systems shown in Figures 1-15, Figure 22, and others. In some embodiments, one or more computer devices may be configured to perform Method 220. The computer device may be embodied by a processor (or more processors) and computer-readable memory in one or more of the above-mentioned or later components. The computer-readable memory may store instructions that can be executed by the processor(s) to perform Method 220. The instructions may include one or more software modules. For example, but not limited to, the computer device may be in the tower 30 shown in Figure 1, a cart shown in Figures 1-4, a bed shown in Figures 5-10, and so on.

[0131] Method 220 may be performed, for example, when a medical instrument is being navigated within an internal region of the body during a wide variety of medical procedures (e.g., endoscopic and / or laparoscopic procedures). The internal region may be, for example, an organ, lumen, network of tubules, and / or cavities. In some embodiments, Method 220 may be initiated when the instrument is introduced into the internal region. Method 220 may be triggered automatically (e.g., during initialization or upon detection of an event) or manually (e.g., upon receipt of user input or a command).

[0132] Method 220 begins in block 221, where the instrument is moved within the internal region of the body. In block 223, Method 220 involves receiving positional information of the instrument from a position sensor. Blocks 221 and 223 may be similar to blocks 203 and 205 of Method 200 described above. For brevity, the features of blocks 221 and 223 are not described again here.

[0133] In block 225, method 220 involves displaying visual markers derived from location information to characterize the structure of internal body regions. In many respects, block 225 is similar to block 207 of method 200. However, in contrast to block 207 of method 200, block 225 does not necessarily involve superimposing the visual markers onto a reference image. In fact, in some embodiments, method 220 may not involve the use of a reference image at all. Instead, block 225 may involve displaying visual markers derived from location information directly to the user (i.e., without referring to any reference image). An example of such a visual marker is shown in Figure 17, which shows location information represented as visual marker 110. As shown in Figure 17, no reference image is shown, but in some cases, visual marker 110 alone may be sufficient to characterize the structure of internal body regions. For example, by tracing the visual markings 110 in Figure 17, a physician can understand the general anatomical structure of previously navigated areas within the internal region. The visual markings 110 shown in block 225 may be any type of visual marking described herein, including visual markings shown as separate points, visual markings connected by lines or fitted into lines, meshes representing anatomical structures, and so on.

[0134] Method 220 allows for the construction of a model or map of the internal regions of the body as the instrument navigates through them. In some cases, the model or map may be used by the physician to return to previously visited locations or to determine when the instrument is navigating into a new part of the internal region. In some cases, the map or model becomes clearer as the instrument navigates further within the internal region.

[0135] In some embodiments, method 220 may include one or more additional blocks. For example, method 220 may include one or more blocks of method 300 (Figure 21A) and / or method 320 (Figure 21B) described below.

[0136] Figure 21A is a flowchart illustrating an exemplary method 300 for navigation of internal body regions. Method 300 may be implemented in certain robotic systems, such as the robotic systems shown in Figures 1 to 15, Figure 22, and others. In some embodiments, one or more computer devices may be configured to perform Method 300. The computer device may be embodied by a processor (or more processors) and computer-readable memory in one or more of the above-mentioned or later components. The computer-readable memory may store instructions that can be executed by the processor(s) to perform Method 300. The instructions may include one or more software modules. For example, but not limited to, the computer device may be in the tower 30 shown in Figure 1, a cart shown in Figures 1 to 4, a bed shown in Figures 5 to 10, and so on.

[0137] Method 300 may be performed, for example, when a medical instrument is being navigated within an internal region of the body during a wide variety of medical procedures (e.g., endoscopic and / or laparoscopic procedures). The internal region may be, for example, an organ, lumen, network of tubules, and / or cavities. In some embodiments, Method 300 may be initiated when the instrument is introduced into the internal region. Method 300 may be triggered automatically (e.g., during initialization or upon detection of an event) or manually (e.g., upon receipt of user input or a command).

[0138] Method 300 begins in block 301 in which the instrument is moved within an internal region of the body. The instrument may be any other medical instrument described herein, such as, for example, an endoscope 13 (Figure 1), a ureteroscope 32 (Figure 3), an instrument 34 (Figure 4), a ureteroscope 56 (Figure 8), a laparoscope 59 (Figure 9), an instrument 70 (Figure 13), or the instrument 86 (Figure 14) described above, or an instrument 401 (Figure 22) described later. In some embodiments, the instrument may be a robot-controllable instrument controlled by a robot-controllable medical system described throughout this disclosure.

[0139] The device may include position sensors such as EM sensors, EM field generators, shape-sensing fibers (e.g., optical fiber shape sensors), impedance tracking devices, accelerometers, gyroscopes, ultrasonic sensors, or any other type of sensor for determining the position of the device.

[0140] In some embodiments of Method 300, the physician controls the movement of the instrument within the internal area by providing commands to a robot-controllable medical system, for example, as described above. The physician can control the movement of the instrument to navigate it to a desired location within the internal area, as described above.

[0141] In block 303, method 300 involves receiving position information from at least one position sensor of the fixture while the fixture is moving. The position information may indicate the position of the fixture (e.g., position and / or orientation) as described above. In some embodiments, the position information may include a 3-degree-of-freedom position of the fixture. In some embodiments, the position information may include a higher-degree-of-freedom position of the fixture, such as a 5-degree-of-freedom position or a 6-degree-of-freedom position.

[0142] In some embodiments, the location information includes multiple location datasets. Each location dataset may indicate the location of the device during its movement (e.g., position and / or orientation). In some embodiments, the location datasets are generated sequentially as the device moves through an internal region. Each location dataset may indicate the location of the device at a specific point in time. Thus, the location datasets may include a log of the device's historical locations.

[0143] In block 305, method 300 involves receiving image data from an imaging device of an instrument within an internal region while the instrument is moving. The imaging device may be any photosensitive substrate or structure configured to convert the energy representing the received light into an electrical signal, such as a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) image sensor. In some examples, the imaging device may include one or more optical fibers. For example, the imaging device may be a bundle of optical fibers configured to transmit light representing an image from the distal end of the instrument to the image sensor. The images captured by the imaging device may be transmitted to a computer system for storage or display as individual frames or as a series of consecutive frames (e.g., video).

[0144] Image data may include one or more images or videos captured by the imaging device at one or more locations within an internal region. In some embodiments, the imaging device is configured to automatically capture image data. For example, the imaging device may be configured to capture images at a duration frequency. Duration frequency may include the time between captured images. For example, images may be captured at intervals of about 0.05 seconds, about 0.1 seconds, about 0.25 seconds, or about 0.5 seconds, and at other durations both shorter and longer than the examples listed. In some embodiments, the duration frequency may vary. For example, there does not need to be a fixed duration between each captured image. The imaging device may be configured to capture images at a positional frequency. For example, positional frequency may include the distance moved by the instrument between captured images. For example, images may be captured at intervals of about 0.05 mm, about 0.1 mm, about 0.25 mm, about 0.5 mm, about 1.0 mm, or about 2.0 mm, and at other distances both shorter and longer than the examples listed. In some embodiments, the positional frequencies may vary. For example, there does not need to be a fixed distance between each captured image. In some embodiments, the images are captured substantially sequentially.

[0145] In some embodiments, images are captured when a user command is received. For example, a doctor can choose when to capture an image.

[0146] In block 307, method 300 involves linking at least a subset of one or more images to at least a subset of a location dataset based on the location where each image was captured, as determined by a location sensor. For example, with respect to at least a subset of one or more images, location data received at the location where the image was taken may be linked to (e.g., stored) the image. The image may be linked to a 3-degree-of-freedom location (e.g., a point in space). The image may also be linked to orientation information by linking the image to a higher-dimensional degree-of-freedom location, such as the 5 or 6-degree-of-freedom location mentioned above. In block 307, method 300 may create a database of images associated with or linked to the location where the image was captured. As described later, the images may be retrieved from the database based on their associated or linked locations for display to a physician.

[0147] In some embodiments, all captured images are linked to their associated location datasets. In some embodiments, only a subset of one or more images is linked to their associated location datasets. The subsets may be selected by duration frequency (e.g., duration frequencies of approximately 0.1 seconds, 0.25 seconds, 0.5 seconds, or 1.0 second) or location frequency (e.g., location frequencies of approximately 0.1 mm, 0.25 mm, 0.5 mm, or 1.0 mm).

[0148] In block 309, method 300 involves determining user input, including location selection. In some embodiments, determining a user command includes receiving a user command. For example, the user may select a location within an internal region. In some embodiments, the user selects a location from displayed visual markers. In some embodiments, the user selects a location relative to a reference image.

[0149] In block 311, method 300 involves displaying one of the linked images that corresponds to the location selection. For example, the image linked to the selected location may be obtained from the aforementioned database. In some embodiments, if no linked image exists that corresponds to the selected location, method 300 retrieves and displays the linked image that most closely corresponds to the selected location.

[0150] Therefore, method 300 may enable a user to view images corresponding to specific locations within an internal region. This may facilitate navigation within the internal region. This may help a physician find a specific location or feature of interest and return to it. In certain embodiments, a physician may facilitate navigation of the instrument by comparing the current image taken by an imaging device on the instrument with one or more previously taken images linked to locations within the internal region.

[0151] Figure 21B is a flowchart illustrating another exemplary method 320 for navigation of internal body regions. Method 320 may be implemented in certain robotic systems, such as the robotic systems shown in Figures 1-15, Figure 22, and others. In some embodiments, one or more computer devices may be configured to perform Method 320. The computer device may be embodied by a processor (or more processors) and computer-readable memory in one or more of the above-mentioned or later components. The computer-readable memory may store instructions that can be executed by the processor(s) to perform Method 320. The instructions may include one or more software modules. For example, but not limited to, the computer device may be in the tower 30 shown in Figure 1, a cart shown in Figures 1-4, a bed shown in Figures 5-10, and so on.

[0152] Method 320 may be performed, for example, when a medical instrument is being navigated within an internal region of the body during a wide variety of medical procedures (e.g., endoscopic and / or laparoscopic procedures). The internal region may be, for example, an organ, lumen, network of tubules, and / or cavities. In some embodiments, Method 320 may be initiated when the instrument is introduced into the internal region. Method 320 may be triggered automatically (e.g., during initialization or upon detection of an event) or manually (e.g., upon receipt of user input or a command).

[0153] Method 320 begins in block 321 in which the instrument is moved within an internal region of the body. The instrument may be any other medical instrument described herein, such as, for example, an endoscope 13 (Figure 1), a ureteroscope 32 (Figure 3), an instrument 34 (Figure 4), a ureteroscope 56 (Figure 8), a laparoscope 59 (Figure 9), an instrument 70 (Figure 13), or the instrument 86 (Figure 14) described above, or an instrument 401 (Figure 22) described later. In some embodiments, the instrument may be a robot-controllable instrument controlled by a robot-controllable medical system described throughout this disclosure.

[0154] The instrument may include at least one position sensor, as described throughout this disclosure. In some embodiments of Method 300, a physician controls the movement of the instrument within the internal area, for example, by providing commands to a robot-controllable medical system. The physician can control the movement of the instrument to navigate it to a desired location within the internal area.

[0155] In block 323, method 320 involves receiving position information from at least one position sensor of the fixture while the fixture is moving. The position information may indicate the position of the fixture (e.g., position and / or orientation) as described above. In some embodiments, the position information may include a 3-degree-of-freedom position of the fixture. In some embodiments, the position information may include a higher-degree-of-freedom position of the fixture, such as a 5-degree-of-freedom position or a 6-degree-of-freedom position.

[0156] In some embodiments, the location information includes multiple location datasets. Each location dataset may indicate the location of the device during its movement (e.g., position and / or orientation). In some embodiments, the location datasets are generated sequentially as the device moves through an internal region. Each location dataset may indicate the location of the device at a specific point in time. Thus, the location datasets may include a log of the device's historical locations.

[0157] In block 325, method 320 involves receiving image data from an imaging device of the instrument within an internal region while the instrument is moving. The image data may include one or more still images of a video of the internal region. Block 325 may be similar to block 305 of method 300. For brevity, the features of block 325 are not described again here.

[0158] In block 327, method 320 involves linking at least a subset of one or more images to at least a subset of a location dataset based on the location where each image was captured, as determined by a location sensor. For example, with respect to at least a subset of one or more images, location data received at the location where the image was taken may be linked to (e.g., stored) the image. Block 327 may be similar to block 307 of method 300. For brevity, the features of block 327 are not described again here.

[0159] In block 329, method 320 involves determining the current position of the device using at least one position sensor, where the current position corresponds to the current position data set among multiple position data sets. For example, in block 329, method 320 may determine the current position of the device within an internal region.

[0160] In block 331, method 320 involves displaying one of the linked images corresponding to the determined current position. Thus, method 320 involves determining whether any linked image is available for the currently determined position of the instrument and whether it is possible to display the linked image to the physician. This may favorably allow the physician to verify the position of the instrument by comparing the linked image with the current image taken by the imaging device. If the images match, the physician can conclude that the instrument is at a previously navigated position.

[0161] Method 320 may also allow a physician to compare images taken over time. For example, a physician can navigate an instrument to its location (e.g., the area of ​​transitional cell carcinoma) during a subsequent procedure. Method 320 may acquire and display images of the area of ​​the target taken during a previous procedure. A physician can compare the linked images with the current images to determine changes in the area of ​​the target.

[0162] Figure 19 shows an example of the display of a linked image. As shown in Figure 19, the display 120 may provide a representation of an anatomical structure. In some embodiments, the display 120 displays an image of the anatomical structure (e.g., a fluoroscopic image or other medical image). In some embodiments, the display 120 displays a representation of the anatomical structure, such as a computer model. The user can select a position 121, as described above with reference to Figure 21A. In another embodiment, position 121 may represent the current position of an instrument within the anatomical structure, for example, as described above with reference to Figure 21B. If a linked image 123 is available for the selected position, the linked image 123 may also be displayed to the user. In the illustrated embodiment, the linked image 123 shows a kidney stone 125.

[0163] Figure 22 is a block diagram showing specific components of one embodiment of a system 400 for mapping and / or navigating internal regions of the body. In the illustrated embodiment, the system 400 includes an instrument 401. The instrument 401 may include an elongated body configured for navigation of internal regions of the body. In some embodiments, the elongated body may include working channels into which one or more tools can be inserted. The elongated body may be articulated; that is, in some embodiments, the shape or orientation of the elongated body can be controlled. For example, the elongated body may include one or more pull wires or tendons that are operable to adjust the shape or orientation of the elongated body.

[0164] The device 401 also includes a position sensor 402. The position sensor may be configured to provide positional information regarding the position of the device 401 (e.g., position and / or orientation). The position sensor 402 may be, for example, the EM sensor described above. The EM sensor may be configured to provide positional information regarding the position of an EM sensor in an EM field generated by an EM field generator (e.g., position and / or orientation). In some embodiments, the position sensor may be an EM field generator positioned on the device. The position of the EM field generator may be determined relative to a plurality of EM sensors positioned outside the patient to determine the position of the device. In some embodiments, other types of position sensors may be used. For example, the position sensor 402 may be a shape-sensing fiber (e.g., an optical fiber shape sensor), an impedance tracker, an accelerometer, a gyroscope, an ultrasonic sensor, or any other type of sensor for determining the position of the device 401.

[0165] As illustrated, the position sensor 402 may be positioned on the instrument 401. For example, the position sensor 402 may be positioned on the distal end of an elongated body. In some embodiments, the position sensor 402 is not positioned on the instrument 401, such as a torque sensor on the proximal end of the instrument 401, or on the motor pack of the robot arm to which the instrument is mounted. Other examples of position sensors may include motion data commanded by a robot-controllable medical system, which may be used to model the estimated orientation and position of the instrument, and / or vision data received from an imaging device, on an image, and to analyze this to determine the movement and position of the instrument.

[0166] The apparatus 401 also includes an imaging device 403. The imaging device 403 may be any photosensitive substrate or structure configured to convert the energy representing received light into an electrical signal, for example, a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) image sensor. In some examples, the imaging device 403 may include one or more optical fibers. For example, the imaging device 403 may be a bundle of optical fibers configured to transmit light representing an image from the distal end of the apparatus to the image sensor. Images captured by the imaging device 403 may include still images and / or video.

[0167] In the illustrated embodiment of system 400, the instrument 401 is attached to an instrument positioning device 404. The instrument positioning device 404 may be configured to operate the instrument 401. For example, the instrument positioning device 404 may be configured to insert or retract the instrument 401 into or into an internal region of the body and / or to control joint movement (i.e., to adjust the shape or orientation of the instrument 401). For example, in some embodiments, the instrument positioning device 404 includes one or more robotic arms configured to move the instrument 401 forward or backward. The instrument positioning device 404 may also include the instrument device manipulator described above. The instrument device manipulator may be configured to actuate a pull wire or tendon of the instrument 401 to control joint movement of the instrument 401.

[0168] System 400 includes a processor 405 and memory 406. Memory 406 may contain instructions that configure the processor 405 to execute various methods or processes. For example, memory 406 may contain instructions that cause the processor 405 to execute method 200 (Figure 20A), method 220 (Figure 20B), method 300 (Figure 21A), and / or 320 (Figure 22B).

[0169] As shown in the figure, system 400 also includes a medical imaging device 407. The medical imaging device 407 may be configured to capture reference images of internal areas of the body. The medical imaging device 407 may be any type of medical image sensor, including, for example, an X-ray machine, an ultrasound machine, a CT scanner, or an MRI machine. The medical imaging device 407 may be connected to a processor 405 and memory 406 to provide reference images for use as described above.

[0170] System 400 also includes a data store 408. The data store 408 may be memory such as a hard disk drive, solid-state drive, or flash drive for storing information. In some embodiments, the data store 408 may be configured to store location information received from a location sensor 402 and image data received from an imaging device 403. In some embodiments, the data store 408 stores the image data in a manner linked to location data such that the location where each image was captured is linked to the image. The data store 408 may also be configured to store visual markers generated when the processor 405 performs methods 200 and 220.

[0171] System 400 also includes a display 409. The display 409 may include, for example, an electronic monitor (e.g., a liquid crystal display (LCD) display, an LED display, or a touch-sensitive display), a virtual reality viewing device (e.g., goggles or glasses), and / or other display devices.

[0172] The display 409 may be configured to display various information to the physician during the procedure. For example, the display 409 may display a reference image, as well as visual markers superimposed on it. As another example, the display 409 may be configured to display one or more tagged images related to the user's selection of a location.

[0173] 3. System Implementation and Terminology Embodiments disclosed herein provide systems, methods, and apparatus for mapping and / or navigating internal areas of the body using robotically controllable medical devices.

[0174] When used herein, the terms “combine,” “combine,” “combine,” or other variations of word combinations may indicate either indirect or direct connection. For example, when a first component is “combine” with a second component, the first component may be indirectly connected to the second component via another component, or it may be directly connected to the second component.

[0175] The mapping and navigation functions described herein may be stored as one or more instructions on a processor-readable medium or computer-readable medium. The term “computer-readable medium” means any available medium that can be accessed by a computer or processor. Examples, but not limited to, such mediums may include random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory, compact disk read-only memory (CD-ROM), or other optical disk storage devices, magnetic disk storage devices or other magnetic storage devices, or any other medium that can be used to store desired program code in the form of instructions or data structures and can be accessed by a computer. Note that computer-readable medium may be tangible and non-temporary. As used herein, the term “code” may mean software, instructions, code, or data that can be executed by a computing device or processor.

[0176] The methods disclosed herein include one or more steps or actions for achieving the described method. The steps and / or actions of the method may be interchangeable with one another without departing from the scope of the claims. In other words, the order and / or use of any particular steps and / or actions may be modified without departing from the scope of the claims, unless a particular order of steps or actions is required for the proper operation of the described method.

[0177] As used herein, the term "plural" refers to two or more. For example, "plural components" refers to two or more components. The term "determine" encompasses a wide variety of actions and therefore "determine" can include calculating, operating, processing, deriving, investigating, looking up (e.g., looking at a table, database, or another data structure), confirming, etc. Also, "determine" can include (e.g., receiving information), accessing (e.g., accessing data in memory), etc. Also, "determine" can include resolving, selecting, electing, establishing, etc.

[0178] The phrase "based on" does not mean "based solely on" unless explicitly specified otherwise. In other words, the phrase "based on" can mean both "based solely on" and "based at least on."

[0179] The foregoing description of the disclosed embodiments is provided to enable any person skilled in the art to manufacture or use the present invention. Various modifications to these embodiments will be readily apparent to a person skilled in the art, and the general principles set forth herein may be applied to other embodiments without departing from the scope of the invention. For example, a person skilled in the art will understand that many corresponding alternative and equivalent structural details may be used, such as equivalent methods for fastening, mounting, joining, or engaging tool components, equivalent mechanisms for producing specific operating motions, and equivalent mechanisms for delivering electrical energy. Accordingly, the present invention is not intended to be limited to the embodiments shown herein, but rather to be given the broadest scope consistent with the principles and novel features disclosed herein.

[0180] [Implementation Method] (1) A method for mapping internal regions of the body, the method being Displaying a reference image of the internal region of the body, Moving the instrument within the internal region of the body, wherein the instrument includes at least one position sensor, Receiving location information from at least one of the location sensors, wherein the location information includes a plurality of location datasets, each location dataset indicating the position of the device during its movement; A method comprising superimposing visual markers derived from at least a subset of the position dataset onto the reference image to characterize the historical position of the instrument in motion within the internal region of the body. (2) The method according to Embodiment 1, wherein the reference image includes an image captured during retrograde pyelonephrography. (3) The method according to Embodiment 1, wherein the reference image includes an X-ray fluoroscopic image or an ultrasound image. (4) The method according to Embodiment 1, wherein the reference image is captured during computed tomography (CT) or magnetic resonance imaging (MRI) procedure. (5) The method according to Embodiment 1, further comprising capturing the reference image during surgery.

[0181] (6) The method according to Embodiment 1, wherein the reference image is captured before surgery. (7) Receiving equipment image data from an imaging device positioned on the equipment, wherein the equipment image data includes a plurality of images captured by the imaging device during the movement of the equipment, The method according to Embodiment 1, further comprising linking each image in at least a subset of the plurality of images to the location dataset indicating the location where the image was captured. (8) The method of Embodiment 7, further comprising storing the linked images for use during future procedures. (9) Receiving user input for location selection, The method according to embodiment 7, further comprising displaying a linked image corresponding to the user input. (10) Using the position sensor, determine the current position of the device, The method according to embodiment 7, further comprising displaying a linked image corresponding to the determined current position.

[0182] (11) The method according to embodiment 7, wherein the device image data is automatically captured. (12) The method according to embodiment 7, wherein the device image data is captured when a user command is received. (13) The method according to Embodiment 1, further comprising tagging a location or feature of an object within the internal region. (14) The method according to Embodiment 13, wherein tagging includes receiving user input. (15) The method according to Embodiment 13, wherein tagging includes automatically detecting the location or characteristics of the subject.

[0183] (16) The method according to Embodiment 13, further comprising overlaying the tagged location or feature of the subject onto the reference image. (17) The method according to Embodiment 1, further comprising connecting the visual markers to characterize the historical path of the movement of the device. (18) The method according to Embodiment 1, wherein the visual mark includes a mesh. (19) The method according to embodiment 18, wherein the mesh shows the anatomical structure of the internal region. (20) The method according to Embodiment 19, wherein the mesh is derived from the subset of the position dataset and image data received from the imaging device on the instrument.

[0184] (21) The method according to Embodiment 1, wherein the subset of the positional dataset is superimposed on the reference image at a duration frequency. (22) The method according to Embodiment 1, wherein the subset of the positional dataset is superimposed on the reference image at positional frequency. (23) The method according to Embodiment 1, further comprising displaying the visual sign during surgery. (24) The method according to Embodiment 1, further comprising memorizing the visual mark for use during a future medical procedure. (25) The internal region of the body includes the kidney, and the method is Moving the aforementioned instrument into the kidney cup, The method according to Embodiment 1, further comprising tagging at least one of the following: the entrance to the cup of the kidney, the pole of the kidney, a gallstone within the kidney, and a region of transitional cell carcinoma.

[0185] (26) Adjusting the position determined by the position sensor in order to take physiological dynamics into consideration, The method according to Embodiment 1, further comprising superimposing the adjusted position onto the reference image. (27) A non-temporary computer-readable storage medium in which instructions are stored, wherein, when an instruction is executed, the processor of the device provides at least: Moving the device within the internal region of the body, Receiving position information from at least one position sensor of the device during the movement of the device, wherein the position information includes a plurality of position datasets, each of which indicates the position of the device. Displaying an image of the internal region of the body, A non-temporary computer-readable storage medium that allows for the superimposition of visual markers derived from at least a subset of the position dataset onto the image to characterize the historical position of the instrument during its movement within the internal region of the body. (28) The reference image is a non-temporary computer-readable storage medium according to Embodiment 27, which includes an image captured during retrograde pelvic imaging. (29) The reference image is a non-temporary computer-readable storage medium according to Embodiment 27, which includes an X-ray fluoroscopic image or an ultrasound image. (30) The reference image is captured during computed tomography (CT) or magnetic resonance imaging (MRI) procedure on a non-temporary computer-readable storage medium as described in Embodiment 27.

[0186] (31) The non-temporary computer-readable storage medium according to Embodiment 27, wherein, when the instruction is executed, the processor further causes the reference image to be captured during surgery. (32) The reference image is captured before surgery on a non-temporary computer-readable storage medium according to Embodiment 27. (33) When the instruction is executed, the processor will Receiving equipment image data from an imaging device positioned on the equipment, wherein the equipment image data includes a plurality of images captured by the imaging device during the movement of the equipment. A non-temporary computer-readable storage medium according to Embodiment 27, further comprising linking each image of the subset with a location dataset indicating the location where the image was captured, with respect to at least a subset of the plurality of images. (34) The non-temporary computer-readable storage medium according to Embodiment 33, wherein, when the instruction is executed, the processor further causes the processor to store the linked image for use during a future procedure. (35) When the instruction is executed, the processor will Receiving user input for location selection, A non-temporary computer-readable storage medium according to embodiment 33, further comprising displaying a linked image corresponding to the user input.

[0187] (36) When the instruction is executed, the processor will The current position of the device is determined using the position sensor, A non-temporary computer-readable storage medium according to embodiment 33, further comprising displaying a linked image corresponding to the determined current position. (37) A non-temporary computer-readable storage medium according to Embodiment 33, on which the device image data is automatically captured. (38) The device image data is captured when a user command is received in a non-temporary computer-readable storage medium according to Embodiment 33. (39) The non-temporary computer-readable storage medium according to Embodiment 27, wherein when the instruction is executed, it causes the processor to further tag the location or feature of an object within the internal area. (40) A non-temporary computer-readable storage medium according to Embodiment 39, wherein tagging includes receiving user input.

[0188] (41) A non-temporary computer-readable storage medium according to Embodiment 39, wherein tagging includes automatically detecting the location or characteristics of the subject. (42) The non-temporary computer-readable storage medium according to Embodiment 39, wherein, when the instruction is executed, the processor further causes the tagged location or feature of the target to be superimposed onto the reference image. (43) The non-temporary computer-readable storage medium according to Embodiment 27, wherein, when the instruction is executed, the processor further causes the visual markers to connect and characterize the historical path of the movement of the device. (44) The non-temporary computer-readable storage medium according to Embodiment 27, wherein the visual markings include a mesh. (45) The non-temporary computer-readable storage medium according to Embodiment 44, wherein the mesh shows the anatomical structure of the internal region.

[0189] (46) The mesh is derived from the subset of the position dataset and the image data received from the imaging device on the instrument, in the non-temporary computer-readable storage medium according to Embodiment 45. (47) The non-temporary computer-readable storage medium according to Embodiment 27, wherein the subset of the positional dataset is superimposed on the reference image at a duration frequency. (48) A non-temporary computer-readable storage medium according to Embodiment 27, wherein the subset of the positional dataset is superimposed on the reference image at positional frequency. (49) The non-temporary computer-readable storage medium according to Embodiment 27, wherein, when the instruction is executed, the processor further causes the visual sign to be displayed during surgery. (50) The non-temporary computer-readable storage medium according to Embodiment 27, wherein, when the instruction is executed, the processor further causes the visual mark to be stored for use during a future medical procedure.

[0190] (51) The internal region of the body includes a kidney, and when the instruction is executed, the processor, Moving the aforementioned instrument into the kidney cup, A non-temporary computer-readable storage medium according to Embodiment 27, further comprising tagging at least one of the following: the entrance to the cup of the kidney, the pole of the kidney, a gallstone within the kidney, and a region of transitional cell carcinoma. (52) When the instruction is executed, the processor will Adjusting the position determined by the position sensor in order to take physiological dynamics into consideration, A non-temporary computer-readable storage medium according to embodiment 27, further comprising superimposing the adjusted position onto the reference image. (53) A robotic surgical system, A device comprising an elongated body and at least one position sensor disposed on the elongated body, At least one computer-readable memory that stores executable instructions, One or more processors communicating with the at least one computer-readable memory, executing the instructions to the system, Moving the aforementioned device within the internal region of the body, During the movement of the aforementioned device, position information is received from at least one position sensor. Displaying an image of the internal region of the body, A robotic surgical system comprising: one or more processors configured to overlay visual markers derived from at least a subset of the position dataset onto the image to characterize the historical position of the instrument during its movement within the internal region of the body. (54) The system according to embodiment 53, wherein the position sensor includes an electromagnetic sensor. (55) The system according to embodiment 53, wherein the position sensor includes a shape-detecting fiber.

[0191] (56) The system according to embodiment 53, wherein the position sensor is located on the distal end of the elongated body. (57) The system according to embodiment 53, wherein the instrument includes an endoscope. (58) The system according to embodiment 53, wherein the instrument includes a ureteroscope. (59) The system according to embodiment 53, wherein the elongated body is articulated to control the posture of the device. (60) The system according to embodiment 53, further comprising an instrument positioning device connected to the instrument, wherein the instrument positioning device is configured to operate the instrument.

[0192] (61) The system according to embodiment 60, wherein the instrument positioning device includes a robotic arm. (62) The system according to embodiment 53, wherein the reference image includes an image captured during retrograde pyelonephrography. (63) The system according to embodiment 53, wherein the reference image includes an X-ray fluoroscopic image or an ultrasound image. (64) The system according to embodiment 53, wherein the reference image is captured during computed tomography (CT) or magnetic resonance imaging (MRI) procedure. (65) The system according to embodiment 53, wherein when the instruction is executed, it causes one or more processors to further capture the reference image during surgery.

[0193] (66) The system according to embodiment 53, wherein the reference image is captured before surgery. (67) When the instruction is executed, it causes one or more processors to: Receiving equipment image data from an imaging device positioned on the equipment, wherein the equipment image data includes a plurality of images captured by the imaging device during the movement of the equipment. The system according to embodiment 53, further comprising: linking each image in at least a subset of the plurality of images with the location dataset indicating the location where the image was captured. (68) The system according to embodiment 67, wherein when the instruction is executed, it causes one or more processors to further store the linked images for use in a future procedure. (69) When the instruction is executed, the processor will Receiving user input for location selection, The system according to embodiment 67, further comprising displaying a linked image corresponding to the user input. (70) When the instruction is executed, it will cause one or more processors to: The current position of the device is determined using the position sensor, The system according to embodiment 67 further causes the system to display a linked image corresponding to the determined current position.

[0194] (71) The system according to embodiment 53, wherein the device image data is automatically captured. (72) The system according to embodiment 67, wherein the device image data is captured when a user command is received. (73) The system according to embodiment 53, wherein when the instruction is executed, it causes one or more processors to further tag the location or feature of the object within the internal region. (74) The system according to embodiment 73, wherein tagging includes receiving user input. (75) The system according to embodiment 73, wherein tagging includes automatically detecting the location or characteristics of the subject.

[0195] (76) The system according to embodiment 73, wherein, when the instruction is executed, it causes one or more processors to further superimpose the tagged location or feature of the target onto the reference image. (77) The system according to embodiment 53, wherein, when the instruction is executed, it causes one or more processors to further connect the visual markers to characterize the historical path of the movement of the device. (78) The system according to embodiment 53, wherein the visual markings include a mesh. (79) The system according to embodiment 78, wherein the mesh shows the anatomical structure of the internal region. (80) The system according to embodiment 79, wherein the mesh is derived from the subset of the position dataset and image data received from the imaging device on the instrument.

[0196] (81) The system according to embodiment 53, wherein the subset of the positional dataset is superimposed on the reference image at a duration frequency. (82) The system according to embodiment 53, wherein the subset of the positional dataset is superimposed on the reference image at positional frequency. (83) The system according to embodiment 53, wherein when the instruction is executed, it causes one or more processors to further display the visual sign during surgery. (84) The system according to Embodiment 53, wherein, when the instruction is executed, it causes one or more processors to further store the visual markers for use during future medical procedures. (85) The internal region of the body includes a kidney, and when the instruction is executed, it is sent to one or more processors, Moving the aforementioned instrument into the kidney cup, The system according to embodiment 53, further comprising tagging at least one of the following: the entrance to the cup of the kidney, the pole of the kidney, a gallstone within the kidney, and a region of transitional cell carcinoma.

[0197] (86) When the instruction is executed, it will cause one or more processors to: Adjusting the position determined by the position sensor in order to take physiological dynamics into consideration, The system according to embodiment 53, further comprising superimposing the adjusted position onto the reference image. (87) A non-temporary computer-readable storage medium in which instructions are stored, wherein, when an instruction is executed, the processor of the device provides at least: Moving the device within the internal region of the body, Receiving position information from at least one position sensor of the device while the device is moving, wherein the position information includes a plurality of position datasets, each position dataset indicating the position of the device during the movement; Receiving image data from an imaging device of the device within its internal region during the movement of the device, wherein the image data includes one or more images captured by the imaging device at one or more locations within the internal region. Linking at least a subset of the one or more images to at least a subset of the position dataset based on the position where each image was captured, as determined by the position sensor; Determining user input, including location selection, A non-temporary computer-readable storage medium that displays one of the linked images corresponding to the position selection. (88) A non-temporary computer-readable storage medium according to Embodiment 87, wherein determining the user command includes receiving the user command. (89) The non-temporary computer-readable storage medium according to Embodiment 87, wherein the image data includes still images. (90) A non-temporary computer-readable storage medium according to Embodiment 87, wherein the image data includes video.

[0198] (91) The subset of the location dataset is selected at a duration frequency in the non-temporary computer-readable storage medium according to Embodiment 87. (92) The subset of the location dataset is selected by location frequency in the non-temporary computer-readable storage medium according to Embodiment 87. (93) A non-temporary computer-readable storage medium according to Embodiment 87, wherein the position information includes information indicating the orientation of the device. (94) The non-temporary computer-readable storage medium according to Embodiment 87, in which the image data is automatically captured. (95) The image data is captured when a user command is received in the non-temporary computer-readable storage medium according to Embodiment 87.

[0199] (96) When the instruction is executed, the processor will Receiving user input associated with the current location, A non-temporary computer-readable storage medium according to embodiment 87, further comprising linking the user input to the linked image corresponding to the current position. (97) The non-temporary computer-readable storage medium according to Embodiment 96, wherein, when the instruction is executed, the processor further causes the processor to detect features of a target within the image data. (98) A robotic surgical system for navigating internal regions of the body, wherein the system is It is a device, The long, slender body, At least one position sensor is disposed on the elongated body, An apparatus including an imaging device disposed on the elongated main body, At least one computer-readable memory that stores executable instructions, One or more processors communicating with the at least one computer-readable memory, executing the instructions to the system, Moving the aforementioned device within the internal region of the body, Receiving location information from at least one location sensor while the device is moving, wherein the location information includes a plurality of location datasets, each location dataset indicating the position of the device during its movement, Receiving image data from the imaging device, wherein the image data includes one or more images captured by the imaging device at one or more locations within the internal region during the movement of the instrument. A robotic surgical system comprising: one or more processors configured to link at least a subset of one or more images with at least a subset of the position dataset based on the position where each image was captured, as determined by the position sensor. (99) The system according to embodiment 98, wherein the position sensor includes an EM sensor. (100) The system according to embodiment 98, wherein the position sensor includes a shape-detecting fiber.

[0200] (101) The system according to embodiment 98, wherein the position sensor is located on the distal end of the elongated body. (102) The system according to embodiment 98, wherein the instrument includes an endoscope. (103) The system according to embodiment 98, wherein the instrument includes a ureteroscope. (104) The system according to embodiment 98, wherein the elongated body is articulated to control the posture of the device. (105) The system according to embodiment 98, further comprising an instrument positioning device connected to the instrument, wherein the instrument positioning device is configured to operate the instrument.

[0201] (106) The system according to embodiment 105, wherein the instrument positioning device includes a robotic arm. (107) A method for navigating an internal region of the body, the method being Moving an instrument within the internal region of the body, wherein the instrument includes at least one position sensor and at least one imaging device. Receiving location information from at least one location sensor of the device, wherein the location information includes a plurality of location datasets, each location dataset indicating the position of the device during movement. Receiving image data from the imaging device of the aforementioned apparatus, wherein the image data includes one or more images captured by the imaging device at one or more locations within the internal region. Linking at least a subset of the one or more images to at least a subset of the position dataset based on the position where each image was captured, as determined by the position sensor; The current position of the device is determined using at least one of the position sensors, wherein the current position corresponds to the current position dataset among the plurality of position datasets. A method comprising displaying an image linked to the current location dataset on a user display.

Claims

1. It is a system for robots, A device comprising an elongated body and at least one position sensor disposed on the elongated body, An imaging device positioned on the aforementioned apparatus, At least one computer-readable memory that stores executable instructions, One or more processors communicating with the at least one computer-readable memory, executing the instructions to the system, Moving the aforementioned device within the internal region of the body, During the movement of the device, the device's position information is received from at least one position sensor, the position information includes a plurality of position datasets, and each position dataset indicates the position of the device during its movement. To generate a visual marker indicating the position of the aforementioned device, Displaying a reference image of the internal region of the body, Based on the positional information, the visual mark is superimposed onto the reference image to characterize the historical position of the device during its movement within the internal region of the body, Receiving image data from the imaging device, wherein the image data represents a plurality of images captured by the imaging device during the movement of the instrument; Linking a subset of the plurality of images to the plurality of location datasets based on the location where each of the plurality of images was captured, determined by the at least one location sensor, Based on the aforementioned position information, the current position of the instrument is determined from among the positions of the instrument captured by the imaging device in the plurality of images, Displaying one of the linked images corresponding to the determined current location, or The system receives user input including selecting a location from the aforementioned visual markers, and based on the user input, displays one of the linked images corresponding to the selected location. A system comprising one or more processors configured to perform the following actions.

2. The system according to claim 1, further comprising an instrument positioning device connected to the instrument, wherein the instrument positioning device is configured to operate the instrument, and the instrument positioning device includes a robotic arm.

3. The system according to claim 1, wherein the reference image includes an image captured during retrograde pelvic imaging.

4. The system according to claim 1, wherein the reference image includes an X-ray fluoroscopic image or an ultrasound image.

5. The system according to claim 1, wherein the reference image is captured during computed tomography (CT) or magnetic resonance imaging (MRI) procedure.

6. The system according to claim 1, wherein, when the instruction is executed, it causes one or more processors to further store the image for use in a future procedure.

7. When the instruction is executed, it causes one or more processors to: The current position of the device is determined using the at least one position sensor, The system according to claim 1, further comprising displaying the image corresponding to the determined current position.

8. The system according to claim 1, wherein, when the instruction is executed, it causes one or more processors to further tag the location or feature of an object within the internal region.

9. The system according to claim 8, wherein, when the instruction is executed, it causes one or more processors to further superimpose the tagged location or feature of the target onto the reference image.

10. The system according to claim 1, wherein, when the instruction is executed, it causes one or more processors to further connect the visual markers to characterize the historical path of the movement of the device.

11. The system according to claim 1, wherein the visual markings include a mesh.

12. The internal region of the body includes a kidney, and when the instruction is executed, it is sent to one or more processors, The system according to claim 1, further comprising tagging at least one of the following: the entrance to the renal cup, the pole of the renal, the gallstone within the renal, and the region of transitional cell carcinoma.

13. When the instruction is executed, it causes one or more processors to: Adjusting the position determined by at least one position sensor in order to take physiological dynamics into consideration, The system according to claim 1, further comprising superimposing the adjusted position onto the reference image.

14. The system according to claim 1, wherein the visual markings include a three-dimensional mesh.

15. The internal region of the body includes a kidney, and when the instruction is executed, it is sent to one or more processors, Moving the aforementioned device into the kidney, The system according to claim 1, further comprising tagging one or more locations within the kidney.

Images