Methods and systems for mapping and navigation
Through robots enabling medical systems and orientation sensor technology, visual markings are generated, which solves the time and radiation problems of navigation and mapping in ureteroscopy, and improves navigation accuracy and efficiency.
Patent Information
- Application Number
- CN201980029394.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2018-03-01
- Filing Date
- 2019-02-28
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2039-02-28
AI Technical Summary
In medical procedures such as ureteroscopy, it is difficult for doctors to effectively navigate and map the internal areas of the body, especially the kidneys, which leads to time-intensive, radiation-intensive and error-prone, and may miss the area of the clove.
Using robots to enable medical systems, use orientation sensors (such as EM sensors) to receive the position and orientation data of the device, generate visual markers, superimpose them on the reference image, and form mapping diagrams of the device's historical path and anatomy structure to help doctors navigate and confirm Instrument position.
Reduces radiation exposure, shortens mapping time, improves navigation accuracy and efficiency, and reduces the risk of missing internal areas.
Smart Images

Figure CN112087981B_ABST
Abstract
Description
[0001] Priority application
[0002] This application claims priority to U.S. Provisional Application No. 62 / 637,048, filed on March 1, 2018, which is incorporated herein by reference. Technical Field
[0003] The present disclosure relates generally to mapping and / or navigating internal regions of the body, and more particularly, to methods and systems for mapping and / or navigating internal regions of the body using robotically enabled medical devices. Background Art
[0004] Medical procedures such as endoscopy (eg, ureteroscopy) may involve accessing and visualizing internal regions of a patient's body (eg, the kidneys) for diagnostic and / or therapeutic purposes.
[0005] Ureteroscopy is a medical procedure commonly used to treat kidney stones. During the procedure, a thin, flexible, tube-like tool or instrument called a ureteroscope is inserted into the urethra, through the bladder and ureters, and into the kidney.
[0006] In certain procedures, a robotically enabled medical system may be used to control the insertion and / or manipulation of an instrument.A robotically enabled medical system may include a robotic arm or other instrument positioning device having a manipulator assembly for controlling the positioning of an instrument during a procedure. Summary of the invention
[0007] Embodiments of the present disclosure relate to systems and techniques for mapping and / or navigating internal regions of the body using robotically enabled medical devices. The systems and techniques may be configured to generate visual markers indicating the historical (e.g., previous) position of an instrument as it is navigated through an internal region. The visual markers may be derived from position information received from a position sensor. The visual markers may be superimposed on a reference image of the internal region. The visual markers may form a mapping diagram that may represent the previous position of the instrument. The mapping diagram may be used to visualize the anatomical structure of the internal region and may be used by a physician to navigate the internal region. In some implementations, the visual markers include points or traces indicating the path of the instrument as it navigates through the internal region. In some implementations, the visual markers may include a grid structure representing the anatomical structure of the internal region.
[0008] The methods and techniques may also be configured to receive image data from an image sensor positioned on the instrument. The image data may include a still image or a video. The methods and techniques may link the image data with the position data so that an image captured at a particular location may be called and displayed to the user. In some implementations, the image is called and displayed when the user selects a position within the interior area or when the instrument is positioned at a position where a linked image exists. In addition, the methods and techniques may be configured to allow the physician to mark certain features or locations of interest.
[0009] Thus, one aspect relates to a method for mapping an internal region of a body. The method includes: displaying a reference image of the internal region of the body; moving an instrument within the internal region of the body, the instrument including at least one position sensor; receiving position information from the at least one position sensor, the position information including a plurality of position data sets, each position data set indicating a position of the instrument during movement of the instrument; and superimposing visual markers derived from at least a subset of the position data sets on the reference image to characterize a historical position of the instrument during movement within the internal region of the body.
[0010] In some embodiments, the method may include one or more of the following features in any combination: (a) wherein the reference image comprises an image captured during a retrograde pyelography procedure; (b) wherein the reference image comprises a fluoroscopic or ultrasound image; (c) wherein the reference image is captured during a computed tomography (CT) or magnetic resonance imaging (MRI) procedure; (d) the reference image is captured during surgery; (e) wherein the reference image is captured pre-surgery; (f) receiving instrument image data from an imaging device positioned on the instrument, the instrument image data comprising images captured during movement of the instrument; (g) comprising storing the linked images for use during future procedures; (h) receiving user input of an orientation selection and displaying a linked image corresponding to the user input; (i) determining a current orientation of the instrument using an orientation sensor and displaying a linked image corresponding to the determined current orientation; (j) wherein the instrument image data is captured automatically; (k) wherein the instrument image data is captured upon receiving a user command; (l) marking a location or feature of interest within the internal region; (m) wherein marking comprises receiving user input; (n) wherein marking comprises automatically detecting the location or feature of interest; (o) superimposing the marked location or feature of interest on a reference image; (p) connecting the visual markers to represent the historical path of movement of the instrument; (q) wherein the visual markers comprise a grid; (r) wherein the grid indicates the anatomical structure of the internal region; (s) wherein the grid is derived from a subset of the position data set and image data received from an imaging device on the instrument; (t) wherein the subset of the position data set is superimposed at a duration frequency; (i) superimposing a subset of the orientation data set on a reference image at the orientation frequency; (ii) displaying a visual marker during surgery; (iii) storing the visual marker for use during future medical procedures; (iv) wherein the internal region of the body includes a kidney, and wherein the method further comprises: moving an instrument into a calyx of the kidney; and marking at least one of: an entrance to the calyx of the kidney, a pole of the kidney, a stone within the kidney, and an area of transitional cell carcinoma; and / or (v) adjusting an orientation determined by a orientation sensor to account for physiological movement, and superimposing the adjusted orientation on the reference image.
[0011] In another aspect, a non-transitory computer-readable storage medium having instructions stored thereon is disclosed. The instructions, when executed, may cause a processor of a device to at least: move an instrument within an internal region of a body; receive position information from at least one position sensor of the instrument during movement of the instrument, the position information comprising a plurality of position data sets, each position data set indicating a position of the instrument; display an image of the internal region of the body; and superimpose visual markers derived from at least a subset of the position data sets on the image to characterize a historical position of the instrument during movement of the instrument within the internal region of the body.
[0012] In some embodiments, the non-transitory computer-readable storage medium may include one or more of the following features in any combination: (a) wherein the reference image includes an image captured during a retrograde pyelography procedure; (b) wherein the reference image includes a fluoroscopic or ultrasound image; (c) wherein the reference image is captured during a computed tomography (CT) or magnetic resonance imaging (MRI) procedure; (d) wherein the instructions, when executed, further cause the processor to capture the reference image during surgery; (e) wherein the reference image is captured pre-surgery; (f) wherein the instructions, when executed, further cause the processor to: receive instrument image data from an imaging device positioned on the instrument, the instrument image data including images captured by the imaging device during movement of the instrument; a plurality of images; and for at least a subset of the plurality of images, linking each image of the subset with an orientation data set indicating the orientation at which the image was captured; (g) wherein the instructions, when executed, further cause the processor to store the linked image for use during a future procedure; (h) wherein the instructions, when executed, further cause the processor to: receive user input of an orientation selection; and display a linked image corresponding to the user input; (i) wherein the instructions, when executed, further cause the processor to: determine a current orientation of the instrument using an orientation sensor; and display a linked image corresponding to the determined current orientation; (j) wherein the instrument image data is captured automatically; (k) wherein the instrument image data is captured upon receipt of a user command; (l) wherein the instructions, when executed, further cause the processor to: receive user input of an orientation selection; and display a linked image corresponding to the determined current orientation; The step causes the processor to mark a location or feature of interest within the internal region; (m) wherein the marking comprises receiving user input; (n) wherein the marking comprises automatically detecting the location or feature of interest; (o) wherein the instructions, when executed, further cause the processor to overlay the marked location or feature of interest on a reference image; (p) wherein the instructions, when executed, further cause the processor to connect the visual markers to represent a historical path of movement of the instrument; (q) wherein the visual markers comprise a grid; (r) wherein the grid indicates anatomy of the internal region; (s) wherein the grid is derived from a subset of the position data set and image data received from an imaging device on the instrument; (t) wherein the subset of the position data set is overlaid on the reference image at a duration frequency; u) wherein a subset of the orientation data set is superimposed on a reference image at the orientation frequency; (v) wherein the instructions, when executed, further cause the processor to display the visual marker during surgery; (w) wherein the instructions, when executed, further cause the processor to store the visual marker for use during future medical procedures; (x) wherein the internal region of the body includes a kidney, and wherein the instructions, when executed, further cause the processor to: move the instrument into a calyx of the kidney; and mark at least one of: an entrance to the calyx of the kidney, a pole of the kidney, a stone within the kidney, and an area of transitional cell carcinoma; and / or (y) wherein the instructions, when executed, further cause the processor to: adjust the orientation determined by the orientation sensor to account for physiological movement; and superimpose the adjusted orientation on the reference image.
[0013] In another aspect, a robotic surgical system is disclosed. The robotic surgical system may include: an instrument having an elongated body and at least one position sensor disposed on the elongated body; at least one computer readable memory having executable instructions stored thereon; and one or more processors in communication with the at least one computer readable memory and configured to execute the instructions to cause the system to at least: move the instrument within an internal region of a body; receive position information from the at least one position sensor during movement of the instrument; display an image of the internal region of the body; and superimpose visual markers derived from at least a subset of the position data set on the image to characterize a historical position of the instrument during movement of the instrument within the internal region of the body.
[0014] In some embodiments, the system can include one or more of the following features in any combination: (a) wherein the position sensor comprises an electromagnetic sensor; (b) wherein the position sensor comprises a shape sensing fiber; (c) wherein the position sensor is positioned on the distal end of the elongated body; (d) wherein the instrument comprises an endoscope; (e) wherein the instrument comprises a ureteroscope; (f) wherein the elongated body is capable of articulation to control the posture of the instrument; (g) an instrument positioning device coupled to the instrument, the instrument positioning device being configured to manipulate the instrument; (h) wherein the instrument positioning device comprises a robotic arm; (i) wherein the reference image comprises an image captured during a retrograde pyelography procedure; (j) wherein the reference image comprises a fluoroscopic or ultrasound image; (k) wherein the reference image comprises an ultrasound image; ) wherein the reference image is captured during a computed tomography (CT) or magnetic resonance imaging (MRI) procedure; (l) wherein the instructions, when executed, further cause the one or more processors to capture the reference image during surgery; (m) wherein the reference image is captured pre-surgery; (n) wherein the instructions, when executed, further cause the one or more processors to: receive instrument image data from an imaging device positioned on the instrument, the instrument image data comprising a plurality of images captured by the imaging device during movement of the instrument; and for at least a subset of the plurality of images, link each image of the subset with an orientation data set indicating an orientation at which the image was captured; (o) wherein the instructions, when executed, further cause the one or more processors to store the linked image for future use during a procedure (p) wherein the instructions, when executed, further cause the processor to: receive user input of a position selection; and display a linked image corresponding to the user input; (q) wherein the instructions, when executed, further cause the one or more processors to: determine the current position of the instrument using an orientation sensor; and display a linked image corresponding to the determined current position; (r) wherein the instrument image data is automatically captured; (s) wherein the instrument image data is captured upon receiving a user command; (t) wherein the instructions, when executed, further cause the one or more processors to mark a location or feature of interest within the internal area; (u) wherein marking includes receiving user input; (v) wherein marking includes automatically detecting a location or feature of interest; (w ... (x) wherein the instructions, when executed, further cause the one or more processors to superimpose the marked locations or features of interest on the reference image; (y) wherein the visual markers include a grid; (z) wherein the grid indicates the anatomical structure of the internal region; (aa) wherein the grid is derived from a subset of the position data set and image data received from an imaging device on the instrument; (bb) wherein the subset of the position data set is superimposed on the reference image at a duration frequency; (cc) wherein the subset of the position data set is superimposed on the reference image at a position frequency; (dd) wherein the instructions, when executed, further cause the one or more processors to display the visual markers during surgery;(ee) wherein the instructions, when executed, further cause the one or more processors to store the visual marker for use during a future medical procedure; (ff) wherein the internal region of the body includes a kidney, and wherein the instructions, when executed, further cause the one or more processors to: move the instrument into a calyx of the kidney; and mark at least one of: an entrance to the calyx of the kidney, a pole of the kidney, a stone within the kidney, and an area of transitional cell carcinoma; and / or (gg) wherein the instructions, when executed, further cause the one or more processors to: adjust the position determined by the position sensor to account for physiological movement; and superimpose the adjusted position on the reference image.;
[0015] In another aspect, a non-transitory computer-readable storage medium having instructions stored thereon is described. The instructions, when executed, may cause a processor of a device to at least: move an instrument within an internal region of a body; receive position information from at least one position sensor of the instrument during movement of the instrument, the position information comprising a plurality of position data sets, each position data set indicating the position of the instrument during movement of the instrument; receive image data from an imaging device of the instrument within the internal region during movement of the instrument, the image data comprising one or more images captured by the imaging device at one or more locations within the internal region; link at least a subset of the one or more images with at least a subset of the position data sets based on the position at which each image was captured determined by the position sensor; determine a user input comprising a position selection; and display a linked image of the linked images corresponding to the position selection.
[0016] In some embodiments, the non-transitory computer-readable storage medium may include one or more of the following features in any combination: (a) wherein determining a user command includes receiving a user command; (b) wherein the image data includes a static image; (c) wherein the image data includes a video; (d) wherein a subset of the orientation data set is selected at a duration frequency; (e) wherein a subset of the orientation data set is selected at an orientation frequency; (f) wherein the orientation information includes information indicating the orientation of the instrument; (g) wherein the image data is captured automatically; (h) wherein the image data is captured upon receiving a user command; (i) wherein the instructions, when executed, further cause the processor to: receive user input associated with a current orientation; and link the user input with a linked image corresponding to the current orientation; and / or (j) wherein the instructions, when executed, further cause the processor to detect a feature of interest within an image of the image data.
[0017] In another aspect, a robotic surgical system for navigating an internal region of a body is described. The system may include: an instrument including an elongated body, at least one position sensor disposed on the elongated body, and an imaging device disposed on the elongated body; at least one computer-readable memory having executable instructions stored thereon; and one or more processors in communication with the at least one computer-readable memory and configured to execute instructions to cause the system to at least: move the instrument within the internal region of the body; receive position information from the at least one position sensor during movement of the instrument, the position information including a plurality of position data sets, each position data set indicating a position of the instrument during movement of the instrument; receive image data from the imaging device, the image data including one or more images captured by the imaging device at one or more locations within the internal region during movement of the instrument; and link at least a subset of the one or more images with at least a subset of the position data sets based on the position at which each image was captured as determined by the position sensor.
[0018] In some embodiments, the system may include one or more of the following features in any combination: (a) wherein the position sensor comprises an EM sensor; (b) wherein the position sensor comprises a shape sensing fiber; (c) wherein the position sensor is positioned at the distal end of the elongated body; (d) wherein the instrument comprises an endoscope; (e) wherein the instrument comprises a ureteroscope; (f) wherein the elongated body is capable of articulating to control a posture of the instrument; (g) an instrument positioning device coupled to the instrument, the instrument positioning device being configured to manipulate the instrument; and / or (h) wherein the instrument positioning device comprises a robotic arm.
[0019] In another aspect, a method for navigating an internal region of a body is disclosed. The method may include: moving an instrument within the internal region of the body, the instrument including at least one position sensor and at least one imaging device; receiving position information from at least one position sensor of the instrument, the position information including a plurality of position data sets, each position data set indicating a position of the instrument during movement of the instrument; receiving image data from an imaging device of the instrument, the image data including one or more images captured by the imaging device at one or more locations within the internal region; linking at least a subset of the one or more images with at least a subset of the position data sets based on a position at which each image was captured determined by the position sensor; determining a current position of the instrument using the at least one position sensor, the current position corresponding to a current position data set in the plurality of position data sets; and displaying the image linked to the current position data set on a user display. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] The disclosed aspects will hereinafter be described in conjunction with the accompanying drawings, which are provided to illustrate and not to limit the disclosed aspects, wherein like reference numerals represent like elements.
[0021] Figure 1 An embodiment of a cart-based robotic system arranged for use in diagnostic and / or therapeutic bronchoscopy procedures is shown.
[0022] Figure 2 Depicted Figure 1 Other aspects of the robotic system.
[0023] Figure 3 Shown is a ureteroscopy arrangement. Figure 1 Implementation scheme of a robotic system.
[0024] Figure 4 Shown is a vascular procedure arranged Figure 1 Implementation scheme of a robotic system.
[0025] Figure 5 An embodiment of a table-based robotic system arranged for use in a bronchoscopy procedure is shown.
[0026] Figure 6 Provided Figure 5 Alternative views of the robotic system.
[0027] Figure 7 An exemplary system configured to stow a robotic arm is shown.
[0028] Figure 8 An embodiment of a table-based robotic system configured for use in a ureteroscopy procedure is shown.
[0029] Fig. 9 An embodiment of a table-based robotic system configured for laparoscopic procedures is shown.
[0030] Fig.10 Shown with pitch or tilt adjustment Figure 5-Figure 9 Implementation of a stage-based robotic system.
[0031] Fig.11 Provided Figure 5-Figure 10 Detailed illustration of the interface between the stage and column of a stage-based robotic system.
[0032] Fig.12 An exemplary instrument driver is shown.
[0033] Fig.13 An exemplary medical instrument having a pair of instrument drivers is shown.
[0034] Fig.14An alternative design of an instrument drive and an instrument is shown, wherein the axis of the drive unit is parallel to the axis of the elongated shaft of the instrument.
[0035] Fig.15 Depicted is a block diagram showing a positioning system that estimates a Figure 1-Figure 10 The position of one or more elements of a robotic system, such as Fig.13 and Fig.14 The location of the device.
[0036] Fig.16A is a first exemplary fluoroscopic image showing an instrument navigated within a kidney.
[0037] Fig. 16B The intrarenal navigation Fig.16A A second exemplary fluoroscopic image of an instrument.
[0038] Fig. 16C The intrarenal navigation Fig.16A A third exemplary fluoroscopic image of an instrument.
[0039] Fig.17 An exemplary representation of visual indicia indicating position data received from a position sensor of an instrument navigated within a kidney is shown.
[0040] Fig.18 shows a superimposed reference image of the kidney Fig.17 A visual marker representation of .
[0041] Fig.19 It is shown that images captured by an imaging device of a medical device navigating within an internal region of a patient can be linked to the position of the captured image of the medical device.
[0042] Fig. 20A is a flow chart illustrating an exemplary method for mapping an internal region of a body.
[0043] Fig. 20B is a flow chart illustrating another exemplary method for mapping an internal region of a body.
[0044] Fig.21A is a flow chart illustrating an exemplary method for navigating an internal region of a body.
[0045] Fig.21B is a flow chart illustrating another exemplary method for navigating an internal region of a body.
[0046] Fig. 22 is a block diagram illustrating certain components of an embodiment of a system for mapping and / or navigating an internal region of a body. DETAILED DESCRIPTION
[0047] 1. Overview .
[0048] Aspects of the present disclosure may be integrated into a robotically enabled medical system that is capable of performing a variety of medical procedures, including both minimally invasive procedures such as laparoscopy, and non-invasive procedures such as endoscopy. In endoscopy procedures, the system may be able to perform bronchoscopy, ureteroscopy, gastroscopy, etc.
[0049] In addition to performing a wide range of procedures, the system may provide additional benefits such as enhanced imaging and guidance to assist the physician. Additionally, the system may provide the physician with the ability to perform procedures from an ergonomic position without the need for awkward arm motions and positions. Additionally, the system may provide the physician with the ability to perform procedures with improved ease of use, such that one or more of the system's instruments may be controlled by a single user.
[0050] For illustrative purposes, various embodiments will be described below in conjunction with the accompanying drawings. It should be understood that many other specific implementations of the disclosed concepts are possible, and various advantages can be achieved using the disclosed specific implementations. Titles are included herein for reference and to help locate the various sections. These titles are not intended to limit the scope of the concepts described therein. Such concepts may have applicability throughout the specification.
[0051] A. Robotic System – Cart .
[0052] Robot-enabled medical systems can be configured in a variety of ways, depending on the specific procedure. Figure 1 An embodiment of a cart-based, robotically enabled system 10 arranged for use in a diagnostic and / or therapeutic bronchoscopy procedure is shown. During a bronchoscopy procedure, the system 10 may include a cart 11 having one or more robotic arms 12 to deliver a medical instrument such as a steerable endoscope 13 (which may be a bronchoscope specific to a procedure used for bronchoscopy) to a natural orifice entry point (i.e., the mouth of a patient positioned on a table in this example) to deliver the diagnostic and / or therapeutic tool. As shown, the cart 11 may be positioned proximate to the patient's upper torso to provide access to the entry point. Similarly, the robotic arms 12 may be actuated to position the bronchoscope relative to the entry point. The robotic arms 12 may also be utilized when performing gastrointestinal (GI) procedures with a gastroscope (a specialized endoscope used for GI procedures). Figure 1 The arrangement in . Figure 2 An exemplary embodiment of a cart is depicted in greater detail.
[0053] Continue to refer Figure 1, once the cart 11 is properly positioned, the robotic arm 12 can robotically, manually, or a combination thereof, insert the steerable endoscope 13 into the patient. As shown, the steerable endoscope 13 may include at least two telescopic parts, such as an inner guide portion and an outer sheath portion, each of which is coupled to a separate instrument driver from a set of instrument drivers 28, each of which is coupled to the distal end of a separate robotic arm. This linear arrangement of the instrument drivers 28 that facilitates coaxial alignment of the guide portion with the sheath portion produces a "virtual track" 29, which can be repositioned in space by manipulating one or more robotic arms 12 to different angles and / or orientations. The virtual tracks described herein are depicted using dashed lines in the accompanying drawings, and therefore the dashed lines do not depict any physical structure of the system. The translation of the instrument driver 28 along the virtual track 29 causes the inner guide portion to telescope relative to the outer sheath portion, or to advance or retract the endoscope 13 from the patient. The angle of the virtual track 29 can be adjusted, translated, and pivoted based on clinical applications or physician preferences. For example, in bronchoscopy, the angle and orientation of the virtual track 29 as shown represents a compromise between providing the physician with access to the endoscope 13 while minimizing the friction caused by bending the endoscope 13 into the patient's mouth.
[0054] After insertion, the endoscope 13 can be guided down the patient's trachea and lungs using precise commands from the robotic system until the target destination or surgical site is reached. To enhance navigation through the patient's pulmonary network and / or to reach the desired target, the endoscope 13 can be manipulated to telescopically extend the inner guide portion from the outer sheath portion to obtain enhanced articulation and a larger bending radius. The use of a separate instrument driver 28 also allows the guide portion and the sheath portion to be driven independently of each other.
[0055] For example, the endoscope 13 can be guided to deliver a biopsy needle to a target, such as a lesion or nodule in a patient's lung. The needle can be deployed downward along a working channel that extends the length of the endoscope to obtain a tissue sample to be analyzed by a pathologist. Depending on the pathological results, additional tools can be deployed downward along the working channel of the endoscope for additional biopsies. After identifying that the nodule is malignant, the endoscope 13 can deliver tools through the endoscope to remove potential cancerous tissue. In some cases, diagnostic and therapeutic treatments may need to be delivered in separate procedures. In these cases, the endoscope 13 can also be used to deliver a benchmark to "mark" the location of the target nodule. In other cases, diagnostic and therapeutic treatments can be delivered during the same procedure.
[0056] The system 10 may also include a movable tower 30 that may be connected to the cart 11 via a support cable to provide control, electronic, fluid, optical, sensor and / or power support to the cart 11. Placing such functionality in the tower 30 allows for a smaller form factor cart 11 that can be more easily adjusted and / or repositioned by the operating physician and his / her staff. Additionally, dividing functionality between the cart / table and the support tower 30 reduces operating room clutter and facilitates improved clinical workflow. While the cart 11 may be positioned close to the patient, the tower 30 may be stowed in a remote location to be out of the way during the procedure.
[0057] To support the robotic system described above, the tower 30 may include components of a computer-based control system that stores computer program instructions, for example, in a non-transitory computer-readable storage medium such as a permanent magnetic storage drive, a solid-state drive, or the like. Whether the execution occurs in the tower 30 or in the cart 11, the execution of these instructions can control the entire system or its subsystems. For example, when executed by a processor of a computer system, the instructions can cause the components of the robotic system to actuate the associated brackets and arm mounts, actuate the robotic arm, and control the medical device. For example, in response to receiving a control signal, a motor in a joint of the robotic arm can position the arm into a specific posture.
[0058] The tower 30 may also include pumps, flow meters, valve controllers, and / or fluid pathways to provide controlled irrigation and aspiration capabilities to systems that may be deployed through the endoscope 13. These components may also be controlled using the computer system of the tower 30. In some embodiments, irrigation and aspiration capabilities may be delivered directly to the endoscope 13 via separate cables.
[0059] The tower 30 may include voltage and surge protectors designed to provide filtered and protected power to the cart 11, thereby avoiding the need to place power transformers and other auxiliary power components in the cart 11, resulting in a smaller, more mobile cart 11.
[0060] The tower 30 may also include support equipment for sensors deployed throughout the robotic system 10. For example, the tower 30 may include optoelectronic equipment for detecting, receiving, and processing data received from optical sensors or cameras throughout the robotic system 10. In conjunction with the control system, such optoelectronic equipment may be used to generate real-time images for display in any number of consoles deployed throughout the system (including display in the tower 30). Similarly, the tower 30 may also include electronic subsystems for receiving and processing signals received from deployed electromagnetic (EM) sensors. The tower 30 may also be used to house and position EM field generators for detection by EM sensors in or on medical devices.
[0061] The tower 30 may include a console 31 in addition to other consoles available in the rest of the system (e.g., a console mounted on top of a cart). The console 31 may include a user interface and display screen, such as a touch screen, for a physician operator. The console in the system 10 is typically designed to provide both robotic control and preoperative information and real-time information for a procedure, such as navigation and positioning information for the endoscope 13. When the console 31 is not the only console available to the physician, it may be used by a second operator (such as a nurse) to monitor the health or vital signs of the patient and the operation of the system, as well as to provide procedure-specific data, such as navigation and positioning information.
[0062] The tower 30 may be coupled to the cart 11 and the endoscope 13 via one or more cables or connections (not shown). In some embodiments, support functions from the tower 30 may be provided to the cart 11 via a single cable, thereby simplifying and eliminating clutter in the operating room. In other embodiments, specific functions may be coupled in separate wiring and connections. For example, while power may be provided to the cart via a single cable, support for control, optics, fluidics, and / or navigation may also be provided via separate cables.
[0063] Figure 2 Provided by Figure 1 Detailed illustration of an embodiment of a cart for a cart-based robotic enabling system is shown. The cart 11 generally includes an elongated support structure 14 (commonly referred to as a "column"), a cart base 15, and a console 16 at the top of the column 14. The column 14 may include one or more brackets, such as for supporting one or more robotic arms 12 ( Figure 2 The bracket 17 (alternatively "arm support") may include a separately configurable arm mount that rotates along a vertical axis to adjust the base of the robotic arm 12 to better position it relative to the patient. The bracket 17 also includes a bracket interface 19 that allows the bracket 17 to translate vertically along the column 14.
[0064] The bracket interface 19 is connected to the column 14 through slots, such as slot 20, which are positioned on opposite sides of the column 14 to guide the vertical translation of the bracket 17. The slot 20 contains a vertical translation interface to position and maintain the bracket at various vertical heights relative to the cart base 15. The vertical translation of the bracket 17 allows the cart 11 to adjust the reach of the robotic arm 12 to meet various table heights, patient sizes, and physician preferences. Similarly, the separately configurable arm mounts on the bracket 17 allow the robotic arm base 21 of the robotic arm 12 to be angled in a variety of configurations.
[0065] In some embodiments, the slot 20 may be supplemented with a slot cover that is flush and parallel to the slot surface to prevent dust and fluid from entering the internal cavity of the column 14 and the vertical translation interface when the carriage 17 is vertically translated. The slot cover can be deployed by a pair of spring reels positioned near the vertical top and bottom of the slot 20. The cover is coiled within the reel until it is deployed to extend and retract from the coiled state of the cover when the carriage 17 is vertically translated up and down. When the carriage 17 is translated toward the reel, the spring loading of the reel provides a force to retract the cover into the reel while also maintaining a tight seal when the carriage 17 is translated away from the reel. The cover can be connected to the carriage 17 using, for example, a bracket in the carriage interface 19 to ensure proper extension and retraction of the cover when the carriage 17 is translated.
[0066] Column 14 may internally include mechanisms such as gears and motors designed to mechanized translate carriage 17 using a vertically aligned lead screw in response to control signals generated in response to user input (eg, input from console 16 ).
[0067] The robotic arm 12 may generally include a robotic arm base 21 and an end effector 22 separated by a series of links 23, the series of links being connected by a series of joints 24, each joint including an independent actuator, each actuator including an independently controllable motor. Each independently controllable joint represents an independent degree of freedom available to the robotic arm. Each of the arms 12 has seven joints and therefore provides seven degrees of freedom. Multiple joints result in multiple degrees of freedom, thereby allowing "redundant" degrees of freedom. The redundant degrees of freedom allow the robotic arm 12 to position its corresponding end effector 22 at a specific position, orientation, and trajectory in space using different link positions and joint angles. This allows the system to position and guide the medical device from a desired point in space, while allowing the physician to move the arm joint to a clinically advantageous position away from the patient to create greater access while avoiding arm collisions.
[0068] The cart base 15 balances the weight of the column 14, bracket 17, and arm 12 on the floor. Thus, the cart base 15 houses heavier components such as electronics, motors, power supplies, and components that enable the cart to be moved and / or fixed. For example, the cart base 15 includes rollable wheel-shaped casters 25 that allow the cart to be easily moved around the room before a procedure. After reaching the proper orientation, the casters 25 can be secured using wheel locks to keep the cart 11 in the proper orientation during the procedure.
[0069] The console 16 positioned at the vertical end of the column 14 allows both a user interface and a display screen (or a dual-purpose device, such as a touch screen 26) for receiving user input to provide both preoperative and intraoperative data to the physician user. Potential preoperative data on the touch screen 26 may include preoperative planning, navigation and mapping data derived from a preoperative computerized tomography (CT) scan and / or a record from a preoperative patient interview. The intraoperative data on the display may include optical information provided from tools, sensors and coordinate information from sensors and important patient statistics, such as respiration, heart rate and / or pulse. The console 16 may be positioned and tilted to allow the physician to approach the console from the side of the column 14 opposite to the bracket 17. From this position, the physician can observe the console 16, the robot arm 12 and the patient while operating the console 16 from behind the cart 11. As shown, the console 16 also includes a handle 27 to help manipulate and stabilize the cart 11.
[0070] Figure 3 An embodiment of a robotic-enabled system 10 arranged for ureteroscopy is shown. In a ureteroscopy procedure, a cart 11 can be positioned to deliver a ureteroscope 32 (a protocol-specific endoscope designed to traverse a patient's urethra and ureters) to the patient's lower abdominal region. In a ureteroscopy, it may be desirable to align the ureteroscope 32 directly with the patient's urethra to reduce friction and forces on sensitive anatomical structures in this area. As shown, the cart 11 can be aligned at the foot of the table to allow the robotic arm 12 to position the ureteroscope 32 for direct linear access to the patient's urethra. The robotic arm 12 can insert the ureteroscope 32 directly into the patient's lower abdomen through the urethra along a virtual track 33 from the foot of the table.
[0071] After insertion into the urethra, the ureteroscope 32 can be navigated into the bladder, ureters, and / or kidneys for diagnostic and / or therapeutic applications using similar control techniques as in bronchoscopy. For example, the ureteroscope 32 can be guided into the ureters and kidneys to break up accumulated kidney stones using a laser or ultrasonic lithotripsy device deployed down the working channel of the ureteroscope 32. After lithotripsy is complete, the resulting stone fragments can be removed using a basket deployed down the ureteroscope 32.
[0072] Figure 4An embodiment of a robotically enabled system similarly arranged for vascular procedures is shown. In a vascular procedure, the system 10 can be configured so that the cart 11 can deliver a medical device 34 (such as a steerable catheter) to an access point in the femoral artery of the patient's leg. The femoral artery presents both a larger diameter for navigation and a relatively less circuitous and tortuous path to the patient's heart, which simplifies navigation. As in a ureteroscopy procedure, the cart 11 can be positioned toward the patient's legs and lower abdomen to allow the robotic arm 12 to provide a virtual track 35 for direct linear access to the femoral artery access point in the patient's thigh / hip region. After insertion into the artery, the medical device 34 can be guided and inserted by translating the instrument drive 28. Alternatively, the cart can be positioned around the patient's upper abdomen to reach alternative vascular access points, such as the carotid and brachial arteries near the shoulder and wrist.
[0073] B. Robotic system – unit .
[0074] Embodiments of the robotic-enabled medical system may also incorporate a patient table. Incorporating a table reduces the amount of capital equipment in the operating room by removing a cart, which allows for greater access to the patient. Figure 5 An embodiment of such a robotically enabled system is shown arranged for use in a bronchoscopy procedure. System 36 includes a support structure or column 37 for supporting a platform 38 (shown as a "table" or "bed") on a floor. Much like a cart-based system, the end effector of a robotic arm 39 of system 36 includes an instrument drive 42 designed to manipulate an elongated medical instrument, such as a slender medical instrument, through or along a virtual track 41 formed by the linear alignment of the instrument drive 42. Figure 5 In practice, a C-arm for providing fluoroscopic imaging can be positioned over the upper abdominal region of the patient by placing the emitter and detector around the table 38.
[0075] Figure 6An alternative view of the system 36 without the patient and medical device for discussion purposes is provided. As shown, the column 37 may include one or more brackets 43 shown as annular in the system 36, on which one or more robotic arms 39 may be based. The bracket 43 may translate along a vertical column interface 44 extending along the length of the column 37 to provide different vantage points from which the robotic arm 39 may be positioned to reach the patient. The bracket 43 may rotate around the column 37 using a mechanical motor positioned within the column 37 to allow the robotic arm 39 to access multiple sides of the table 38, such as both sides of the patient. In an embodiment with multiple brackets, the brackets may be positioned individually on the column and may translate and / or rotate independently of the other brackets. Although the bracket 43 need not be around the column 37 or even circular, the annular shape as shown facilitates the rotation of the bracket 43 around the column 37 while maintaining structural balance. The rotation and translation of the bracket 43 allows the system to align medical devices such as endoscopes and laparoscopes to different entry points on the patient.
[0076] The arm 39 may be mounted on the carriage via a set of arm mounts 45 including a series of joints that may be individually rotated and / or telescopically extended to provide additional configurability to the robotic arm 39. Additionally, the arm mount 45 may be positioned on the carriage 43 such that when the carriage 43 is appropriately rotated, the arm mount 45 may be positioned on the same side of the table 38 (e.g., Figure 6 As shown), on the opposite side of the platform 38 (as shown Fig. 9 shown) or on an adjacent side of stage 38 (not shown).
[0077] The column 37 provides structural support for the stage 38 and provides a path for the vertical translation of the carriage. Internally, the column 37 can be equipped with a lead screw for guiding the vertical translation of the carriage, and a motor for mechanizing the translation of the carriage based on the lead screw. The column 37 can also transmit power and control signals to the carriage 43 and the robotic arm 39 mounted thereon.
[0078] The base 46 has Figure 2 The cart base 15 in the illustrated cart 11 functions similarly, accommodating heavier components to balance the table / bed 38, column 37, bracket 43, and robotic arm 39. The table base 46 may also incorporate rigid casters to provide stability during the procedure. Casters deployed from the bottom of the table base 46 may extend in opposite directions on both sides of the base 46 and retract when the system 36 needs to be moved.
[0079] continue Figure 6, the system 36 may also include a tower (not shown) that divides the functionality of the system 36 between the table and the tower to reduce the form factor and volume of the table. As in the previously disclosed embodiments, the tower may provide various support functions to the table, such as processing, computing and control capabilities, power, fluid and / or optical and sensor processing. The tower may also be movable to be positioned away from the patient, thereby improving physician access and eliminating clutter in the operating room. In addition, placing components in the tower allows for more storage space in the table base for potential stowage of the robotic arm. The tower may also include a console that provides both a user interface (such as a keyboard and / or pendant) for user input and a display screen (or touch screen) for preoperative and intraoperative information (such as real-time imaging, navigation, and tracking information).
[0080] In some embodiments, the table base can fold up and store the robotic arm when not in use. Figure 7 A system 47 for stowing the robotic arm in an embodiment of a table-based system is shown. In the system 47, a carriage 48 can be translated vertically into a base 49 to stow a robotic arm 50, an arm mount 51, and the carriage 48 within the base 49. A base cover 52 can be translated and retracted open to deploy the carriage 48, the arm mount 51, and the arm 50 around a column 53, and closed to stow the carriage 48, the arm mount 51, and the arm 50 to protect them when not in use. The base cover 52 can be sealed along the edges of its opening with a membrane 54 to prevent dust and fluids from entering when closed.
[0081] Figure 8 An embodiment of a robotically enabled table-based system configured for a ureteroscopy procedure is shown. During ureteroscopy, the table 38 may include a rotating portion 55 for positioning the patient at an angle to the column 37 and the table base 46. The rotating portion 55 may rotate or pivot about a pivot point (e.g., located below the patient's head) so as to position the bottom portion of the rotating portion 55 away from the column 37. For example, the pivoting of the rotating portion 55 allows a C-arm (not shown) to be positioned above the patient's lower abdomen without competing for space with the column (not shown) below the table 38. By rotating the bracket 35 (not shown) about the column 37, the robotic arm 39 can insert the ureteroscope 56 directly into the patient's groin area along a virtual track 57 to reach the urethra. During ureteroscopy, stirrups 58 may also be fixed to the rotating portion 55 of the table 38 to support the orientation of the patient's legs during the procedure and allow full access to the patient's groin area.
[0082] In a laparoscopic procedure, minimally invasive instruments (elongated in shape to fit the size of the incision or incisions) can be inserted into the patient's anatomy through small incisions in the patient's abdominal wall. After the patient's abdominal cavity is inflated, instruments, generally called laparoscopes, can be guided to perform surgical tasks such as grasping, cutting, resecting, suturing, etc. Fig. 9 An embodiment of a robotically enabled table-based system configured for laparoscopic procedures is shown. Fig. 9 As shown, the carriage 43 of the system 36 can be rotated and vertically adjusted to position the pair of robotic arms 39 on opposite sides of the table 38 so that the laparoscope 59 can be positioned using the arm mounts 45 to pass through minimal incisions on both sides of the patient to reach his / her abdominal cavity.
[0083] To accommodate laparoscopic procedures, the robotic-enabled table system can also tilt the platform to a desired angle. Fig.10 An embodiment of a robotic-enabled medical system with pitch or tilt adjustment is shown. Fig.10 As shown, system 36 can accommodate the tilt of table 38 to position one portion of the table at a greater distance from the ground than another portion. Additionally, arm mount 45 can be rotated to match the tilt so that arm 39 maintains the same planar relationship with table 38. To accommodate steeper angles, column 37 can also include telescoping portion 60 that allows vertical extension of column 37 to prevent table 38 from contacting the floor or colliding with base 46.
[0084] Fig.11 A detailed illustration of the interface between the table 38 and the column 37 is provided. The pitch rotation mechanism 61 can be configured to change the pitch angle of the table 38 relative to the column 37 with multiple degrees of freedom. The pitch rotation mechanism 61 can be implemented by positioning orthogonal axes 1, 2 at the column-table interface, each axis being actuated by a separate motor 3, 4 in response to an electrical pitch angle command. Rotation along one screw 5 will enable tilt adjustment in one axis 1, while rotation along another screw 6 will enable tilt adjustment along another axis 2.
[0085] For example, pitch adjustment is particularly useful when attempting to position the table in a Trendelenburg orientation (i.e., positioning the patient's lower abdomen at a higher level from the floor than the patient's lower abdomen) for lower abdominal surgery. The Trendelenburg orientation causes the patient's internal organs to glide toward his / her upper abdomen by gravity, thereby clearing the abdominal cavity to allow minimally invasive tools to enter and perform lower abdominal surgical procedures, such as laparoscopic prostatectomy.
[0086] C. Device Drivers and Interfaces .
[0087] The end effector of the system's robotic arm includes: (i) an instrument driver (alternatively referred to as an "instrument drive mechanism" or "instrument device manipulator"), which incorporates an electromechanical device for actuating a medical device; and (ii) a removable or detachable medical device, which may be free of any electromechanical components, such as a motor. This dichotomy may be driven by the need to sterilize medical instruments used in medical procedures; and the inability to adequately sterilize expensive capital equipment due to its complex mechanical assemblies and sensitive electronics. Therefore, the medical device may be designed to be disassembled, removed, and interchanged from the instrument driver (and therefore from the system) for separate sterilization or disposal by a physician or physician's staff. In contrast, the instrument driver does not need to be changed or sterilized, and can be covered for protection.
[0088] Fig.12 An exemplary instrument driver is shown. The instrument driver 62, positioned at the distal end of the robotic arm, includes one or more drive units 63 arranged in parallel axes to provide controlled torque to a medical device via a drive shaft 64. Each drive unit 63 includes a separate drive shaft 64 for interacting with the instrument, a gear head 65 for converting motor shaft rotation into a desired torque, a motor 66 for generating the drive torque, an encoder 67 for measuring the speed of the motor shaft and providing feedback to the control circuit, and a control circuit 68 for receiving control signals and actuating the drive unit. Each drive unit 63 is independently controlled and motorized, and the instrument driver 62 can provide multiple (e.g., Fig.12 In operation, the control circuit 68 will receive the control signal, transmit the motor signal to the motor 66, compare the resulting motor speed measured by the encoder 67 with the desired speed, and modulate the motor signal to generate the desired torque.
[0089] For the procedures requiring a sterile environment, the robotic system can be combined with a drive interface, such as a sterile adapter connected to a sterile cover, which is located between the instrument driver and the medical device. The main purpose of the sterile adapter is to transfer angular motion from the drive shaft of the instrument driver to the drive input of the instrument, while maintaining the physical separation between the drive shaft and the drive input and thus maintaining sterility. Therefore, an exemplary sterile adapter may include a series of rotational inputs and outputs intended to cooperate with the drive shaft of the instrument driver and the drive input on the instrument. The sterile cover composed of a thin flexible material (such as transparent or translucent plastic) connected to the sterile adapter is designed to cover capital equipment, such as instrument drivers, robot arms and carts (in cart-based systems) or tables (in tables-based systems). The use of the cover will allow capital equipment to be positioned near the patient while still being located in an area that does not require sterilization (i.e., non-sterile area). On the other side of the sterile cover, medical devices can dock with patients in areas that require sterilization (i.e., sterile areas).
[0090] D. Medical devices .
[0091] Fig.13 An exemplary medical device with a paired instrument driver is shown. Similar to other instruments designed for use with a robotic system, the medical device 70 includes an elongated shaft 71 (or elongated body) and an instrument base 72. The instrument base 72, also referred to as an "instrument handle" due to its intended design for manual interaction by a physician, can typically include a rotatable drive input 73 (e.g., a socket, pulley, or reel) that is designed to mate with a drive output 74 of a drive interface on an instrument driver 75 extending through a distal end of a robotic arm 76. When physically connected, latched, and / or coupled, the mating drive input 73 of the instrument base 72 can share an axis of rotation with the drive output 74 in the instrument driver 75 to allow torque to be transferred from the drive output 74 to the drive input 73. In some embodiments, the drive output 74 can include a spline that is designed to mate with a socket on the drive input 73.
[0092] The elongated shaft 71 is designed to be delivered through an anatomical opening or lumen (e.g., as in endoscopy) or through a minimally invasive incision (e.g., as in laparoscopy). The elongated shaft 66 can be flexible (e.g., having properties similar to an endoscope) or rigid (e.g., having properties similar to a laparoscope), or a customized combination of both flexible and rigid portions. When designed for laparoscopy, the distal end of the rigid elongated shaft can be connected to an end effector including an articulated wrist formed by a connecting fork having an axis of rotation and a surgical tool (e.g., a grasper or scissors) that can be actuated based on force from a tendon when the drive input rotates in response to a torque received from a drive output 74 of an instrument driver 75. When designed for endoscopy, the distal end of the flexible elongated shaft can include a manipulable or controllable curved segment that articulates and bends based on a torque received from a drive output 74 of an instrument driver 75.
[0093] Tendons within the shaft 71 are used to transmit torque from the instrument driver 75 along the elongated shaft 71. These individual tendons (e.g., pull wires) can be individually anchored to individual drive inputs 73 within the instrument handle 72. From the handle 72, the tendons are guided downward along one or more traction cavities within the elongated shaft 71 and anchored at the distal portion of the elongated shaft 71. In laparoscopy, these tendons can be coupled to a distally mounted end effector, such as a wrist, a gripper, or a pair of scissors. In such an arrangement, the torque applied to the drive input 73 transmits tension to the tendons, causing the end effector to actuate in some manner. In laparoscopy, the tendons can cause the joint to rotate around the axis, causing the end effector to move in one direction or another. Alternatively, the tendons can be connected to one or more jaws of a gripper at the distal end of the elongated shaft 71, wherein the tension from the tendons causes the gripper to close.
[0094] In endoscopy, the tendon can be coupled to a curved or articulated segment located along the elongated shaft 71 (e.g., at the distal end) via an adhesive, a control ring, or other mechanical fastener. When fixedly attached to the distal end of the curved segment, the torque applied to the drive input 73 will be transmitted downward along the tendon, causing the softer curved segment (sometimes referred to as an articulated segment or region) to bend or articulate. Along the uncurved segment, it may be advantageous to spiral or swirl a separate traction cavity that guides a separate tendon along the wall (or inside) of the endoscope shaft to balance the radial forces caused by the tension in the traction wire. For specific purposes, the angle of the spiral and / or the spacing therebetween can be changed or designed, wherein a tighter spiral presents less axial compression under load force, while a lower spiral amount causes greater axial compression under load force, but also presents limited bending. In another case, the traction cavity can be guided parallel to the longitudinal axis of the elongated shaft 71 to allow controlled articulation in the desired curved or articulated segment.
[0095] In endoscopy, the elongated shaft 71 houses a plurality of components to assist in robotic procedures. The shaft may include a working channel at the distal end of the shaft 71 for deploying surgical tools, flushing and / or aspiration of the surgical field. The shaft 71 may also house wires and / or optical fibers to transmit signals to / from an optical assembly at the distal end, which may include an optical camera. The shaft 71 may also house an optical fiber to carry light from a proximally located light source (e.g., a light emitting diode) to the distal end of the shaft.
[0096] At the distal end of the instrument 70, the distal tip may also include an opening for a working channel for delivering tools for diagnosis and / or treatment, flushing and suctioning the surgical site. The distal tip may also include a port for a camera (such as a fiberscope or a digital camera) to capture images of the internal anatomical space. Relatedly, the distal tip may also include a port for a light source, which is used to illuminate the anatomical space when the camera is used.
[0097] exist Fig.13 In the example of , the drive shaft axis, and therefore the drive input axis, is orthogonal to the axis of the elongated shaft. However, this arrangement complicates the ability of the elongated shaft 71 to roll. Rolling the elongated shaft 71 along its axis while holding the drive input 73 stationary can cause undesirable tangling of the tendon as it extends from the drive input 73 and enters the traction cavity within the elongated shaft 71. Such resulting tangling of the tendon may disrupt any control algorithm intended to predict the movement of the flexible elongated shaft during an endoscopic procedure.
[0098] Fig.14An alternative design of an instrument driver and an instrument is shown, wherein the axis of the drive unit is parallel to the axis of the elongated shaft of the instrument. As shown, a circular instrument driver 80 includes four drive units, and its drive output 81 is aligned in parallel at the end of a robot arm 82. The drive units and their respective drive outputs 81 are contained in a rotating assembly 83 of an instrument driver 80 driven by a drive unit in the drive unit in the assembly 83. In response to the torque provided by the rotating drive unit, the rotating assembly 83 rotates along a circular bearing, which connects the rotating assembly 83 to the non-rotating portion 84 of the instrument driver. Electric power and control signals can be transmitted to the rotating assembly 83 from the non-rotating portion 84 of the instrument driver 80 by electrical contact, and the electrical contact can be maintained by the rotation of the brush slip ring connection (not shown). In other embodiments, the rotating assembly 83 can be responsive to a separate drive unit integrated into the non-rotating portion 84, and is therefore not parallel to other drive units. The rotating mechanism 83 allows the instrument driver 80 to allow the drive unit and its corresponding drive output 81 to rotate around the instrument driver axis 85 as a single unit.
[0099] Similar to previously disclosed embodiments, the instrument 86 may include an elongated shaft portion 88 and an instrument base 87 (shown with a transparent outer skin for discussion purposes) that includes a plurality of drive inputs 89 (such as sockets, pulleys, and spools) configured to receive drive outputs 81 in the instrument driver 80. Unlike previously disclosed embodiments, the instrument shaft 88 extends from the center of the instrument base 87, with the axis of the instrument base being substantially parallel to the axis of the drive inputs 89, rather than being substantially parallel to the axis of the drive inputs 89. Fig.13 as orthogonal in the design.
[0100] When coupled to the rotation assembly 83 of the instrument driver 80, the medical device 86 including the instrument base 87 and the instrument shaft 88 rotates in combination with the rotation assembly 83 about the instrument driver axis 85. Since the instrument shaft 88 is located at the center of the instrument base 87, the instrument shaft 88 is coaxial with the instrument driver axis 85 when attached. Therefore, the rotation of the rotation assembly 83 causes the instrument shaft 88 to rotate about its own longitudinal axis. In addition, when the instrument base 87 rotates with the instrument shaft 88, any tendons connected to the drive input 89 in the instrument base 87 do not tangle during rotation. Therefore, the parallelism of the axes of the drive output 81, the drive input 89, and the instrument shaft 88 allows the shaft to rotate without tangling any control tendons.
[0101] E. Navigation and Control .
[0102] Conventional endoscopy may involve the use of fluoroscopy (e.g., as may be delivered by a C-arm) and other forms of radiation-based imaging modalities to provide intracavitary guidance to the operating physician. In contrast, the robotic system contemplated by the present disclosure may provide non-radiation-based navigation and positioning devices to reduce physician exposure to radiation and reduce the amount of equipment in the operating room. As used herein, the term "positioning" may refer to determining and / or monitoring the orientation of an object in a reference coordinate system. Technologies such as preoperative mapping, computer vision, real-time EM tracking, and robotic command data may be used alone or in combination to achieve a radiation-free operating environment. In other cases where radiation-based imaging modalities are still used, preoperative mapping, computer vision, real-time EM tracking, and robotic command data may be used alone or in combination to improve information obtained only by radiation-based imaging modalities.
[0103] Fig.15 is a block diagram illustrating a positioning system 90 for estimating the position of one or more elements of a robotic system, such as the position of an instrument, according to an example embodiment. The positioning system 90 may be a set of one or more computer devices configured to execute one or more instructions. The computer device may be embodied by a processor (or multiple processors) and a computer readable memory in one or more of the components discussed above. By way of example and not limitation, the computer device may be located at Figure 1 The tower 30 shown, Figure 1-Figure 4 The cart shown, Figure 5-Figure 10 The bed shown, etc.
[0104] like Fig.15 As shown, the positioning system 90 may include a positioning module 95 that processes the input data 91-94 to generate position data 96 for the distal tip of the medical device. The position data 96 may be data or logic that represents the position and / or orientation of the distal end of the device relative to a reference frame. The reference frame may be a reference frame relative to the patient's anatomy or a known object such as an EM field generator (see below for a discussion of EM field generators).
[0105] The various input data 91-94 are now described in more detail. Preoperative mapping can be accomplished using a collection of low-dose CT scans. The preoperative CT scans are reconstructed into three-dimensional images that are visualized, for example, as "slices" of cross-sectional views of the patient's internal anatomy. When analyzed as a whole, an image-based model of the anatomical cavities, spaces, and structures of the patient's anatomical structure (such as the patient's lung network) can be generated. Techniques such as centerline geometry can be determined and approximated from the CT images to form a three-dimensional volume of the patient's anatomical structure, which is referred to as preoperative model data 91. The use of centerline geometry is discussed in U.S. patent application Ser. No. 14 / 523,760, the contents of which are incorporated herein in their entirety. Network topology models can also be derived from CT images and are particularly suitable for bronchoscopy.
[0106] In some embodiments, the instrument can be equipped with a camera to provide visual data 92. The positioning module 95 can process the visual data to implement one or more vision-based position tracking. For example, preoperative model data can be used in conjunction with the visual data 92 to implement computer vision-based tracking of a medical instrument (e.g., an endoscope or an instrument advanced through a working channel of an endoscope). For example, using the preoperative model data 91, the robotic system can generate a library of expected endoscopic images from the model based on the expected path of travel of the endoscope, each image linked to a location within the model. In operation, the robotic system can reference the library to compare real-time images captured at a video camera (e.g., a camera at the distal end of an endoscope) with those in the image library to assist in positioning.
[0107] Other computer vision based tracking techniques use feature tracking to determine the motion of the camera, and therefore the motion of the endoscope. Some features of the localization module 95 may identify circular geometric structures in the preoperative model data 91 that correspond to anatomical cavities and track changes in those geometric structures to determine which anatomical cavity is selected, as well as track relative rotational and / or translational motion of the camera. The use of topological maps may further enhance vision based algorithms or techniques.
[0108] Optical flow (another computer vision-based technique) can analyze the displacement and translation of image pixels in a video sequence in the visual data 92 to infer camera movement. Examples of optical flow techniques can include motion detection, object segmentation calculations, brightness, motion compensated encoding, stereo disparity measurements, etc. Through multiple iterations of multi-frame comparisons, the movement and position of the camera (and therefore the endoscope) can be determined.
[0109] The positioning module 95 can use real-time EM tracking to generate the real-time position of the endoscope in a global coordinate system that can be aligned to the patient's anatomical structure represented by the preoperative model. In EM tracking, an EM sensor (or tracker) comprising one or more sensor coils in one or more positions and orientations embedded in a medical device (e.g., an endoscopic tool) measures changes in the EM field generated by one or more static EM field generators positioned at known locations. The position information detected by the EM sensor is stored as EM data 93. The EM field generator (or transmitter) can be placed close to the patient to generate a low-intensity magnetic field that can be detected by the embedded sensor. The magnetic field induces a small current in the sensor coil of the EM sensor, which can be analyzed to determine the distance and angle between the EM sensor and the EM field generator. These distances and orientations can be "aligned" to the patient's anatomical structure (e.g., a preoperative model) during surgery to determine a geometric transformation that aligns a single position in the coordinate system with an orientation in the preoperative model of the patient's anatomical structure. Once registered, an embedded EM tracker in one or more locations of a medical device (eg, the distal tip of an endoscope) can provide a real-time indication of the progress of the medical device through the patient's anatomy.
[0110] The robotic commands and kinematic data 94 may also be used by a positioning module 95 to provide orientation data 96 for the robotic system. The pitch and yaw of the device derived from the joint motion commands may be determined during preoperative calibration. In surgery, these calibration measurements may be used in conjunction with known insertion depth information to estimate the orientation of the instrument. Alternatively, these calculations may be analyzed in conjunction with EM, vision, and / or topological modeling to estimate the orientation of the medical device within the network.
[0111] like Fig.15 As shown, the positioning module 95 can use a variety of other input data. For example, although Fig.15 Not shown, but an instrument utilizing shape sensing fibers may provide shape data that positioning module 95 may use to determine the position and shape of the instrument.
[0112] The localization module 95 may use the input data 91-94 in combination. In some cases, such a combination may use a probabilistic approach, where the localization module 95 assigns a confidence weight to a position determined from each of the input data 91-94. Thus, in situations where the EM data may be unreliable (such as where EM interference may be present), the confidence in the position determined by the EM data 93 may be reduced, and the localization module 95 may rely more heavily on the visual data 92 and / or the robot command and kinematics data 94.
[0113] As discussed above, the robotic systems discussed herein may be designed to incorporate a combination of one or more of the above techniques. A computer-based control system for a robotic system located in a tower, bed, and / or cart may store computer program instructions, for example, in a non-transitory computer-readable storage medium (such as a permanent magnetic storage drive, a solid-state drive, etc.) that, when executed, causes the system to receive and analyze sensor data and user commands, generate control signals for the entire system, and display navigation and positioning data, such as the position of an instrument in a global coordinate system, an anatomical map, etc.
[0114] 2. Mapping and navigation of internal body regions .
[0115] Embodiments of the present disclosure relate to systems and techniques for mapping and / or navigating internal regions of a body using a robot-enabled medical device. As will be described in more detail below, the systems and techniques can be configured to generate visual markers indicating the previous position of an instrument when the instrument is navigated through an internal region. The visual markers can be superimposed on a reference image of the internal region and displayed to a user. The visual markers can be derived from the position information received by the position sensor. The position sensor can be positioned on the instrument. The visual markers can form a mapping diagram representing the previous position of the instrument. The mapping diagram can be used to visualize the anatomical structure of the internal region and can be used by a physician to navigate instruments within the internal region. For example, a physician can use a mapping diagram created by a visual marker to navigate to a previously visited portion of an internal region or to determine when an instrument is navigated to a new (i.e., previously unvisited) portion of an internal region.
[0116] The methods and techniques may also be configured to receive image data from an image sensor positioned on the instrument. The image data may include a still image or a video. The methods and techniques may link the image data with the position data so that an image captured at a particular location may be called up and displayed to the user. In some implementations, the image is called up and displayed when the user selects the position or when the instrument is positioned at a position where a linked image exists. In addition, the methods and techniques may be configured to allow the physician to mark certain features or locations of interest.
[0117] Systems and techniques for mapping and / or navigating internal areas of the body using robotic enabled medical devices can be employed during many medical procedures such as endoscopic procedures or laparoscopic procedures. The systems and techniques can improve conventional techniques by reducing or eliminating one or more complications or challenges associated with conventional techniques as described below. In some embodiments, the techniques described herein can be used for manual medical procedures.
[0118] In many of the examples described herein, systems and techniques for mapping and / or navigating internal regions of the body are described with reference to a ureteroscopic procedure for removing kidney stones from the kidney. However, it should be understood that the systems and techniques can be used for other medical procedures in which a medical device is navigated within an internal region of a patient's body, such as bronchoscopy, gastroscopy, etc.
[0119] A. Introduction .
[0120] At the beginning and end of a ureteroscopic or percutaneous renal stone removal procedure, the physician typically performs a mapping step. This may involve navigating a flexible endoscope (such as a ureteroscope or cystoscope) to the first (cephalad) calyx, visually observing the stone fragments with an imaging device on the endoscope, and then sequentially moving to the next calyx until the bottom-most calyx is reached. The physician uses this systematic approach in an effort to ensure that all areas of the kidney have been observed despite the wide variation in renal morphology.
[0121] There may be several challenges with this mapping procedure. First, such mapping may be time intensive. For example, each time a fragment is found and treated, the physician needs to repeat the mapping process, starting again from the first calyx. Second, such mapping may be radiation intensive. For example, based solely on the endoscopic view, it is often unclear where the endoscope is within the kidney. Therefore, the physician typically takes fluoroscopic images at each calyx, and sometimes selective pyelograms, in an attempt to verify the position of the endoscope within the kidney. Third, such mapping may be prone to error. The rate of residual fragments indicates that physicians often miss calyces during this mapping procedure.
[0122] Some embodiments discussed herein may relate to systems and techniques for mapping and / or navigating internal regions of the body (e.g., kidneys or other regions) using robotic-enabled medical instruments (e.g., endoscopes, ureteroscopes, cystoscopes, etc.), which may improve the above-described mapping techniques and / or reduce or eliminate one or more of the associated challenges. For example, in some specific implementations, the systems and techniques for mapping and / or navigation discussed herein may increase a physician's confidence that she has navigated an instrument through each calyx of the kidney while reducing the amount of radiation and the procedure time required for mapping.
[0123] In some implementations, systems and techniques for mapping and / or navigation can provide the physician with continuous tracking of the position of the instrument and a mapping-like environment of the anatomical structure. In addition, in some implementations, marking of images captured by the endoscope and imaging device at the location of interest further enhances the ability to inform the physician of the current location of the instrument within the anatomical structure.
[0124] For example, a flexible ureteroscope or flexible cystoscope including an EM sensor (or other orientation sensor) built into its distal end (or other places on the mirror) can be inserted through the working channel of an endoscope for navigating the kidney. The EM sensor can provide orientation data indicating the orientation (e.g., position and / or orientation) of the instrument. As will be described in more detail below, in addition to or in place of the EM sensor, other sensors and methods (e.g., shape sensing fibers, impedance tracking, or other forms of positioning) for determining the orientation of the instrument can also be used. Visual markers derived from the orientation information can be displayed to the physician. In some specific implementations, the visual markers are superimposed on the top of a reference image (e.g., preoperative CT image or intraoperative or preoperative retrograde pyelogram) of the anatomical structure. The orientation data can be, for example, registered with one, two, or three points to align the coordinate system of the orientation data with the coordinate system of the reference image.
[0125] The registration of the coordinate system of the position data with the coordinate system of the reference image can be realized in a variety of ways. For example, the mirror can be navigated to one or more features in the anatomical structure that can be identified from the perspective of the mirror (for example, in the image captured by the imaging device on the mirror) and / or in the reference image, such as from both the perspective of the mirror and in the reference image. When positioned at the anatomical feature identified, the position data from the mirror can be used to align the coordinate system of the position data with the coordinate system of the reference image. In some examples, with respect to the kidney, the imaging device on the mirror can be used to navigate the mirror to one or more funnels in the funnel of the kidney. The position data about the funnel position from the EM sensor can be aligned with the funnel position in the reference image to achieve registration. Other methods for image-based registration (that is, the position identified in the image captured by the imaging device on the mirror is related to the position in the reference image) are also possible. As another registration example, when the mirror is positioned in the anatomical structure, a reference image or another image (for example, a CT image) of the anatomical structure can be captured so that the mirror itself appears in the image. The position of the mirror determined via the position data (eg, from the EM data) and the position of the mirror in the image can be used to align the coordinate systems of both the position data and the reference image.Other registration methods are also possible.
[0126] In some implementations, after registration, the position of the instrument as it moves within the internal region may be displayed on the reference image. In some embodiments, the visual markers are displayed as breadcrumbs (e.g., data points, traces, or any other suitable visual indicator) indicating the historical (e.g., previous) position of the instrument. The displayed visual markers may help the physician verify that all calyces of the kidney have been navigated with the instrument. In some implementations, as the physician navigates the instrument through the kidney, the displayed visual markers form a map representing the anatomical structure of the kidney. The physician may use the map to revisit previously visited locations within the kidney and / or determine whether the instrument is currently positioned within a new (i.e., previously unvisited) calyx.
[0127] In some implementations, the visual markers can include a three-dimensional grid structure representing the anatomical structure of the kidney. Thus, as the physician navigates the instrument through the kidney, the visual markers can construct a three-dimensional model of the previously visited anatomical structure of the kidney. In some implementations, the grid is derived from positional information received from the EM sensor and, in some cases, from image data received from an imaging device on the instrument.
[0128] The systems and techniques for mapping and / or navigation described herein may allow for marking locations and / or features of interest as an instrument is navigated through an anatomical structure. For example, while a physician is navigating, she may "mark" points of interest, such as "calyx 1," "upper pole," "large stone," and the like. In some implementations, when a point is marked, an image (e.g., taken with an imaging device on the instrument) is automatically saved along with the orientation of the location where the image was taken (e.g., as determined by an EM sensor). The marked point may be displayed on a reference image, and the system may allow the user to add supplemental data, such as a label or written description. In some implementations, during subsequent navigation (during the same or subsequent procedures), the systems and techniques for mapping and / or navigation described herein may determine when the instrument is positioned near a previously marked point and call up a saved image for reference or allow the user to call up a marked point based on a user selection.
[0129] In addition to being able to create a three-dimensional map of the kidney, systems and techniques for mapping and / or navigation can be used to correlate recorded images or videos (e.g., taken with an on-device imaging device) with their anatomical location and orientation within the kidney. This can allow a physician to understand which calyx they are viewing and / or allow them to call up images from a specific location.
[0130] Superimposing visual markers derived from the position information received from the EM sensor onto a reference image of the anatomical structure can allow the physician to put the data into context. This can allow the physician to visualize the anatomical structure and the position of the instrument within the anatomical structure. This can further allow the physician to identify any areas of the kidney that she may have missed. The markers can further enhance the physician's ability to understand the visual markers, and calling up an image of the marked location can allow the physician to perform a visual confirmation that the marked location is the same as a previously observed location. These systems and techniques can reduce procedure time, radiation exposure, and / or the error rate of missed calyces.
[0131] Currently, conventional ureteroscopy relies on multiple fluoroscopic images to confirm that the ureteroscope has reached each calyx. Typically, the kidney is not filled with contrast during the procedure, which means there are very few anatomical landmarks to confirm that the correct calyx has been entered.
[0132] For example, Figure 16A-16C Three exemplary fluoroscopic images taken during a ureteroscopy procedure are shown. Fig.16A 1 is a first exemplary fluoroscopic image 101a showing an instrument navigated within a kidney 103. As shown in image 101a, the outline of the kidney 103 is visible, but specific internal structures of the kidney 103 are not readily apparent. An instrument 105 navigated within the kidney 103 is visible in a first position. The first position may be a first lumina of the kidney 103. A physician may capture image 101a to verify that the instrument 105 is positioned within the first lumina. Fig. 16B is a second exemplary fluoroscopic image 101b showing the instrument 105 in a second position within the kidney 103. The second position may be a second umbilicus of the kidney 103. The physician may capture the image 101b to verify that the instrument 105 is positioned within the second umbilicus. Fig. 16C FIG. 1 is a third exemplary fluoroscopic image 101c showing the instrument 105 in a third position within the kidney 103. The third position may be a third lamp. The physician may capture image 101c to verify that the instrument 105 is positioned within the third lamp. Figure 16A-16C As shown, during a conventional ureteroscopy procedure, the physician must take multiple fluoroscopic images at different stages of navigation in an attempt to determine the position of the instrument within the kidney 103. As described above, this can be disadvantageous because it requires multiple exposures to radiation, is time consuming, and can be prone to error.
[0133] Fig.17An exemplary representation of a visual marker 110 indicating, representing, or derived from position data received by a position sensor of an instrument navigated within a kidney is shown. In the illustrated example, the visual marker 110 of the position data is displayed as a trace. In some embodiments, the trace may include a series of points connected by lines. The trace may indicate the path that the instrument travels while navigating within the kidney. As shown, the path may form a mapping diagram that may represent the anatomical structure of the internal region. The visual marker 110 and / or the position data may represent continuous or nearly continuous position tracking of the end of the instrument as it travels through the kidney. The visual marker 110 may produce a useful image or mapping diagram, particularly when superimposed on a reference image of the internal region.
[0134] Fig.18 The kidney is shown superimposed on the reference image 115. Fig.17 In this context, the visual markers 110 provide a map of anatomical structures that can be used to assist in navigation. In addition, the physician can mark certain locations of interest. The marking information can also be superimposed on the reference image. For example, in Fig.18 In the example shown, the seven calyces of the kidney are labeled.
[0135] In some implementations, visual markers 110 representing anatomical structures of the internal region may be derived. Fig.18 As shown, an outline 117 of the anatomical structure can be derived from the position data and / or the visual markers 110 and superimposed on the reference image. In the illustrated embodiment, the outline 117 is shown as a two-dimensional representation of the anatomical structure; however, in some implementations, the outline 117 can include a three-dimensional mesh representing the anatomical structure. Thus, the outline 117 can provide a model of the internal area that can assist the physician in navigating during the procedure.
[0136] These and other features and advantages of methods and systems for mapping and navigating internal regions of the body will now be described in more detail with reference to several specific exemplary methods and systems. The described methods and systems are provided by way of example only and are intended to illustrate the principles of the present disclosure. The present disclosure should not be limited to the described examples.
[0137] B. Exemplary Methods and Systems for Mapping and Navigating Internal Body Regions .
[0138] Fig. 20A is a flow chart illustrating an exemplary method 200 for mapping an internal region of a body. The method 200 may be implemented in certain robotic systems, such as Figure 1-Figure 15 and Fig. 22In some implementations, one or more computer devices may be configured to perform method 200. The computer device may be embodied by a processor (or multiple processors) and a computer readable memory, such as one or more of the components discussed above or below. The computer readable memory may store instructions that can be executed by the processor to perform method 200. The instructions may include one or more software modules. By way of example and not limitation, the computer device may be located at Figure 1 The tower 30 shown, Figure 1-Figure 4 The cart shown, Figure 5-Figure 10 The bed shown, etc.
[0139] Method 200 can be performed, for example, when a medical instrument is navigated within an internal region of a body, such as during various medical procedures (e.g., endoscopic and / or laparoscopic procedures). The internal region can be, for example, an organ, a lumen, a network of lumens, and / or a cavity, etc. In some implementations, method 200 can be initiated when an instrument is introduced into the internal region. Method 200 can be triggered automatically (e.g., upon event initiation or detection) or manually (e.g., upon receipt of user input or command).
[0140] Method 200 begins at block 201, where a reference image of an internal region of a body is displayed. The reference image may represent or illustrate an internal region of the body. In some implementations, the reference image may be, for example, one or more of an image captured during a retrograde pyelography procedure, a fluoroscopic image, an ultrasound image, an image captured during a computed tomography (CT) procedure, and an image captured during a magnetic resonance imaging (MRI) procedure, or any other type of medical image.
[0141] In some implementations, the reference image can be captured intraoperatively. For example, the reference image can be captured during a medical procedure implementing method 200. In such cases, method 200 can include a step for capturing the reference image. In some implementations, the reference image can be captured preoperatively. For example, an image of the internal area taken prior to the procedure (e.g., during a previous procedure or during a previous visit to a physician) can be used as a reference image.
[0142] The reference image may be displayed to a physician performing a medical procedure. As previously described, the reference image may provide some indication of the anatomical structure of the internal region. However, in some implementations, the reference image alone may not provide the physician with enough detail to fully understand the anatomical structure. For example, Figure 16A-16C As shown in the exemplary fluoroscopic image of FIG, the fluoroscopic image may only provide a rough outline of the internal area. Figure 16A-16C In the diagram, the general shape of the kidney 103 can be seen, while the specific internal anatomy and layout of the kidney (e.g., poles, calyces, etc.) are not fully visible.
[0143] As will be discussed in more detail below, the reference image can provide context upon which visual indicia derived from position information received by a position sensor on the instrument can be superimposed.
[0144] The method 200 can reduce the amount of radiation exposure during a procedure or eliminate it entirely. For example, in some implementations, the patient is only exposed to radiation once during the procedure, when the reference image is captured. In contrast, during a conventional ureteroscopy procedure, the physician typically takes multiple fluoroscopic images (exposing the patient to multiple radiation doses) throughout the procedure, as discussed above. Figure 16A-16C Exemplary fluoroscopic images are shown, all of which may be captured during a single ureteroscopy procedure. In a conventional ureteroscopy procedure, a physician may rely on successive fluoroscopic images to determine that each calyx has been visited. However, in some implementations of method 200, only a single reference image need be captured, thereby limiting the amount of radiation to which the patient is exposed. Furthermore, in some implementations of method 200, radiation exposure during a procedure may be eliminated entirely, for example, by using reference images previously captured during a previous procedure or physician visit.
[0145] At block 203, method 200 involves moving (eg, navigating) an instrument within an internal region of the body. The instrument may be, for example, the endoscope 13 ( Figure 1 )、ureteroscope32( Figure 3 )、Equipment 34( Figure 4 )、Ureteroscopy56( Figure 8 )、Laparoscopy59( Fig. 9 )、Equipment 70( Fig.13 ) or device 86( Fig.14 ), or any other medical device described herein, such as device 401 (described below) Fig. 22 ). In some implementations, the instrument can be a robotically enabled instrument controlled by a robotically enabled medical system, as described throughout this disclosure. In some embodiments, the instrument can be a manually controlled instrument.
[0146] The instrument may include at least one position sensor. The position sensor may be configured to provide position information about the position (e.g., location and / or orientation) of the instrument. The position sensor may be, for example, an EM sensor as described above. The EM sensor may be configured to provide position information about the position (e.g., location and / or orientation) of the EM sensor within the EM field generated by the EM field generator. In some implementations, the position sensor may be an EM field generator positioned on the instrument. The position of the EM field generator may be determined relative to a plurality of EM sensors positioned outside the patient's body to determine the position of the instrument. In some implementations, other types of position sensors may be used. For example, the position sensor may be a shape sensing fiber (e.g., a fiber optic shape sensor), an impedance tracker, an accelerometer, a gyroscope, an ultrasonic sensor, or any other type of sensor for determining the position of the instrument.
[0147] One or more of the orientation sensors may be positioned on the instrument. For example, one or more of the orientation sensors may be positioned on the distal end of the instrument. In some implementations, one or more of the orientation sensors are not positioned on the instrument. For example, the orientation sensor may be a torque sensor on the proximal end of the instrument or on a motor pack of a robotic arm to which the instrument is attached. Other examples of orientation sensors may include movement data commanded by a robotic-enabled medical system that can be used to model an estimated pose and orientation of the instrument, and / or visual data received from an imaging device on an image that can be analyzed to determine movement and orientation of the instrument.
[0148] In some embodiments of method 200, a physician controls movement of an instrument within an internal region by, for example, providing commands to a robotic-enabled medical system. The physician may control movement of the instrument to navigate the instrument within the internal region. For example, during a ureteroscopic kidney stone removal procedure, the physician may navigate the instrument to the stone to be removed, or, as previously described, the physician may navigate the instrument through each calyx of the kidney to visually inspect each calyx to verify that the stone and stone fragments have been removed. As another example, the physician may navigate the instrument to an area of interest. For example, the physician may navigate the instrument to an area of transitional cell carcinoma to monitor, biopsy, treat, or resect the area. As described below, movement or navigation of the instrument may be assisted by visual markers derived from position information received from a position sensor, which have been superimposed on a reference image.
[0149] At box 205, method 200 involves receiving orientation information from at least one orientation sensor. As described above, the orientation information may indicate the orientation (e.g., position and / or orientation) of the instrument. In some specific implementations, the orientation information may include a three-degree-of-freedom orientation of the instrument. For example, the orientation information may include x, y, and z coordinates representing the orientation of the instrument in a Cartesian coordinate system. Other coordinate systems (e.g., polar coordinate systems) may also be used. In some specific implementations, the orientation information may include a higher degree-of-freedom orientation of the instrument. For example, the orientation information may include a five-degree-of-freedom orientation, which includes, for example, x, y, and z coordinates of the position and an indication of the pitch and yaw angles of the instrument, which indicate the orientation of the instrument. The six-degree-of-freedom orientation of the instrument may also include roll information of the instrument.
[0150] In some implementations, the position information includes a plurality of position data sets. Each position data set may indicate the position (e.g., location and / or orientation) of the instrument during movement of the instrument. In some implementations, the position data sets are continuously generated as the instrument moves through the interior area. Each position data set may indicate the position of the instrument at a particular point in time. Thus, the position data sets may include a log of the historical positions of the instrument.
[0151] At block 207, method 200 involves superimposing visual indicia derived from the position information onto a reference image to characterize the historical position of the instrument. For example, block 207 may involve superimposing visual indicia derived from at least a subset of the position data set onto a reference image to characterize the historical position of the instrument during movement within an internal region of the body. The superimposed visual indicia may form a map that may assist a physician in navigating the internal region. The visual indicia may represent a path that the instrument travels while navigating the internal region. The visual indicia may be displayed to the physician via a user display, superimposed on top of the reference image. Thus, the reference image provides context for understanding and visualizing the displayed visual indicia.
[0152] In some implementations, the visual markers may include points (e.g., breadcrumbs) corresponding to at least a subset of the position data set. These points may be plotted onto the reference image to show the historical position of the instrument. These points may be represented by a variety of visual markers. For example, the visual marker may be represented as any suitable shape or mark, such as a dot, a dashed line, an X, other shapes, etc.
[0153] Various criteria may be used to determine when to superimpose a visual marker onto a reference image (i.e., the frequency of visual marker superimposition). These criteria may include, for example, the distance traveled by the instrument (e.g., the distance traveled since the previous visual marker), the direction of travel, the time elapsed since the previous superimposed visual marker, etc. One skilled in the art will appreciate that these criteria may be varied to determine the frequency with which visual markers are generated and superimposed onto a reference image, as well as the distance between consecutive visual markers.
[0154] In some implementations, a subset of the orientation data set is superimposed on the reference image with a duration frequency. The duration frequency may include the time between superimposed visual markers. For example, the visual marker may be superimposed approximately every 0.05 seconds, that is, superimposed at a time interval of approximately 0.05 seconds. In other examples, the visual marker may be superimposed at a time interval of approximately 0.1 seconds, approximately 0.25 seconds, or approximately 0.5 seconds, as well as other durations shorter and longer than the examples listed. In some embodiments, the duration frequency may vary. For example, a fixed duration is not required between each superimposed visual marker.
[0155] In some implementations, a subset of the orientation data set is superimposed on the reference image at an orientation frequency. The orientation frequency may include the distance between the superimposed visual markers. For example, the visual markers may be superimposed at intervals of about 0.05 mm, about 0.1 mm, about 0.25 mm, about 0.5 mm, about 1.0 mm, or about 2.0 mm, as well as other distances shorter and longer than the listed examples. In some embodiments, the orientation frequency may vary. For example, a fixed distance is not required between each superimposed visual marker.
[0156] In some implementations, the visual markers are substantially continuously superimposed on the reference image to form a continuous trace representing the historical position of the instrument (see, e.g., Fig.17 and Fig.18 ). In some implementations, visual markers superimposed with duration or orientation frequency (e.g., as discrete points) can be connected to show the historical path of movement of the instrument. For example, in some implementations, a line can be used to connect adjacent superimposed visual markers. In another example, a curve can be fitted to the superimposed visual markers.
[0157] As the physician navigates the instrument through the interior region and the visual markers are superimposed on the reference image, the visual markers may begin to form a map. The map may represent the historical position of the instrument. The map may also represent the anatomical structure of the interior region. For example, as the physician navigates into and out of each calyx, the path formed by the visual markers may represent the structure of each calyx. The physician may use the map to determine whether the instrument is currently being navigated to a new (i.e., previously unvisited) calyx or returning to a previously visited calyx.
[0158] In some implementations, the visual marker superimposed on the reference image may include a grid representing the anatomical structure of the internal region. For example, in some cases, a tubular structure may be fit around the orientation data set to form a three-dimensional grid or model representing the anatomical structure. The diameter or other shape of the tubular structure may be determined based on the movement of the instrument within the internal region. In some implementations, the diameter or other shape of the tube is determined based on the orientation data set and images received from an imaging device positioned on the instrument. For example, the method may estimate the diameter of the tube at a given orientation based on images captured at that orientation. Thus, a three-dimensional grid representing the anatomical structure of the internal region may be developed and displayed as a visual marker superimposed on the reference image.
[0159] In some cases, the visual markers may change over time. For example, the visual markers may fade over time so that recently traveled portions are darker than previously traveled portions. As another example, visual markers from different regions of the interior region may be represented as different colors (see, e.g., Fig.17 and Fig.18 ), where different colors are represented by different types of dashed lines. For example, different colors can be used to show the path through each different lamp.
[0160] In some implementations, data may be associated with visual markers. For example, a physician may create a note containing data associated with certain visual markers. A user may input data via a command console and may view the data via a display.
[0161] The visual markers superimposed on the reference image may be displayed to the user intraoperatively (during a procedure).The visual markers superimposed on the reference image may be stored or saved for use during future medical procedures.
[0162] In some implementations, method 200 may include one or more additional steps. For example, method 200 may include method 300 (described below). Fig.21A ) and / or method 320( Fig. 21B ) steps.
[0163] The method 200 may also be operable to mark a location or feature of interest. For example, the method 200 may include receiving and displaying a current image from an imaging device positioned on the instrument. A physician viewing the image may determine that the image contains a location or feature of interest. For example, a physician may determine that the image contains an entrance to a calyx, a pole of a kidney, an area of transitional cell carcinoma, a kidney stone, stone fragments, etc. The physician may mark the location. The physician may enter data associated with the marked location. The marked location and / or the entered data may be superimposed on a reference image (see, e.g., Fig.18 , which shows the seven calyces of the kidney labeled).
[0164] In some implementations, the method 200 can be operated to automatically detect and mark features of interest. For example, images received from an imaging device on the instrument can be processed and analyzed to determine whether they contain any locations or features of the instrument. For example, an automated process can detect stones, stone fragments, or entrances to calyces within the image and automatically mark these locations. As previously described, the marked locations and / or associated data can be superimposed on a reference image.
[0165] In some implementations, the method 200 may be operable to adjust the orientation determined by the orientation sensor to account for physiological movement, and to overlay the adjusted orientation on the reference image. For example, a motion sensor may be positioned on the patient. The motion sensor may detect movement of the patient (e.g., breathing), and adjust the orientation determined by the orientation sensor to account for the movement of the patient.
[0166] Fig. 20B 2 is a flow chart illustrating another exemplary method 220 for mapping an internal region of a body. The method 220 may be implemented in certain robotic systems, such as Figure 1-Figure 15 and Fig. 22 In some implementations, one or more computer devices may be configured to perform method 220. The computer device may be embodied by a processor (or multiple processors) and a computer readable memory, such as one or more of the components discussed above or below. The computer readable memory may store instructions that can be executed by the processor to perform method 220. The instructions may include one or more software modules. By way of example and not limitation, the computer device may be located at Figure 1 The tower 30 shown, Figure 1-Figure 4 The cart shown, Figure 5-Figure 10 The bed shown, etc.
[0167] Method 220 can be performed, for example, when a medical instrument is navigated within an internal region of the body, such as during various medical procedures (e.g., endoscopic and / or laparoscopic procedures). The internal region can be, for example, an organ, a lumen, a network of lumens, and / or a cavity, etc. In some implementations, method 220 can be initiated when an instrument is introduced into the internal region. Method 220 can be triggered automatically (e.g., upon event initiation or detection) or manually (e.g., upon receipt of user input or command).
[0168] The method 220 begins at block 221 where the instrument is moved within the internal region of the body. At block 223, the method 220 involves receiving position information of the instrument from a position sensor. Blocks 221, 223 may be similar to blocks 203, 205 of the method 200 previously described. For the sake of brevity, the features of blocks 221, 223 are not described here again.
[0169] At block 225, method 220 involves displaying visual indicia derived from the position information to characterize the structure of the internal region of the body. In many respects, block 225 is similar to block 207 of method 200. However, in contrast to block 207 of method 200, block 225 does not necessarily involve superimposing the visual indicia onto a reference image. In fact, in some implementations, method 220 may not involve using a reference image at all. Instead, block 225 may involve directly displaying visual indicia derived from the position information to the user (i.e., without reference to any reference image). Examples of such visual indicia are described in Fig.17 , which shows the position information represented as a visual marker 110. Fig.17 As shown, although no reference image is shown, in some cases, visual markers 110 alone may be sufficient to characterize the structure of the internal region of the body. Fig.17 By following the traces of the visual markers 110 in the figure, the physician can gain an understanding of the approximate anatomical structure of the previously navigated region of the internal region. The visual markers 110 displayed at box 225 can be any type of visual markers described herein, including visual markers displayed at discrete points, visual markers connected by lines or provided with lines, grids representing anatomical structures, etc.
[0170] Method 220 can advantageously build a model or map of the internal region of the body as the instrument navigates through the internal region. In some cases, the physician can use the model or map to return to a previously visited location or to determine when the instrument navigates into a new portion of the internal region. In some cases, the more the instrument navigates within the internal region, the clearer the map or model becomes.
[0171] In some implementations, method 220 may include one or more additional blocks. For example, method 220 may include method 300 (described below) Fig.21A ) and / or method 320( Fig. 21B ) in one or more boxes.
[0172] Fig.21A is a flow chart illustrating an exemplary method 300 for navigating an internal region of a body. The method 300 may be implemented in certain robotic systems, such as Figure 1-Figure 15 and Fig. 22 In some implementations, one or more computer devices may be configured to perform method 300. The computer device may be embodied by a processor (or multiple processors) and a computer readable memory, such as one or more of the components discussed above or below. The computer readable memory may store instructions that can be executed by the processor to perform method 300. The instructions may include one or more software modules. By way of example and not limitation, the computer device may be located at Figure 1 The tower 30 shown, Figure 1-Figure 4The cart shown, Figure 5-Figure 10 The bed shown, etc.
[0173] Method 300 can be performed, for example, when a medical instrument is navigated within an internal region of the body, such as during various medical procedures (e.g., endoscopic and / or laparoscopic procedures). The internal region can be, for example, an organ, a lumen, a network of lumens, and / or a cavity, etc. In some implementations, method 300 can be initiated when an instrument is introduced into the internal region. Method 300 can be triggered automatically (e.g., upon event initiation or detection) or manually (e.g., upon receipt of user input or command).
[0174] The method 300 begins at block 301 where an instrument is moved within an internal region of a body. The instrument may be, for example, the endoscope 13 ( Figure 1 )、ureteroscope32( Figure 3 )、Equipment 34( Figure 4 )、Ureteroscopy56( Figure 8 )、Laparoscopy59( Fig. 9 )、Equipment 70( Fig.13 ) or device 86( Fig.14 ), or any other medical device described herein, such as device 401 (described below) Fig. 22 ). In some implementations, the instrument can be a robotically enabled instrument controlled by a robotically enabled medical system, as described throughout this disclosure.
[0175] The instrument may include a position sensor, such as an EM sensor, an EM field generator, a shape sensing fiber (e.g., a fiber optic shape sensor), an impedance tracker, an accelerometer, a gyroscope, an ultrasonic sensor, or any other type of sensor for determining the position of the instrument.
[0176] In some implementations of method 300, the physician controls movement of the instrument within the internal region by, for example, providing commands to the robotically enabled medical system, as described above. The physician can control movement of the instrument to navigate the instrument to a desired location within the internal region, as described above.
[0177] At block 303, method 300 involves receiving position information from at least one position sensor of the instrument during movement of the instrument. The position information may indicate the position (e.g., location and / or orientation) of the instrument as described above. In some implementations, the position information may include a three-degree-of-freedom position of the instrument. In some implementations, the position information may include a higher degree-of-freedom position of the instrument, such as a five-degree-of-freedom position or a six-degree-of-freedom position.
[0178] In some implementations, the position information includes a plurality of position data sets. Each position data set may indicate the position (e.g., location and / or orientation) of the instrument during movement of the instrument. In some implementations, the position data sets are continuously generated as the instrument moves through the interior area. Each position data set may indicate the position of the instrument at a particular point in time. Thus, the position data sets may include a log of the historical positions of the instrument.
[0179] At box 305, method 300 involves receiving image data from an imaging device of the instrument within the internal area during movement of the instrument. The imaging device can be any photosensitive substrate or structure configured to convert energy representing received light into an electrical signal, such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) image sensor. In some examples, the imaging device may include one or more optical fibers. For example, the imaging device may be a fiber optic bundle that is configured to transmit light representing an image from the distal end of the instrument to the image sensor. The image captured by the imaging device can be transmitted to a computer system as a single frame or as a series of consecutive frames (e.g., video) for storage or display.
[0180] The image data may include one or more images or videos captured by the imaging device at one or more locations within the internal area. In some specific implementations, the imaging device is configured to automatically capture image data. For example, the imaging device may be configured to capture images with a duration frequency. The duration frequency may include the time between the captured images. For example, images may be captured at intervals of about 0.05 seconds, about 0.1 seconds, about 0.25 seconds, or about 0.5 seconds, and other durations shorter and longer than the listed examples. In some embodiments, the duration frequency may vary. For example, a fixed duration is not required between each captured image. The imaging device may be configured to capture images with an azimuth frequency. For example, the azimuth frequency may include the distance moved by the instrument between the captured images. For example, images may be captured at intervals of about 0.05mm, about 0.1mm, about 0.25mm, about 0.5mm, about 1.0mm, or about 2.0mm, and other distances shorter and longer than the listed examples. In some embodiments, the azimuth frequency may vary. For example, a fixed distance is not required between each captured image. In some embodiments, images are captured substantially continuously.
[0181] In some implementations, the image is captured after receiving a user command. For example, a physician may choose when to capture the image.
[0182] At block 307, method 300 involves linking at least a subset of one or more images with at least a subset of the orientation data set based on the orientation at which each image was captured as determined by the orientation sensor. For example, for at least a subset of the one or more images, orientation data received at the orientation at which the image was captured may be linked to the image (e.g., saved with the image). The image may be linked to a three-degree-of-freedom orientation (e.g., a point in space). The image may also be linked to orientation information, for example, by linking the image to a higher-order degree-of-freedom orientation (such as the five-degree-of-freedom orientation or the six-degree-of-freedom orientation described previously). At block 307, method 300 may create a database of images associated or linked to the orientation at which the image was captured. As described below, the image may be called from the database based on the associated or linked orientation of the image for display to the physician.
[0183] In some implementations, all captured images are linked with their associated position data sets. In some implementations, only a subset of one or more images are linked with their associated position data sets. The subset can be selected with a duration frequency (e.g., a duration frequency of about 0.1 seconds, about 0.25 seconds, about 0.5 seconds, or about 1.0 seconds) or an position frequency (e.g., a position frequency of about 0.1 mm, about 0.25 mm, about 0.5 mm, or about 1.0 mm).
[0184] At block 309, method 300 involves determining a user input including an orientation selection. In some implementations, determining a user command includes receiving a user command. For example, a user may select an orientation within an interior area. In some implementations, a user selects an orientation from displayed visual markers. In some implementations, a user selects an orientation relative to a reference image.
[0185] At block 311, method 300 involves displaying a link image in the link image corresponding to the orientation selection. For example, images linked to the selected orientation may be retrieved from the database described previously. In some implementations, if there is no link image corresponding to the selected orientation, method 300 retrieves and displays the link image that most closely corresponds to the selected orientation.
[0186] Thus, method 300 may allow a user to view images corresponding to specific locations within the interior region. This may facilitate navigation within the interior region. This may help a physician find and return to a specific location or feature of interest. In certain implementations, a physician may compare a current image captured using an imaging device on the instrument with one or more previously captured images linked to locations within the interior region to facilitate navigation of the instrument.
[0187] Fig. 21Bis a flow chart illustrating another exemplary method 320 for navigating an internal region of a body. The method 320 may be implemented in certain robotic systems, such as Figure 1-Figure 15 and Fig. 22 In some implementations, one or more computer devices may be configured to perform method 320. The computer device may be embodied by a processor (or multiple processors) and a computer readable memory, such as one or more of the components discussed above or below. The computer readable memory may store instructions that can be executed by the processor to perform method 320. The instructions may include one or more software modules. By way of example and not limitation, the computer device may be located at Figure 1 The tower 30 shown, Figure 1-Figure 4 The cart shown, Figure 5-Figure 10 The bed shown, etc.
[0188] Method 320 can be performed, for example, when a medical instrument is navigated within an internal region of the body, such as during various medical procedures (e.g., endoscopic and / or laparoscopic procedures). The internal region can be, for example, an organ, a lumen, a network of lumens, and / or a cavity, etc. In some implementations, method 320 can be initiated when an instrument is introduced into the internal region. Method 320 can be triggered automatically (e.g., upon event initiation or detection) or manually (e.g., upon receipt of user input or command).
[0189] Method 320 begins at block 321 where an instrument is moved within an internal region of the body. The instrument may be, for example, the endoscope 13 ( Figure 1 )、ureteroscope32( Figure 3 )、Equipment 34( Figure 4 )、Ureteroscopy56( Figure 8 )、Laparoscopy59( Fig. 9 )、Equipment 70( Fig.13 ) or device 86( Fig.14 ), or any other medical device described herein, such as device 401 (described below) Fig. 22 ). In some implementations, the instrument can be a robotically enabled instrument controlled by a robotically enabled medical system, as described throughout this disclosure.
[0190] The instrument may include at least one position sensor as described throughout this disclosure. In some specific implementations of method 300, the physician controls the movement of the instrument within the internal area by, for example, providing commands to the robotic-enabled medical system. The physician may control the movement of the instrument to navigate the instrument to a desired position within the internal area.
[0191] At block 323, method 320 involves receiving position information from at least one position sensor of the instrument during movement of the instrument. The position information may indicate the position (e.g., location and / or orientation) of the instrument as described above. In some implementations, the position information may include a three-degree-of-freedom position of the instrument. In some implementations, the position information may include a higher degree-of-freedom position of the instrument, such as a five-degree-of-freedom position or a six-degree-of-freedom position.
[0192] In some implementations, the position information includes a plurality of position data sets. Each position data set may indicate the position (e.g., location and / or orientation) of the instrument during movement of the instrument. In some implementations, the position data sets are continuously generated as the instrument moves through the interior area. Each position data set may indicate the position of the instrument at a particular point in time. Thus, the position data sets may include a log of the historical positions of the instrument.
[0193] At block 325, method 320 involves receiving image data from an imaging device of the instrument within the internal area during movement of the instrument. The image data may include one or more static images of a video of the internal area. Block 325 may be similar to block 305 of method 300. For the sake of brevity, the features of block 325 will not be described here.
[0194] At block 327, method 320 involves linking at least a subset of the one or more images with at least a subset of the position data set based on the position at which each image was captured as determined by the position sensor. For example, for at least a subset of the one or more images, the position data received at the position at which the image was captured may be linked to the image (e.g., saved with the image). Block 327 may be similar to block 307 of method 300. For the sake of brevity, features of block 327 will not be described again herein.
[0195] At block 329, method 320 involves determining, using at least one position sensor, a current position of the instrument, the current position corresponding to a current position data set in the plurality of position data sets. For example, at block 329, method 320 may determine a current position of the instrument within an interior area.
[0196] At block 331, method 320 involves displaying one of the linked images that corresponds to the determined current position. Thus, method 320 involves determining whether any linked images are available for the current determined position of the instrument, and displaying the linked images to the physician is available. This can advantageously allow the physician to verify the position of the instrument by comparing the linked images to the current image captured with the imaging device. If the images match, the physician can conclude that the instrument is in the position to which it has been previously navigated.
[0197] Method 320 may also allow the physician to compare images taken over time. For example, the physician may navigate the instrument to the location of the instrument (e.g., an area of transitional cell carcinoma) during a subsequent procedure. Method 320 may retrieve and display images of the location of interest taken during a previous procedure. The physician may compare the linked image with the current image to determine changes in the location of interest.
[0198] Fig.19 An example of displaying a link image is shown. Fig.19 As shown, display 120 may provide a representation of the anatomical structure. In some embodiments, display 120 shows an image of the anatomical structure (e.g., a fluoroscopic image or other medical image). In some embodiments, display 120 shows a representation of the anatomical structure, such as a computer model. A user may select position 121, as described above with reference to Fig.21A In another embodiment, the position 121 may represent the current position of the instrument within the anatomical structure, for example, as described above with reference to Fig.21B If the link image 123 is available for the selected location, the link image 123 may also be displayed to the user. In the illustrated embodiment, the link image 123 shows a kidney stone 125.
[0199] Fig. 22 is a block diagram showing certain components of an embodiment of a system 400 for mapping and / or navigating an internal region of a body. In the illustrated embodiment, the system 400 includes an instrument 401. The instrument 401 may include an elongated body configured to navigate an internal region of a body. In some embodiments, the elongated body may include a working channel through which one or more tools may be inserted. The elongated body may be articulated. That is, in some embodiments, the shape or posture of the elongated body may be controlled. For example, the elongated body may include one or more pull lines or tendons operable to adjust the shape or posture of the elongated body.
[0200] The instrument 401 also includes a position sensor 402. The position sensor may be configured to provide position information about the position (e.g., location and / or orientation) of the instrument 401. The position sensor 402 may be, for example, an EM sensor as described above. The EM sensor may be configured to provide position information about the position (e.g., location and / or orientation) of the EM sensor within the EM field generated by the EM field generator. In some implementations, the position sensor may be an EM field generator positioned on the instrument. The position of the EM field generator may be determined relative to a plurality of EM sensors positioned outside the patient's body to determine the position of the instrument. In some implementations, other types of position sensors may be used. For example, the position sensor 402 may be a shape sensing fiber (e.g., a fiber optic shape sensor), an impedance tracker, an accelerometer, a gyroscope, an ultrasonic sensor, or any other type of sensor for determining the position of the instrument 401.
[0201] As shown, the position sensor 402 can be positioned on the instrument 401. For example, the position sensor 402 can be positioned on the distal end of the elongated body. In some embodiments, the position sensor 402 is not positioned on the instrument 401, such as a torque sensor on the proximal end of the instrument 401 or on a motor pack of a robotic arm to which the instrument is attached. Other examples of position sensors can include movement data commanded by the robotic-enabled medical system that can be used to model the estimated pose and position of the instrument, and / or visual data received from an imaging device on an image that can be analyzed to determine the movement and position of the instrument.
[0202] The apparatus 401 also includes an imaging device 403. The imaging device 403 can be any photosensitive substrate or structure configured to convert energy representing received light into an electrical signal, such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) image sensor. In some examples, the imaging device 403 can include one or more optical fibers. For example, the imaging device 403 can be a fiber optic bundle configured to transmit light representing an image from the distal end of the apparatus to the image sensor. The image captured by the imaging device 403 can include a still image and / or a video.
[0203] In the illustrated embodiment of the system 400, the instrument 401 is attached to an instrument positioning device 404. The instrument positioning device 404 can be configured to manipulate the instrument 401. For example, the instrument positioning device 404 can be configured to insert or retract the instrument 401 into an internal area of the body and / or control joint movement (i.e., adjust the shape or posture of the instrument 401). For example, in some embodiments, the instrument positioning device 404 includes one or more robotic arms configured to move to advance or retract the instrument 401. The instrument positioning device 404 can also include an instrument device manipulator as described above. The instrument device manipulator can be configured to actuate a pull wire or tendon of the instrument 401 to control the joint movement of the instrument 401.
[0204] System 400 includes a processor 405 and a memory 406. The memory 406 may include instructions for configuring the processor 405 to perform various methods or processes. For example, the memory 406 may include instructions for causing the processor 405 to perform the above method 200 ( Fig. 20A )、Method 220( Fig. 20B )、Method 300( Fig.21A ) and / or 320( Fig. 22 B) instructions.
[0205] As shown, system 400 also includes a medical imaging device 407. Medical imaging device 407 can be configured to capture reference images of internal areas of the body. Medical imaging device 407 can be any type of medical imager, including, for example, an X-ray machine, an ultrasound, a CT scanner, or an MRI machine. Medical imaging device 407 can be connected to processor 405 and memory 406 to provide reference images for use, as described above.
[0206] The system 400 also includes a data storage device 408. The data storage device 408 can be a memory for storing information, such as a hard drive, a solid state drive, a flash drive, etc. In some specific implementations, the data storage device 408 can be configured to store the position information received from the position sensor 402 and the image data received from the imaging device 403. In some specific implementations, the data storage device 408 stores the image data in a manner linked to the position data, so that the position at which each image was captured is linked to the image. The data storage device 408 can also be configured to store visual markers generated when the processor 405 executes the methods 200 and 220.
[0207] System 400 also includes a display 409. Display 409 may include, for example, an electronic monitor (e.g., a liquid crystal display (LCD) display, an LED display, or a touch-sensitive display), a virtual reality viewing device (e.g., goggles or glasses), and / or other display devices.
[0208] The display 409 may be configured to display various information to the physician during the procedure. For example, the display 409 may display a reference image and visual markers superimposed thereon. As another example, the display 409 may be configured to display one or more marker images associated with the user's selection of an orientation.
[0209] 3. Implement systems and terminology .
[0210] The disclosed embodiments provide systems, methods, and apparatus for mapping and / or navigating internal regions of the body using robotically enabled medical devices.
[0211] It should be noted that as used herein, the terms "couple", "coupling", "coupled" or other variations of the word coupled may indicate an indirect connection or a direct connection. For example, if a first component is "coupled" to a second component, the first component may be indirectly connected to the second component via another component or directly connected to the second component.
[0212] The mapping and navigation functions described herein may be stored as one or more instructions on a processor-readable or computer-readable medium. The term "computer-readable medium" refers to any available medium that can be accessed by a computer or processor. By way of example and not limitation, such a medium may include a random access memory (RAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a compact disk read-only memory (CD-ROM) or other optical disk storage device, a magnetic disk storage device or other magnetic storage device, or any other medium that can be used to store the desired program code in the form of an instruction or data structure and can be accessed by a computer. It should be noted that a computer-readable medium may be tangible and non-transitory. As used herein, the term "code" may refer to software, instructions, code or data that can be executed by a computing device or processor.
[0213] The method disclosed herein includes one or more steps or actions for implementing the method. The method steps and / or actions may be interchangeable with each other without departing from the scope of the claims. In other words, unless the correct operation of the method being described requires a specific order of steps or actions, the order and / or use of specific steps and / or actions may be modified without departing from the scope of the claims.
[0214] As used herein, the term "plurality" means two or more. For example, a plurality of components indicates two or more components. The term "determine" encompasses a variety of actions, and thus, "determine" may include calculating, computing, processing, deriving, investigating, searching (e.g., searching in a table, a database, or another data structure), ascertaining, etc. Additionally, "determine" may include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory), etc. Additionally, "determine" may include parsing, selecting, choosing, establishing, etc.
[0215] Unless explicitly stated otherwise, the phrase “based on” does not mean “based only on.” In other words, the phrase “based on” describes both “based only on” and “based at least on.”
[0216] The foregoing specific embodiments of the disclosed specific implementations are provided to enable any person skilled in the art to make or use the present invention. Various modifications to these specific implementations will be apparent to those skilled in the art, and the general principles defined herein may be applied to other specific implementations without departing from the scope of the present invention. For example, it should be understood that one of ordinary skill in the art will be able to adopt a plurality of corresponding alternatives and equivalent structural details, such as equivalent ways of fastening, mounting, coupling or engaging tool parts, equivalent mechanisms for producing specific actuating motions, and equivalent mechanisms for delivering electrical energy. Therefore, the present invention is not intended to be limited to the specific implementations shown herein, but is given the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A non-transitory computer readable storage medium having stored thereon instructions that, when executed by a processor of a robotic surgical system, cause the processor of the robotic surgical system to at least: moving an instrument of the robotic surgical system within the kidney; receiving position information from at least one position sensor of the instrument during movement of the instrument, the position information comprising a plurality of position data sets, each position data set indicating the position of the instrument at a particular point in time; displaying a reference image of the kidney on a display of the robotic surgical system; causing visual indicia derived from at least a subset of the position data set to be superimposed on the reference image to characterize a historical position of the instrument during the movement of the instrument within the kidney, the visual indicia varying over time; receiving image data from an imaging device of the robotic surgical system positioned on the instrument, the image data comprising a plurality of images captured by the imaging device during movement of the instrument; for at least a subset of the plurality of images, linking each image of the subset with a position data set indicating a position at which the image was captured; as well as A user input of an orientation selection is received, and a linked image of a subset of the plurality of images corresponding to the user input is displayed. 2 . The non-transitory computer-readable storage medium of claim 1 , wherein the reference image comprises an image captured during a retrograde pyelography procedure. 3 . The non-transitory computer-readable storage medium of claim 1 , wherein the reference image comprises a fluoroscopic or ultrasound image.
4. The non-transitory computer-readable storage medium of claim 1, wherein the reference image is captured during a computed tomography (CT) or magnetic resonance imaging (MRI) procedure.
5. The non-transitory computer-readable storage medium of claim 1, wherein the instructions, when executed by a processor of the robotic surgical system, further cause the processor of the robotic surgical system to capture the reference image intraoperatively.
6. The non-transitory computer-readable storage medium of claim 1, wherein the reference image is captured pre-operatively.
7. The non-transitory computer readable storage medium of claim 1, wherein the instructions, when executed by a processor of the robotic surgical system, further cause the processor of the robotic surgical system to store the linked image for use during a future procedure.
8. The non-transitory computer readable storage medium of claim 1, wherein the instructions, when executed by a processor of the robotic surgical system, further cause the processor of the robotic surgical system to: determining a current position of the device using the position sensor; and A linked image corresponding to the determined current position is displayed.
9. The non-transitory computer-readable storage medium of claim 1, wherein the instrument image data is captured automatically.
10. The non-transitory computer-readable storage medium of claim 1, wherein the instrument image data is captured upon receipt of a user command.
11. The non-transitory computer readable storage medium of claim 1 , wherein the instructions, when executed by a processor of the robotic surgical system, further cause the processor of the robotic surgical system to mark a location or feature of interest within the kidney.
12. The non-transitory computer-readable storage medium of claim 11, wherein marking comprises receiving user input.
13. The non-transitory computer-readable storage medium of claim 11, wherein marking comprises automatically detecting the location or feature of interest.
14. The non-transitory computer-readable storage medium of claim 11, wherein the instructions, when executed by a processor of the robotic surgical system, further cause the processor of the robotic surgical system to superimpose the marked location or feature of interest on the reference image.
15. The non-transitory computer readable storage medium of claim 1, wherein the instructions, when executed by a processor of the robotic surgical system, further cause the processor of the robotic surgical system to connect the visual marker to represent a historical path of movement of the instrument.
16. The non-transitory computer-readable storage medium of claim 1, wherein the visual indicia comprises a grid.
17. The non-transitory computer readable storage medium of claim 16, wherein the grid indicates an anatomical structure of the kidney.
18. The non-transitory computer readable storage medium of claim 17, wherein the grid is derived from the subset of the position data set and image data received from an imaging device on the instrument.
19. The non-transitory computer-readable storage medium of claim 1, wherein the subset of the position data set is superimposed on the reference image at a duration frequency.
20. The non-transitory computer-readable storage medium of claim 1, wherein the subset of the orientation data set is superimposed on the reference image at an orientation frequency.
21. The non-transitory computer readable storage medium of claim 1, wherein the instructions, when executed by a processor of the robotic surgical system, further cause the processor of the robotic surgical system to display the visual marker intraoperatively.
22. The non-transitory computer readable storage medium of claim 1, wherein the instructions, when executed by a processor of the robotic surgical system, further cause the processor of the robotic surgical system to store the visual marker for use during a future medical procedure.
23. The non-transitory computer readable storage medium of claim 1, wherein the instructions, when executed by a processor of the robotic surgical system, further cause the processor of the robotic surgical system to: moving the instrument into the calyx of the kidney; and At least one of the following is marked: the entrance of the calyx of the kidney, the pole of the kidney, stones within the kidney, and areas of transitional cell carcinoma.
24. The non-transitory computer readable storage medium of claim 1, wherein the instructions, when executed by a processor of the robotic surgical system, further cause the processor of the robotic surgical system to: adjusting the orientation determined by the orientation sensor to account for physiological movement; and The adjusted orientation is superimposed on the reference image.
25. A robotic surgical system, include: An apparatus having an elongated body and at least one position sensor disposed on the elongated body; monitor; an imaging device positioned on the instrument; The non-transitory computer readable storage medium according to any one of claims 1 to 24; as well as One or more processors in communication with the non-transitory computer-readable storage medium and configured to execute instructions on the non-transitory computer-readable storage medium.
26. The system of claim 25, wherein the position sensor comprises an electromagnetic sensor.
27. The system of claim 25, wherein the position sensor comprises a shape sensing fiber.
28. The system of claim 25, wherein the position sensor is positioned on a distal end of the elongated body.
29. The system of claim 25, wherein the instrument comprises an endoscope.
30. The system of claim 25, wherein the instrument comprises a ureteroscope.
31. The system of claim 25, wherein the elongated body is capable of articulation to control a posture of the device.
32. The system of claim 25, further comprising an instrument positioning device coupled to the instrument, the instrument positioning device configured to manipulate the instrument.
33. The system of claim 32, wherein the instrument positioning device comprises a robotic arm.
34. A non-transitory computer readable storage medium having stored thereon instructions that, when executed by a processor of a robotic surgical system, cause the processor of the robotic surgical system to at least: moving an instrument of the robotic surgical system within the kidney; receiving position information from at least one position sensor of the instrument during movement of the instrument, the position information comprising a plurality of position data sets, each position data set indicating the position of the instrument at a particular point in time during movement of the instrument; receiving image data from an imaging device of the instrument within the kidney during movement of the instrument, the image data comprising one or more images captured by the imaging device at one or more locations within the kidney; linking at least a subset of the one or more images with at least a subset of the position data set based on the position at each image capture determined by the position sensor; causing visual indicia derived from at least a subset of the position data set to be superimposed on an image to characterize a historical position of the instrument during the movement of the instrument within the kidney, the visual indicia varying over time; determining user input including a location selection; as well as A linked image corresponding to the position selection among the linked images is displayed.
35. The non-transitory computer-readable storage medium of claim 34, wherein determining the user input comprises receiving the user input.
36. The non-transitory computer-readable storage medium of claim 34, wherein the image data comprises a static image.
37. The non-transitory computer-readable storage medium of claim 34, wherein the image data comprises a video.
38. The non-transitory computer-readable storage medium of claim 34, wherein the subset of the position data set is selected with a duration frequency.
39. The non-transitory computer-readable storage medium of claim 34, wherein the subset of the orientation data set is selected by orientation frequency.
40. The non-transitory computer-readable storage medium of claim 34, wherein the position information comprises information indicating an orientation of the instrument.
41. The non-transitory computer-readable storage medium of claim 34, wherein the image data is captured automatically.
42. The non-transitory computer-readable storage medium of claim 34, wherein the image data is captured upon receipt of a user command.
43. The non-transitory computer readable storage medium of claim 34, wherein the instructions, when executed by a processor of the robotic surgical system, further cause the processor of the robotic surgical system to: receiving user input associated with the current location; and The user input is linked with a linked image corresponding to the current location.
44. The non-transitory computer readable storage medium of claim 43, wherein the instructions, when executed by a processor of the robotic surgical system, further cause the processor of the robotic surgical system to detect a feature of interest within an image of the image data.
45. A robotic surgical system for intrarenal navigation, the system include: An apparatus comprising Lengthen the body, at least one position sensor disposed on the elongated body, and an imaging device, the imaging device being disposed on the elongated body; monitor; The non-transitory computer readable storage medium according to any one of claims 34 to 44; as well as One or more processors in communication with the non-transitory computer-readable storage medium and configured to execute instructions on the non-transitory computer-readable storage medium.
46. The system of claim 45, wherein the position sensor comprises an electromagnetic sensor.
47. The system of claim 45, wherein the position sensor comprises a shape sensing fiber.
48. A system according to claim 45, wherein the position sensor is positioned on the distal end of the elongated body.
49. The system of claim 45, wherein the instrument comprises an endoscope.
50. The system of claim 45, wherein the instrument comprises a ureteroscope.
51. The system of claim 45, wherein the elongated body is capable of articulation to control a posture of the device.
52. The system of claim 45, further comprising an instrument positioning device coupled to the instrument, the instrument positioning device configured to manipulate the instrument.
53. The system of claim 52, wherein the instrument positioning device comprises a robotic arm.
