Workflow for an ultrasound guided percutaneous needle robot

Abstract

Systems and methods are described for ultrasound guided robotic percutaneous instrument insertion into a subject. The method may include (i) obtaining images of a target volume of a subject, the images obtained by a sensing probe physically positioned by a probe holder that is coupled to a repositionable structure; (ii) reconstructing a three-dimensional (3D) representation of the target volume from the images of the target volume; (iii) segmenting the 3D representation of the target volume to identify one or more objects, boundaries, or features in the 3D representation of the target volume; (iv) determining an instrument insertion pathway including an insertion site and a treatment site; and (v) controlling the repositionable structure and an instrument holder to advance an instrument along the insertion pathway.

Description

WORKFLOW FOR AN ULTRASOUND GUIDED PERCUTANEOUS NEEDLE ROBOTCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to and the benefit of the filing date of U.S. Provisional Patent Application No. 63 / 580,860 entitled “WORKFLOW FOR AN ULTRASOUND GUIDED PERCUTANEOUS NEEDLE ROBOT,” filed on September 6, 2023 and U.S. Provisional Patent Application No. 63 / 580,868 entitled “ROBOTIC SURGICAL MANIPULATOR SYSTEM,” filed on September 6, 2023. The entire contents of each of which are hereby expressly incorporated herein by reference.FIELD

[0002] Disclosed examples relate to guided robotic control. In particular, the disclosed examples relate to systems and methods for analyzing image data and determining insertion paths for medical instruments to reach a treatment region of a subject.BACKGROUND

[0003] Minimally invasive medical techniques are intended to reduce the amount of tissue that is damaged during medical procedures, thereby reducing patient recovery time, discomfort, and harmful side effects. Such minimally invasive techniques may be performed through natural orifices in a patient anatomy or through one or more surgical incisions. Through these natural orifices or incisions, physicians may insert minimally invasive medical instruments (including surgical, diagnostic, therapeutic, and / or biopsy instruments) to reach a target tissue location. Various medical instruments may be used to reach the target volume including needles and probes of various lengths, sizes, and flexibilities. For example, some medical instruments may be rigid and only be able to traverse a linear insertion path, while other instruments may be able to bend to reach a treatment volume along one or more curves of an insertion path. Additionally, the path may be altered or changed during insertion due to the movement of the subject, or to compensate for other errors.

[0004] Image data of the subject is used to plan a pathway towards the region of interest as well as to ensure proper navigation during the procedure. As one example, the image data may be analyzed to segment the traversable vessels from other objects in the image, which may harm the patient if ruptured by the invasive medical instrument. Ultrasound imaging is a tool often used forperforming imaging to plan insertion pathways for instruments. Ultrasound imaging allows a clinician to visualize organs and tissues in a subject and to identify areas of interest which may include abnormalities or target organs or tissues. These types of imaging techniques have been implemented in planning for performing biopsies, and for drug delivery to a patient.

[0005] However, performing ultrasound imaging, and further conducting insertion path planning is a time consuming process that is cumbersome, non-ergonomic, and taxing on a clinician or operator. For example, to perform ultrasound imaging during instrument insertion, a, operator must observe an ultrasound image on a screen, while controlling the insertion path of the medical instrument (e.g., a needle, probe etc.) with one hand, and applying an ultrasound probe to a region of a patient with the other hand. This often results in slow insertion of the instrument as the probe and motion of the insertion device are constantly manually updated to ensure the instrument traverses a desired path. Additionally, the accuracy of the insertion and imaging depends greatly on the ability and skill of an individual operator.

[0006] As such, there is need for more accurate methods and systems for determining a medical instrument insertion path, for further, for actively observing the insertion path during instrument insertion.SUMMARY

[0007] The following presents a simplified summary of various examples described herein and is not intended to identify key or critical elements or to delineate the scope of the claims.

[0008] In some examples, a computer-assisted system is provided. The system may include (i) a repositionable structure, (ii) a probe holder coupled to the repositionable structure, (iii) the probe holder supporting a sensing probe, (iv) an instrument holder coupled to the repositionable structure, the instrument holder supporting an instrument and including an instrument driver configured to control the instrument, and (v) a control system including one or more processors, where in the control system is operably coupled to the repositionable structure, the probe holder, and the instrument holder. The control system is configured to (1) obtain, by the sensing probe, images of a target volume of a subject;, (2) reconstruct a three-dimensional (3D) representation of the target volume from the images of the target volume, (3) segment the 3D representation of the target volume to identify one or more objects, boundaries, or features in the 3D representation, (4) determine an instrument insertion pathway including an insertion site and a treatment site, (5)control the repositionable structure and the instrument holder to advance the instrument along the insertion pathway.

[0009] In further examples, a method for performing ultrasound guided robotic percutaneous instrument insertion is provided. The method includes (1) obtaining, by a sensing probe, images of a target volume of a subject, (2) reconstructing, by a control system, a three-dimensional (3D) representation of the target volume from the images of the target volume, (3) segmenting, by the control system, the 3D representation of the target volume to identify one or more objects, boundaries, or features in the 3D representation of the target volume, (4) determining, by the control system, an instrument insertion pathway including an insertion site and a treatment site, (5) controlling, by the control system, a repositionable structure and an instrument holder to advance an instrument along the insertion pathway, wherein the position of the sensing probe is supported by a probe holder that is coupled to the repositionable structure.

[0010] In still further examples, a non-transitory computer- readable medium storing instructions thereon is provided. The instructions, when executed by one or more processors, cause the one or more processors to (1) obtain, by a sensing probe, images of a target volume of a subject, (2) reconstruct, by a control system, a three-dimensional (3D) representation of the target volume from the images of the target volume, (3) segment, by the control system, the 3D representation of the target volume to identify one or more objects, boundaries, or features in the 3D representation of the target volume, (4) determine, by the control system, an instrument insertion pathway including an insertion site and a treatment site, (5) control, by the control system, a repositionable structure and an instrument holder to advance an instrument along the insertion pathway, wherein the position of the sensing probe is supported by a probe holder that is coupled to the repositionable structure.

[0011] It is to be understood that both the foregoing general description and the following detailed description are illustrative and explanatory in nature and are intended to provide an understanding of the present disclosure without limiting the scope of the present disclosure. In that regard, additional aspects, features, and advantages of the present disclosure will be apparent to one skilled in the art from the following detailed description.BRIEF DESCRIPTIONS OF THE DRAWINGS

[0012] FIG. 1 is a schematic diagram for a robotically-assisted manipulator system according to some examples.

[0013] FIG. 2 A is a perspective view of a manipulator system according to some examples.

[0014] FIG. 2B is an alternative perspective view of the manipulator system of FIG. 2A according to some examples.

[0015] FIG. 2C is an alternative perspective view of the manipulator system of FIG. 2A according to some examples.

[0016] FIG. 3 is a perspective view of an instrument holder for a manipulator system according to some examples.

[0017] FIG. 4 is a diagrammatic view of high and low frequency vibration mechanisms for an instrument driver of an instrument holder according to some examples.

[0018] FIG. 5 is a schematic view of a manipulator system according to some examples.

[0019] FIG. 6 is an example flow diagram of an example method for performing ultrasound guided robotic percutaneous instrument insertion, according to some examples.

[0020] FIG. 7 is a diagram illustrating an implementation of performing ultrasound guided robotic percutaneous instrument insertion into a target region, according to some examples.

[0021] FIG. 8 is a diagram of a target region with multiple insertion pathways for performing ultrasound guided robotic percutaneous instrument insertion, according to some examples.

[0022] FIG. 9 is an example flow diagram of an example method for performing multi-stage treatment via ultrasound guided robotic percutaneous instrument insertion, according to some examples.

[0023] Examples of the present disclosure and their advantages are best understood by referring to the detailed description that follows. It should be appreciated that like reference numerals are used to identify like elements illustrated in one or more of the figures, wherein showings therein arc for purposes of illustrating examples of the present disclosure and not for purposes of limiting the same.DETAILED DESCRIPTION

[0024] In the following description, specific details are set forth describing some examples consistent with the present disclosure. Numerous specific details arc set forth in order to provide a thorough understanding of the examples. It will be apparent, however, to one skilled in the art that some examples may be practiced without some or all of these specific details. The specific examples disclosed herein are meant to be illustrative but not limiting. One skilled in the art may realize other elements that, although not specifically described here, are within the scope and the spirit of this disclosure. In addition, to avoid unnecessary repetition, one or more features shown and described in association with one example may be incorporated into other examples unless specifically described otherwise or if the one or more features would make an example nonfunctional. In some instances, well known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the examples.

[0025] This disclosure describes various instruments and portions of instruments in terms of their state in three-dimensional space. As used herein, the term “position” refers to the location of an object or a portion of an object in a three-dimensional space (e.g., three degrees of translational freedom along Cartesian x-, y-, and z-coordinates). As used herein, the term “orientation” refers to the rotational placement of an object or a portion of an object (e.g., one or more degrees of rotational freedom such as, roll, pitch, and yaw). As used herein, the term “pose” refers to the position of an object or a portion of an object in at least one degree of translational freedom and to the orientation of that object or portion of the object in at least one degree of rotational freedom (e.g., up to six total degrees of freedom). As used herein, the term “shape” refers to a set of poses, positions, and / or orientations measured along an object. As used herein, the term “distal” refers to a position that is closer to a procedural site and the term “proximal” refers to a position that is further from the procedural site. Accordingly, the distal portion or distal end of an instrument is closer to a procedural site than a proximal portion or proximal end of the instrument when the instrument is being used as designed to perform a procedure.

[0026] This disclosure occasionally refers to the disclosed techniques being applied to “patients” undergoing a “medical procedure.” It should be appreciated that these references are not intended to limit the application of the disclosed techniques to applied medicine contexts. For example, the described techniques can be applied to facilitate physician training, equipment testing and / or calibration, and / or other contexts. Accordingly, any reference to the term “patient” is donefor ease of explanation and also envisions the application of the described techniques to a generic “subject.”

[0027] This disclosure describes various types of image data. In some embodiments, the image data includes 2D and / or 3D images. While the term “pixel” is used herein to refer to a particular point within the image data, the term “pixel” should not be understood to imply that the corresponding image data is a 2D image. In embodiments where the image data is 3D image data, the term “pixel” should be understood to refer to a “voxel” of the 3D image.

[0028] Systems and methods are provided utilizing a manipulator arm and device holders movable about several degrees of freedom of motion, allowing complex positioning and control. In some examples, the systems and methods control needle insertion and track movement of the needle along an insertion path, such as with an ultrasound probe. In these examples, the manipulator arm may have at least four degrees of freedom (e.g., three pivots about vertical axes and a distal pivot about a horizontal axis), while holders for the needle and ultrasound probe may have at least three degrees of freedom (e.g., x-axis, z-axis, pitch, and / or yaw).

[0029] In further or additional examples, the systems and methods control movement of the manipulator and an instrument holder according to a remote center around a needle insertion point at the skin level. In these examples, a control system determines an insertion point at the skin and direction vector for needle insertion and controls movement of the manipulator arm and instrument holder to position an instrument driver to align an insertion axis thereof with the insertion point along the direction vector. The control system can then operate the instrument driver to drive the needle along the insertion axis to the insertion point and control movement of the manipulator arm and instrument holder about the insertion point as a remote center in response to the needle reaching the insertion point at the skin level.

[0030] An instrument driver for the needle can be configured to impart two degrees of freedom of movement on the needle, including insertion along an insertion axis and rotation about the insertion axis. The systems and methods can further include a stylus that is coupled to and driven by the instrument driver through a lumen of the needle. The instrument driver may also impart two degrees of freedom of movement on the stylus, including insertion along the insertion axis and rotation about the insertion axis.

[0031] In the above or alternative examples, a needle driver for a needle includes both high and low frequency vibration mechanisms coupled to the needle. The high frequency vibration mechanism can be utilized to aid needle insertion through tissue and aid visualization of the needle on ultrasound. In one example, the high frequency vibration mechanism includes a stack of piezoelectric members received within a housing coupled to the needle. The low frequency vibration mechanism can be utilized to decrease pain during needle insertion. In one example, the low frequency vibration mechanism can be an eccentric motor or linear actuator coupled to the needle. The needle driver can be utilized within the overall system as a component of the instrument holder. The system can also include the manipulator arm and probe holder.

[0032] Aspects of this disclosure herein can be part of a computer-assisted teleoperational manipulator system, sometimes referred to as a robotically-assisted manipulator system or a robotic system. The manipulator system can include one or more manipulators that can be operated with the assistance of an electronic controller (e.g., computer) to move and control functions of one or more instruments when coupled to the manipulators.

[0033] FIG. 1 illustrates an embodiment of a robotically-assisted manipulator system for use with the tools described herein. The manipulator system can be used, for example, in surgical, diagnostic, therapeutic, biopsy, or non-medical procedures, and is generally indicated by the reference numeral 100. As shown in FIG. 1, a robotically-assisted manipulator system 100 can include one or more manipulator assemblies 102 for operating one or more medical instrument systems 104 in performing various procedures on a patient P positioned on a table T in a medical environment 101. For example, the manipulator assembly 102 can drive catheter or end effector motion, can apply treatment to target tissue, and / or can manipulate control members. The manipulator assembly 102 can be teleoperated, non-teleoperated, or a hybrid teleoperated and nonteleoperated assembly with select degrees of freedom of motion that can be motorized and / or teleoperated and select degrees of freedom of motion that can be non-motorized and / or non- teleoperated. An operator input system 106, which can be inside or outside of the medical environment 101, generally includes one or more control devices for controlling manipulator assembly 102. Manipulator assembly 102 supports medical instrument system 104 and can optionally include a plurality of actuators or motors that drive inputs on medical instrument system 104 in response to commands from a control system 112. The actuators can optionally include drive systems that when coupled to medical instrument system 104 can advance medicalinstrument system 104 into a naturally or surgically created anatomic orifice. Other drive systems can move the distal end of medical instrument in multiple degrees of freedom, which can include three degrees of linear motion (e.g., linear motion along the X, Y, Z Cartesian axes) and in three degrees of rotational motion (e.g., rotation about the X, Y, Z Cartesian axes). The manipulator assembly 102 can support various other systems for irrigation, treatment, or other purposes. Such systems can include fluid systems (including, for example, reservoirs, heating / cooling elements, pumps, and valves), generators, lasers, interrogators, ultrasound gel reservoirs and / or dispensers, and ablation components.

[0034] Robotically-assisted manipulator system 100 also includes a display system 110 for displaying an image or representation of the surgical site and medical instrument system 104 generated by an imaging system 109 which can include an imaging system, such as an ultrasound imaging system. Display system 110 and operator input system 106 can be oriented so an operator O can control medical instrument system 104 and operator input system 106 with the perception of telepresence. A graphical user interface can be displayable on the display system 110 and / or a display system of an independent planning workstation.

[0035] In some examples, the ultrasound imaging system components of the imaging system 109 can be integrally or removably coupled to medical instrument system 104. However, in some examples, a separate imaging device, such as an endoscope, 3D video, or infrared camera, attached to a separate manipulator assembly can be used with medical instrument system 104 to image the surgical site. The ultrasound imaging system 109 can be implemented as hardware, firmware, software, or a combination thereof which interact with or are otherwise executed by one or more computer processors, which can include the processors of the control system 112.

[0036] Robotically-assisted manipulator system 100 can also include a sensor system 108. The sensor system 108 can include a position / location sensor system (e.g., an actuator encoder or an electromagnetic (EM) sensor system) and / or a shape sensor system (e.g., an optical fiber shape sensor) for determining the position, orientation, speed, velocity, pose, and / or shape of the medical instrument system 104. The sensor system 108 can also include temperature, pressure, force, or contact sensors or the like.

[0037] Robotically-assisted manipulator system 100 can also include the control system 112. Control system 112 includes at least one memory 116 and at least one computer processor 114 foreffecting control between medical instrument system 104, operator input system 106, sensor system 108, and display system 110. Control system 112 also includes programmed instructions (e.g., a non-transitory machine-readable medium storing the instructions) to implement a procedure using the robotically-assisted manipulator system including for navigation, steering, imaging, engagement feature deployment or retraction, applying treatment to target tissue (e.g., via the application of energy), or the like.

[0038] Control system 112 can optionally further include a virtual visualization system to provide navigation assistance to operator O when controlling medical instrument system 104 during an image-guided surgical procedure. Virtual navigation using the virtual visualization system can be based upon reference to an acquired pre-operative or intra-operative dataset of anatomic passageways. The virtual visualization system processes images of the surgical site imaged using imaging technology such as computerized tomography (CT), magnetic resonance imaging (MRI), fluoroscopy, thermography, ultrasound, optical coherence tomography (OCT), thermal imaging, impedance imaging, laser imaging, nanotube X-ray imaging, and / or the like. The control system 112 can use a pre-operative image to locate the target tissue (using vision imaging techniques and / or by receiving user input) and create a pre-operative plan, including a determined first location for performing treatment. The pre-operative plan can include, for example, a planned size to expand an expandable device, a treatment duration, a treatment temperature, and / or multiple deployment locations. Other configurations of teleoperated manipulator systems are also contemplated, such as systems configured for multi-port or single-port procedures. For example, the embodiments described herein can be used with a da Vinci® Surgical System, such as the da Vinci X®, Xi®, or SP® Surgical Systems, all commercialized by Intuitive Surgical, Inc., of Sunnyvale, California. Additionally, the embodiments described herein can be used in combination with real time CT guidance (on a CT table), real-time Cone beam CT and X-ray image guidance (in an interventional angiography room), with an X-ray table and or surgical operating table.

[0039] FIGS. 2A-2C illustrate an example embodiment of a manipulator system 200 that can be used as part of the manipulator system 100. The manipulator system 200 includes a repositionable structure 202 that includes a plurality of links 204 and a plurality of joints 206. The system 200 further includes an instrument holder 208 configured to receive an instrument 210 (e.g., a needle, cannula, etc.) and a probe holder 212 configured to receive a sensing probe 214(e.g., an ultrasound probe) having a field of view. The instrument holder 208 and the probe holder 212 arc coupled to one or more of the links 204. With this configuration, the rcpositionablc structure 202 has a number of mechanical degrees of freedom of motion through which the instrument holder 208 and the probe holder 212 can be positioned and controlled to scan a patient with the sensing probe 214 and insert the instrument 210 into the patient. In some examples, the instrument holder 208 has at least three degrees of freedom of motion and the probe holder 212 has at least three degrees of freedom of motion. In some examples, the repositionable structure 202 has at least four mechanical degrees of freedom of motion.

[0040] A control system 216 (FIG. 5) of the manipulator system 200 is operably coupled to the repositionable structure 202, the instrument holder 208, and the probe holder 212 to control operation of the instrument holder 208 and movement of the repositionable structure 202, the instrument holder 208, and the probe holder 212. In some examples, the control system 216 can correspond to the control system 112 described above.

[0041] The control system 216 is configured to move the instrument 210 along an insertion path and move the sensing probe 214 to keep the instrument 210 within the field of view of the sensing probe 214 as the instrument 210 moves along the insertion path. The repositionable structure 202 is configured such that movement of the repositionable structure 202 moves both the instrument holder 208 and the probe holder 212. As shown, both the instrument holder 208 and the probe holder 212 are coupled to a distal link 204a of the repositionable structure 202, such that when any of the links 204 and joints 206 move, the repositionable structure 202 moves both the instrument 210 and the sensing probe 214 through the instrument holder 208 and the probe holder 212.

[0042] In other words, the control system 216 is configured to operate the instrument holder 208 to move the instrument 210 along the insertion path at least by moving the repositionable structure 202 according to a first set of movements and moving the instrument holder 208 according to a second set of movements. The control system 216 is also configured to move the sensing probe 214 to keep the instrument 210 within the field of view of the sensing probe 214 as the instrument 210 moves along the insertion path at least by moving the repositionable structure 202 according to the first set of movements and moving the probe holder 212 according to a third set of movements.

[0043] In some examples, the repositionable structure 202 includes a support assembly 218 including a base 220 and a support column 222. An arm 224 is connected to the support column 222. The arm 224 and, in some examples, the support column 222, can be used as the manipulator assembly 102. The arm 224 includes the links 204 and joints 206. As shown, the arm 224 can include shoulder 206a, elbow 206b, and wrist 206c joints that are pivotable about vertical axes. The arm 224 can further includes a distal joint 206d that is pivotable about a horizontal axis. As such, the arm 224 has four mechanical degrees of freedom of motion, but additional or fewer links 204 and joints 206 can be utilized to provide a desired number of mechanical degrees of freedom of motion.

[0044] As shown, a drive 226 can be disposed between the support column 222 and the arm 224. The drive 226 can be operated to move the arm 224 along a Z axis (e.g., height) of the support column 222 for an additional degree of freedom of motion for the repositionable structure 202.

[0045] The probe holder 212 includes a drive 228 operable to move the sensing probe 214 along an X axis, a drive 230 to move the sensing probe 214 along a Z axis, and a joint 232 to pivot the sensing probe 214 about a pitch axis. This configuration gives the probe holder 212 three degrees of freedom of motion. In some examples, a pressure sensor 234 can be operably coupled to the probe holder 212 so that the control system 216 can maintain a desired pressure on the patient with the sensing probe 214 using data from the pressure sensor 234. The pressure sensor 234 can be part of the sensing probe 214, part of the probe holder 212, or a separate sensor coupled to the probe holder 212, etc.

[0046] The instrument holder 208 includes a drive 236 operable to move the instrument holder 208 along an X axis, a joint 238 to pivot the instrument holder 208 about a yaw axis, and a joint 240 to pivot the instrument holder 208 about a pitch axis. Furthermore, the instrument holder 208 can include an instrument driver 242 that includes an insertion drive 244 to move the instrument 210 along an insertion axis X and, in some examples, a rotation drive 246 to rotate the instrument 210 about the insertion axis X. This configuration gives the instrument holder 208 four or five degrees of freedom of motion. In some examples, a pressure sensor 248 can be operably coupled to the instrument driver 242 so that the control system 216 can monitor a pressure at the tip of the instrument 210 (e.g., during piercing of the patient’s skin and / or during insertion) to adjust for tissue pressure and collect insertion data. This can be helpful for when the instrument 210transitions from soft tissue to hard tissue (e.g., a tumor nodule) and from hard tissue to soft tissue. The pressure sensor 248 can be a load cell, for example.

[0047] With the above configuration, the instrument holder 208 can move the instrument 210 along the insertion path. Moving the instrument 210 along the insertion path can include both aligning the insertion axis X to drive in and out along the insertion axis X using the insertion drive 244 for straight segments of the insertion path and rotating the instrument 210 using the rotation drive 246 (e.g., when using an instrument with a beveled tip) to subsequently or concurrently cause curving along the insertion path when moving the instrument 210 using the insertion drive 244 for when the insertion path includes non-linear segments.

[0048] As shown, both the instrument holder 208 and the probe holder 212 can be coupled to the distal link 204a of the repositionable structure 202. Furthermore, both of the X axis drives 228, 236 can interact with the distal link 204a to move the probe holder 212 and instrument holder 208, respectively along the distal link 204a. For example, the X axis drives 228, 236 can be rack and pinion drives that utilize a common rack extending along the distal link 204a.

[0049] The instrument holder 208 can include one or more vibration mechanisms to aid in instrument insertion. As such, the instrument holder 208 can include a high frequency vibration mechanism 250 coupled to the instrument 210. Vibrating the instrument 210 at a high frequency can help visualizing the instrument 210 with the sensing probe 214 (e.g., under ultrasound). High frequency vibration can also reduce an insertion force needed for instrument insertion and drag induced tissue distortion. Additionally or alternatively, the instrument holder 208 can include a low frequency vibration mechanism 252 coupled to the instrument 210. Vibrating the instrument 210 at a low frequency can reduce patient pain during instrument insertion by numbing the local area around the insertion path. Vibration of the instrument can also assist with increased visualization of the instrument 210 in ultrasound imaging. Some instruments 210, such as needles, may be coated with a microbubble or other element that increases the visualization and detection of the instrument 210 in an ultrasound image.

[0050] Additionally, vibration of the instrument 210 combined with rotation of the instrument 210 via the rotation drive 246 can help the instrument 210 penetrate a free floating / moving object (e.g., a lymph node or similar structure not fixed inside tissue) rather than undesirably pushing the object away as can happen with conventional systems.

[0051] In some examples, the frequency range for the high frequency vibration mechanism 250 can be about 27 kHz to about 44 kHz, which can be advantageous for visibility under ultrasound vision, between about 20 kHz to about 50 kHz, which can be advantageous to reduce insertion forces, or about 34 kHz; and the frequency range for the low frequency vibration mechanism 252 can be about 75 Hz to about 250 Hz, which can be advantageous for pain reduction, or about 166 Hz. The vibration mechanisms 250, 252 can be part of the instrument holder 208, coupled to the instrument holder 208, or be a separate component directly coupled to the instrument 210 separate from the instrument holder 208.

[0052] In one example, the high frequency vibration mechanism 250 can include a plurality of piezoelectric members 254 received within a housing 256 coupled to the instrument 210. If desired, the housing 256 can be coupled to a proximal mount 258 of the instrument 210 with a radial coupling such that the piezoelectric members 254 are offset relative to insertion axis X and instrument 210.

[0053] In one example, the instrument holder 208 can include a distal instrument guide 260 that supports the instrument 210 adjacent to an insertion location. The instrument guide 260 can have any suitable form. For example, the instrument guide 260 can be side or top loaded and define a path for the instrument 210 to be inserted therethrough. The low frequency vibration mechanism 252 can be coupled to the instrument guide 260 to vibrate the instrument 210 through the instrument guide 260. In some examples, the low frequency vibration mechanism 252 can be an eccentric motor or linear actuator.

[0054] In some embodiments, the instrument 210 can define a lumen that extends longitudinally through a length of the instrument 210. With this configuration, in some examples, the instrument 210 can be utilized to deliver fluid at a treatment location. Additionally or alternatively, the system 200 can include a second instrument 262 (e.g., a stylus) that is inserted through the lumen of the instrument 210. The second instrument 262 can be utilized for any desired reason. In some examples, the second instrument 262 can have a solid cross-section and complementary bevel to provide a solid overall tip between the first and second instruments 210, 262, the second instrument 262 can be a biopsy stylus or needle, the second instrument 262 can be an anchor stylus, and so forth. Additionally, the second instrument 262 can also be a mini camera, an endoscope, an OCT, optical imaging laser and so forth.

[0055] Pursuant to the above, the instrument driver 242 can further include an insertion drive 264 to move the second instrument 262 along the insertion axis X within the lumen of the first instrument 210 and, in some examples, a rotation drive 266 to rotate the second instrument 262 about the insertion axis X. This configuration gives the instrument holder 208 an additional one or two degrees of freedom of motion. In one example, the insertion drives 244, 264 can be rack and pinion drives that utilize a common rack.

[0056] If desired, a high frequency vibration mechanism 268 can be coupled to the second instrument 210 to provide similar benefits as described above. For example, the high frequency vibration mechanism 268 can be coupled to a proximal mount 270 of the second instrument 262. The high frequency vibration mechanism 268 can be any suitable mechanism, including a plurality of piezoelectric members as described above, an ultrasonic transducer, and so forth.

[0057] As shown in FIG. 5, the control system 216 can be configured to control the movement and operation of the components of the system 100. These include the support assembly drive 226, the joints 206 of the arm 224, the drives and joints 228, 230, 232 of the prob holder 212, the drives and joints 236, 238, 240, 244, 246, 264, 266, and the vibration mechanisms 250, 252, 268, if included.

[0058] In some examples, the control system 216 can be configured to control the movement of the repositionable structure 202 (e.g., drive 226 and joints 206) and the instrument holder 208 (e.g., drives 236 and joints 238, 240) to selectively move and operate about a remote center during a procedure. For example, the remote center can correspond to an insertion point at the skin of the instrument 210 into a patient along the insertion path. This configuration allows the system 100 to move as needed during a procedure to insert the instrument 210 along the insertion path while minimizing or avoiding damage at the insertion point at the skin level. In one example, the control system 216 can be configured to control movement of the repositionable structure 202 and instrument holder 208 about the remote center to deflect the insertion path, such as by rotating the instrument 210 with the rotation drive 246.

[0059] Pursuant to the above, the control system 216 can be configured to control movement of the repositionable structure 202 and the instrument holder 208 about the remote center corresponding to the instrument insertion point, at the skin, into the patient. Movement about the remote center imposes a set of constraints on at least one of: one or more of the degrees of freedomof motion of the repositionable structure 202 or one or more degrees of freedom of motion of the instrument holder 208.

[0060] Furthermore, the control system 216 can be configured to control movement of the repositionable structure 202 and the probe holder 212 to perform an initial scan of a target region within the patient with the sensing probe 214. The initial scan can be utilized to determine the insertion path. For example, the control system 216 can utilize the initial scan to determine the instrument insertion point and a direction vector for instrument insertion.

[0061] To allow the system 100 to complete a full scan, the initial scan can be performed without imposing the set of constraints on the set of degrees of freedom of motion of the repositionable structure 202 or of the instrument holder 208 that are imposed while movement is performed around the remote center. Accordingly, the remote center functionality of the system 100 can be turned off for the initial scan. Where the initial scan is an ultrasound scan, the sensing probe 214 should be in physical contact with the patient with a range of pressures between the sensing probe 214 and the patient to transfer the ultrasound imaging energy. To achieve this, the system 100 may utilize the various degrees of freedom to maintain a pressure as measured by the pressure sensor 234 coupled to the instrument driver 242. In implementations, an ultrasound gel may be used to improve the contact between the sensing probe 214 and the patient to assist in providing the imaging energy to the patient.

[0062] Thereafter, the remote center functionality can be activated in response to the instrument 210 reaching the insertion point, after which the control system 216 controls movement of the repositionable structure 202 and instrument holder 208 about the remote center. Reaching the insertion point at the skin can be determined based data from the pressure sensor 234, a length of the instrument 210 that has been driven forward by the insertion drive 244, and so forth. Further, control of movement about the remote center can be maintained until the instrument 210 is withdrawn from the patient.

[0063] In some examples, the control system 216 can control operation of the instrument holder 208 and the probe holder 212 to both move the instrument 210 along the insertion path and move the sensing probe 214 to keep the instrument 210 within the field of view as the instrument 210 moves along the insertion path. This functionality can include the control system 216 controlling movement of the repositionable structure 202 about the remote center and controlling movementof the probe holder 212 to keep the instrument 210 within the field of view as the instrument 210 moves along the insertion path.

[0064] In the example of the sensing probe 214 being an ultrasound probe having a 2D field of view, the control system 216 can be configured to move the sensing probe 214 to maintain a tip of the instrument 210 within the ultrasound 2D slice at all times during insertion.

[0065] In some examples, the system 100 can be configured to insert multiple instruments 210 into a patient by decoupling an inserted instrument and loading a subsequent instrument 210. Pursuant to this, the instrument driver 242 can include one or more slots 272 allowing the instrument 210 to be offloaded from the instrument driver 242. Alternatively, the instrument driver 242 can include a releasable collet allowing the instrument to be decoupled therefrom. Additionally, in examples having the high frequency vibration mechanism, the housing 256 can also include a slot 274 or releasable connector for the radial coupling.

[0066] FIG. 6 is a flow diagram of a method 600 for performing ultrasound guided robotic percutaneous instrument insertion, according to some examples. The method 600 may be performed using the robotically-assisted manipulator system 100 of FIG. 1, the manipulator system 200 of FIG. 2, or by another manipulator system. The method 600 may be performed by one or more processors of a control system (e.g., the control system 216 of FIG. 5) executing instructions stored in one or more computer-readable media (e.g., non-volatile memory), or by another processor. FIGS. 7 and 8 are diagrams of a target volume 700 of a subject 702 that is illustrative of example implementations of the method 600. For clarity, the method 600 will be described with reference to elements of FIGS. 1, 2, 7, and 8.

[0067] At block 602, the system obtains one or more images of the target volume 700. As shown in FIG. 7, the sensing probe 214 is positioned relative to, or against, the subject 702 to obtain images of the target volume 700. The image data may be pre-operative image data or intraoperative image data. In some embodiments, the image data is 3D image data, such as computed tomography (CT) image data, cone-beam CT (CBCT) image data, positron emission tomography (PET) image data, ultrasound image data, or magnetic resonance imaging (MRI) image data. In examples, the sensing probe 214 may include one or more of an ultrasound probe, photoacoustic probe, microscope, optical imaging probe, video camera, infrared camera, handheld x-ray system, electronic palpation device, impulse-radar detector, focused ultrasound probe, gamma probe,perfusion imager, optical coherence tomography device. In an example, the sensing probe 214 includes an ultrasound probe with a ficld-of-vicw (FOV) 715 for capturing images of objects, such as a target 705, in the FOV 715.

[0068] At block 604, the system reconstructs a 3D representation of the target volume 700. The 3D representation of the target volume 700 includes a plurality of voxels and may have voxels in free space not including tissue, voxels that include tissue of the subject 702, and voxels that may include a portion of both free space outside of the subject 702, and tissue of the subject 702. The system 100 constructs the 3D representation using both the FOV of the ultrasound probe, and information and data pertaining to the position and orientation of the sensing probe 214 relative to other FOVs of the ultrasound probe, or relative to other image FOVs and scans. The method further includes, at block 606, segmenting the 3D representation to identify one or more objects, boundaries, or features in the 3D representation of the target volume. Examples of objects and features that may be identified include, without limitation, one or more tissues, organs, fluids, blood vessels, bones, medical structures (e.g., implanted devices, stints, medical instruments, etc.), etc. In implementations, segmenting the 3D representation may further include the processor, or the control system 216, identifying a treatment site 712 of the subject 702.

[0069] At block 608, the control system 216 determines an instrument insertion pathway. The control system 216 determines an insertion site 710 for insertion of an instrument into the subject 702, and further identifies the treatment site 712 either during the segmenting, or from further processing of the 3D representation. The treatment site 712 is a location inside of the subject 702 in which the instrument 210, or at least a portion of the instrument 210 such as a tip of a needle or distal end of the instrument, is to reach to provide a treatment. The treatment may include providing a drug to the treatment site, providing a therapy to the treatment site, performing a biopsy at the treatment site, blood drawing, delivery of electronic leads, microrobots, or microsensors, delivering energy, providing gene therapy, providing a picc line, providing a dialysis fistula, delivery of hydrogel, delivery of an implant, placement of needle electrodes, extracting tissue, delivering tissue, measuring tissue, and / or destroying tissue. The treatment site 712 may include an organ, a tissue, a tumor, a cancer, a blood vessel, a live tumor, the epidermis, a ligament, muscles, sinuses, the brain, pain sources or areas, nerves, lymph vessels, or a portion or subsection of any thereof. In some implementations, the method may include receiving a user identified treatment site from the 3D representation of the target volume. For example, a user may indicatea specified region as the treatment site, an object, boundary, feature, organ etc. as the treatment site 712. The user may provide such an indication via a user interface such as a keyboard, mouse, touch screen, via a network, etc.

[0070] To determine one or more of (i) the instrument insertion pathway, (ii) the insertion site 710, and / or (iii) the treatment site 712, the control system 216 may determine two or more potential insertion pathways and select a determined insertion pathway from the two or more candidate pathways. For example, as illustrated in FIG. 8, the control system may determine a first insertion pathway 720a with a first insertion site 710a, a second insertion pathway 720b with a second insertion site 710b, and a third insertion pathway 720c with a third insertion site 710c. Each of the first, second and third insertion pathways 720a-720c provide a path to the treatment site 712.

[0071] In determining a determined insertion pathway, the control system 216 may select the determined insertion pathway based on a pathway length into the subject 702, or a number of voxels that the pathway traverses in the 3D representation. For example, one insertion pathway may include a voxel with free space, a voxel with both free space and tissue, and three voxels with tissue to reach the treatment site 712. Another second pathway may include no voxels with free space, and four voxels with tissue to reach the treatment site 712. The determined pathway may be determined as the second pathway with a total of four voxels because of the reduced distance of the overall insertion pathway. Conversely, the determined insertion pathway may be determined as the first pathway with a total of five voxel with only three voxels in the tissue of the subject 702, which may be chosen to reduce the distance of the insertion pathway inside of the subject. The determined pathway may be determined by a number of voxels with free space in the 3D representation, or the determined insertion pathway may be determined by a number of voxels in free space versus a number of voxels with tissue.

[0072] The determined insertion pathway and insertion site may be determined based on the position of organs and tissue, and further may be determined to avoid identified objects such as to avoid a bone, blood vessel, organ, tissue region, etc. While illustrated as linear insertion pathways in FIG. 8, the insertion pathway may include one or more curves or arcs (further described with reference to FIG. 7), and the determined pathway may be determined based on an amount of curvature or direction of curvature of the insertion pathway. The control system 216 may determine the determined pathway based on an instrument type of the instrument (e.g., a needle, anendoscope, an implant, a catheter, a picc line, a stylus) or a needle type for inside which the instrument is inserted.

[0073] A user may further provide user preferences that the determined pathway may further be determined from. For example, a user may select a determined insertion pathway from the plurality of candidate insertion pathways. A user may provide specific tissues, organs, or regions inside of the patient to avoid during insertion of the instrument. Similarly, the user may define one or more restricted regions of the 3D representation to exclude any insertion pathway from traversing the restricted regions. The user preferences may include a maximum or minimum distance for an insertion pathway into the subject 702, a maximum or minimum curvature of the insertion pathway, a preference for a straight path, a user identified determined insertion site or region for one or more insertion sites, a restriction of an insertion site or restricted region that forbids insertion sites, and / or a user defined treatment site 712.

[0074] In determining an insertion pathway, or the two or more candidate insertion pathways, the control system may further determine a restricted region in the 3D representation. The restricted region indicates a region of the 3D representation, and further, of the subject 702, that is not to be traversed by either the determined insertion pathway and / or the two or more candidate insertion pathways. The restricted region is a 3D volume and the control system 216 may determine the a candidate insertion pathways as paths that do not intersect or traverse any of the volume of the restricted region. The processor may determine the restricted region automatically from the one or more objects, boundaries, or features of the 3D representation.

[0075] As illustrated, each potential insertion pathway 720a-720c may each have an independent respective insertion site 710a-710c. In contrast, two or more of the candidate insertion pathways 720a-720c may have a same insertion site. For example, a fourth curved insertion pathway 720d may have a same insertion site 710a as the first insertion pathway 720a. As such, a single insertion site 710a may be used for more than one insertion pathway.

[0076] As illustrated in FIG. 7, an insertion pathway 720 may include one or more curves or arcs. For inserting the instrument 210 with a curved path, the instrument 210 may be inserted into the subject 702 at a first position 730a. The distal end of the instrument may be imaged by the sensing probe 214 while the instrument 210 is being inserted into the patient to monitor a position of the instrument 210 along the insertion path 720. At a point during the insertion of the instrument 210, the instrument 210 may be repositioned to a second position 730b having an angle relative tothe first position 730 to cause the instrument to curve along a first arc 721a. At another point during insertion of the instrument 210, the instrument 210 may be repositioned again to a third position 730c with an angle relative to the first and second positions 730a and 730b to cause the instrument 210 to bend along a second arc 721b to reach the treatment site 712. As such, an insertion path may include one or more curves or arcs to reach a treatment site, avoid restricted regions, avoid objects, reduce a number of traversed voxels, etc. Additionally, the sensing probe 214 may continuously image the FOV 715 to observe the position of a distal end of the instrument to provide feedback for controlling the position of the instrument to achieve a certain insertion pathway depth and / or pathway curves.

[0077] In examples, the method 600 may optionally include, at block 610, obtaining preoperative imaging data and merging the preoperative imaging data and the 3D volume representation to provide a combined image of the target volume 700. The preoperative image data may include one or more of CT scan data, MRI scan data, cone beam CT data, and / or ultrasound data. The combined image of the target volume may allow for some tissues, voids, and structures to be imaged, or allow for better imaging of a region of a patient. For example, ultrasound imaging does not pass through regions of air, such as in the lungs. Ultrasound imaging can image the airlung surface, and combined with a CT scan, the combined image shows that surface of the lung along with the mostly filled air cavity of the lung structure. For another example, a bone may obscure an ultrasound image from imaging a portion of an organ behind the bone, combined with an MRI scan, the resultant image allows for imaging of the organs that were obscured by the bone structure.

[0078] A preoperative MRI or CT scan may provide a higher image quality and potentially can show additional features or targets that may be hard to image using ultrasound imaging. In addition, a preoperative imaging set may have a larger field of view and could be analyzed by an operator to preselect a target, such as a tumor target. Matching pre-operative FOV imaging data with an ultrasound image data set can guide the positioning and orientation of an ultrasound probe or sensor towards targets that may hard to reach. Data or image fusion can be performed using rigid registration or non-rigid registration (e.g. vessel based registration). Because the preoperative dataset is not updated with real-time anatomical changes, the ultrasound scan can provide the updated real time information. For one example, when performing an ablation procedure, apreoperative scan can be combined with a postoperative 3D ultrasound volume to identify if a tumor has been fully ablated.

[0079] At block 612 the control system 216, or a user, may adapt the determined instrument insertion pathway based on the combined image of the target volume. For example, the preoperative data may further provide insight into tissue densities, objects, features, or boundaries that the original 3D representation may not have captured. Therefore, the instrument insertion path may be updated or otherwise adapted based on information gleaned from the combined image of two different images and / or 3D representations of the target region 700.

[0080] In examples, the method 600 may further include receiving pressure data from the pressure sensor 248 coupled to the instrument driver 242, with the pressure data indicative of a pressure exerted on the instrument. The control system 216 may then determine a current instrument position or trajectory based on the received pressure data. For example, the control system 216 may determine that the instrument has reached a border between different types of tissue, or the surface of an organ based on a change in the pressure data, and may further determine the position of the instrument along the insertion pathway. Additionally, the control system 216 may determine, from the pressure data, that the position of the instrument is not along the desired insertion pathway and the insertion trajectory may be updated to compensate for the current position of the instrument. The pressure data, along with a current position or trajectory of the instrument, may be used to update the insertion trajectory, or to adjust control of the instrument to compensate for any deviation from a desired insertion trajectory.

[0081] The system 100 can continuously perform the ultrasound scans to image and analyze the motion of the patient due to breathing by imaging the distance and positions between the target, and needle tip or instrument 210 to correct the path or trajectory of the instrument 210 based on the patients breathing. The system 100 may determine motions of the patient from functions such as breathing, heartbeat, digestion, or another biological motion or function. Additionally, the breathing of the patient may be measured by placing a sensor on the body of the patient, extracted from an ECG, determined during an evaluation and having the patient take a deep breath or provide a breathing pattern sample, from the pressure sensor 234 coupled to the sensing probe 214, or by another means. Further, the system may only move the instrument 210 or a needle during certain breathing or motion phases, such as only during exhalation.

[0082] At block 614, the method 600 includes controlling the repositionable structure 202 and the instrument holder 208 to advance the instrument 210 along the determined or identified insertion pathway. The instrument 210 is advanced until it reaches the treatment site 712 and treatment is administered. The instrument is then extracted out of the subject 702 along the insertion pathway. In implementations with a second instrument 262, such as a stylus, the stylus may alone be retracted back through the instrument 210, such as a needle, while the needle remains inside of the patient, creating a channel in the needle to deliver additional therapies. In examples, the instrument 210 may further be rotated or vibrated to advance the instrument 210 along the insertion pathway to assist with reduction of induced pain, reduction of required insertion force, to cross from one tissue to another, or to insert the instrument 210 into denser tissues, harder tissues or objects, rolling or movable objects or targets (e.g., a lymph node), etc. The pressure sensor 234 coupled to the instrument holder 208 may be used to provide a close-feedback loop and the needle insertion pathway or trajectory may be updated or further determined from derived forces from the pressure sensor feedback loop.

[0083] The probe holder 212 may further be controlled to reposition the probe to perform one or more update scans or image sweeps of the target region 700 during the insertion of the instrument 210. In examples, the method 600 may further include controlling at least one of the repositionable structure 202 and the probe holder 212 to keep the position of the instrument 210 within the FOV 715 of the sensing probe 214. Together the repositionable structure 202 and probe holder 212 control a position and orientation of the sensing probe 214, and as such, the sensing probe 214 may be moved and reoriented during insertion of the instrument 210 to actively image the position of the instrument 210 as the instrument 210 moves along the insertion pathway. In controlling the repositionable structure 202, the method 600 may include moving the repositionable structure 202 according to a first set of movements which cause the movement of both the probe holder 212 and instrument holder 208 simultaneously. Once repositioned, the sensing probe 214 may then obtain images of the instrument 210 (e.g., a portion of the instrument, a distal end of the instrument, or another part of the instrument) to track and provide feedback pertaining to the position and trajectory of the instrument 210 during advancement of the instrument 210 along the insertion pathway. The sensing probe 214 may be repositioned actively along with the insertion of the instrument 210 with the sensing probe 214 obtaining real-time images to track the position and / or trajectory of the instrument 210. In specific examples, themethod 600 may include moving the probe holder 212 and the instrument holder 212 in tandem such that a distal end of the instrument 210 is maintained in the FOV 715 of the sensing probe 214 during movement of the instrument 210. The control system may move the sensing probe 214 and the instrument 210 synchronously to maintain the distal end of the instrument 210 within a 2D imaging plane of the sensing probe 214. In embodiments, the distal end is maintained within the 2D imaging plane as the distal end inserts from an insertion site towards the treatment site or retracts from the treatment site towards the insertion site. Additionally, movement of the repositionable structure 202 and the probe holder 208 may be controlled to maintain a 2D plane imaging depth while also maintaining a given exerted pressure of the sensing probe 214 against the subject 702 or against a target during scanning of the target volume 700.

[0084] The method 600 may including performing one or more update image sweeps (also referred to herein as an “update real-time probe sweep”). For example, the method 600 may further include controlling the instrument driver 242 to pause the advancement of the instrument 210. The repositionable structure 202 and / or the probe holder 212 may then be controlled to perform an update image sweep of the target region 700 to obtain a plurality of images representative of realtime conditions of the target region. The update image sweep may be performed by sweeping over a same region with a same pressure between the sensing probe 214 and the subject 205, or the update image sweep may be performed at a different pressure against the subject 205 or at different angles of points of view of the target region 702. The control system 216 may then construct an updated 3D representation of the target volume from the obtained images. The updated 3D representation may be used to update the previous 3D representation, the previous 3D representation may be saved in a memory, or the previous 3D representation may be provided simultaneously with the updated 3D representation for comparison. The updated 3D representation may reflect a current position and / or trajectory of the instrument 210 as well as any movement of organs and tissues relative to the insertion pathway and the instrument 210. The insertion pathway may then be updated based on the updated 3D representation.

[0085] The update image sweep may be periodically performed based on a predefined parameter or setting. For example, the update image sweep may be performed in response to the advancement of the instrument by a predefined distance along the insertion pathway. The predefined distance may be on the order of millimeters, less than 1 cm, about 2 cm, about 3 cm, between 3cm and 5 cm, 10 cm, or greater than 10 cm. The updated probe sweep may be performedbased on a given time interval (e.g., every half second, every second, every 5 seconds, every 10 seconds, every 30 seconds, etc.). In some examples, the update image sweep may be performed based on pressure data received from the pressure sensor 248. The pressure data may include one or more unexpected pressure values indicating that the instrument may not be at a position along the desired insertion pathway, or that the instrument has encountered a restricted region, tissue, or organ. As such, the repositionable structure 202 and the instrument driver 242 may stop the insertion of the instrument 210 to perform the update image sweep to update the insertion pathway (and / or validate a current insertion pathway) based on the updated 3D representation.

[0086] In implementations, the method 600 may further include, at block 616, repositioning the repositionable structure and moving the sensing probe to a new position to image a different part of the target volume 700, a different orientation of the target volume 700, a different angle or view of the target 705, or a different region of the subject 702. At block 618, the control system 216 identifies the position of the target volume 700 relative to the new position of the sensing probe 214. The control system 216 may store in memory previous spatial coordinates and compare the previous spatial coordinates to new repositioned spatial coordinates to determine the position of the target volume 7090 relative to the new position of the sensing probe 214. The control system 216 may map the relative position of the target volume 700 based on known positions of the sensing probe 214, or by a repositioning vector between the old position and the repositioned position and orientation of the sensing probe 214. In examples, to identify the position of the target volume relative to the new position of the sensing probe, the control system 216 may receive a user input identifying the location of the target volume 700 such as via a user interface or by providing spatial coordinates of the target volume or of the new position of the sensing probe.

[0087] At block 620, probe holder 212 repositions the sensing probe 214 such that at least a portion of the target volume 700 is within the FOV 715 of the sensing probe 214. At block 622, the sensing probe obtains one or more images of the target volume 700 from the new position and orientation of the sensing probe 214. The sensing probe 214 may be controlled to perform a sweep over the FOV 215 of the sensing probe 214 to obtain a plurality of images of the target volume. The control system 216 may then reconstruct a new 3D reconstruction of the target volume based on the plurality of images from performing the sweep of the sensing probe 214.

[0088] The method may further include receiving pressure data indicative of a pressure exerted on the probe, from the pressure sensor 248 operatively coupled to the sensing probe 214. Basedon the pressure data indicative of the pressure exerted on the probe 214, the control system 216 may control the probe holder 212 to maintain a desired pressure of the sensing probe 214 against the subject 702.

[0089] In examples, the method 600 may further include controlling the repositionable structure 202 and the instrument holder 208 to extract the instrument 210 along the instrument insertion pathway. During the extraction, the controller may control the repositionable structure 202 and the probe holder 212 to maintain the distal end of the instrument 210 in the FOV of the sensing probe 214. After the instrument 210 is extracted from the subject 702, the instrument 210 may be replaced by another instrument, and the controller may control the repositionable structure 202 and the instrument holder 208 to insert the second instrument into the subject along the instrument insertion pathway. The second instrument may be inserted to further provide treatment to the treatment site 712 of the target 705, or to observe the treatment site 712, or to extract tissue or part of the target 705.

[0090] The method 600 described above may be implemented to perform a multi-stage process using a first instrument to perform a first stage of the treatment process, and a second instrument to perform a second stage of the treatment process. FIG. 9 is an example flow diagram of an example method 900 for performing ultrasound guided robotic percutaneous instrument insertion for performing a multi-stage treatment, according to some examples.

[0091] The method 900, at block 902, includes obtaining a plurality of images of the target volume 700 having a target 705 for providing multi-stage treatment. The sensing probe 214 obtains the plurality of images and, at block 904, reconstructs a 3D representation of the target volume 700. At block 906, the control system 216 segments the 3D representation to identify objects, boundaries and features in the 3D representation. At block 908, the control system 216 then determines one or more instrument insertion pathways from the 3D representation and the identified objects, boundaries, and / or features.

[0092] At block 910, the repositionable structure 202 and the instrument holder 208 insert the first instrument along a first stage insertion pathway. The first instrument provides a first stage treatment to the treatment site 712. The first stage treatment may provide a multistage treatment to a target region or target or a patient. For example, a multistage treatment may include a first stage treatment including using an ultrasound probe at a needle tip to image a target or target region, and then replacing the ultrasound probe with a medication delivery catheter. For manyelectrical treatments for tumors, multiple electrodes need to be placed around a tumor volume to treat the full tumor which also requires multiple stages of delivering electrodes. Yet another multistage treatment could include placing a catheter tube at a target, and then deliver pain medication through the catheter. Additionally, multiple biopsies may be performed as a multistage treatment in order to retrieve enough tissue using a smaller needle. Further, a multistage treatment may include repositioning or recalculation of a needle trajectory. For example, a first stage can include insertion of the needle through the skin and into the first two to three centimeters of the tissue. The needle trajectory can be reassessed at that point and if the needle trajectory still correlates with the planned trajectory along a path to reach the target. If not, the system 100 can adjust the angle of the needle to adjust the pathway towards a target facing angle and continue to further insert the needle.

[0093] At block 912, the repositionable structure 202 and the instrument holder 208 extract the first instrument along the first stage insertion pathway. The first instrument is then replaced with a second instrument, or a second instrument having a second instrument is used to insert the second instrument into the subject 702.

[0094] In examples, the method 900 may be performed using a single insertion pathway, or multiple insertion pathways. For the single insertion pathway, at block 914, the repositionable structure 202 and the instrument holder 208 insert the second instrument along the first stage insertion pathway. The second instrument is used to provide second treatment to the treatment site 712. The second treatment may include performing one or more of delivering a drug to the treatment site, providing a therapy to the treatment site, performing a biopsy at the treatment site, blood drawing, delivery of electronic leads, microrobots, or microsensors, delivering energy, providing gene therapy, providing a pice line, providing a dialysis fistula, delivery of hydrogel, delivery of an implant, placement of needle electrodes, extracting tissue, delivering tissue, measuring tissue, and / or destroying tissue. At block 916 repositionable structure 202 and the instrument holder 208 extract the second instrument along the first insertion pathway. Further, a needle canular may be left in the first insertion pathway and other instruments such as biopsy tools, or energy delivery tools, may be inserted through the needle cannula to perform additional treatments.

[0095] Additionally, multiple needles may be inserted into a patient to provide one or more treatments. For example, a first needle may be inserted into a tumor to provide energy to the tumor.The needle can be left inserted into the tumor and decoupled from the system 100. The system 100 may then be loaded with a second needle. The second needle may then be inserted into the patient or the tumor adjacent to the first needle according to a treatment planning. This is particularly useful when delivering an energy based therapy where a single needle is used on an ellipsoid volume to treat the volume that is less than the tumor volume wherein multiple needles need to be delivered to cover the full tumor which can be as much as ten needles in current technologies.

[0096] In other examples, the multi-stage treatment method 900 may utilize multiple insertion pathways for performing multiple stages of treatment. The multiple insertion pathways may utilize pathways traversing the fewest voxels in the 3D representation. Utilizing multiple insertion pathways allows for providing treatment to different regions r areas of a target, for example, performing multiple tumor biopsies of different parts of a tumor. Additionally, the insertion sites for the multiple insertion pathways may be determined by the control system 216 or by input from a user. At block 918 the repositionable structure 202 and the instrument holder 208 insert the second instrument into a second insertion pathway that is different than the first insertion pathway. As illustrated in FIG. 8, the first insertion pathway may be first insertion pathway 720a, and the second insertion pathway may be second insertion pathway 720b. As such, the second insertion pathway 720b has a second insertion site 710b that is different than the first insertion site 710a. Both the first and second insertion pathways 720a and 720b provide a pathway for the first and second instruments to provide treatment to a same, or nearby, treatment site 712 of a target 705. At block 920, the repositionable structure 202 and the instrument holder 208 extract the second instrument along the second insertion site 720b.

[0097] In examples, the method 900 may further include inserting a third instrument to provide additional treatment stages during the multi-stage treatment. The third instrument may be inserted along either then first insertion pathway 720a, second insertion pathway 720b, or along a third insertion pathway that is different than the first and second insertion pathways 720a and 720b. For some multi-stage treatments, it may be required that the first instrument and the second instrument are inserted through respective insertion pathways to provide simultaneous treatment at the treatment site 712 of the target 705. For example, a first treatment may include providing active anesthesia to the treatment site, while the second treatment performs a biopsy at the treatment site 712 or nearby on the target 705. Additionally, the first stage and second stage may utilize a same insertion site with different insertion pathways to a same treatment site, or to two differenttreatment sites. In further examples, the first and second instruments are a same instrument that may be used to perform multiple stages of the treatment such as to provide two doses of a drug, or to perform two biopsies. Further, the same instrument may be used for the first and second stages with different insertion sites to provide treatment to different areas of the target 705, or to provide treatment to different sides or from different angles of the treatment site 712.

[0098] In some examples, the control system 216 may further provide a recommendation of an instrument type based on the determined instrument insertion pathway. For example, the control system 216 may provide a suggestion of a needle type for a given insertion pathway. A straight insertion pathway may allow for a thicker or more rigid needle type, whereas a curved insertion path may require a thinner needle or a needle of a more flexible material. Further, various needle tips (e.g., sharp blunt, conical, coned, beveled etc.) may be used for traversing different types of tissue and performing different types of treatments. The control system 216 may suggest a needle recommendation including one or more of a needle size, needle width, needle length, needle diameter, needle tip, or needle material.

[0099] One or more components of the embodiments discussed in this disclosure, such as control system 112, may be implemented in software for execution on one or more processors of a computer system. The software may include code that when executed by the one or more processors, configures the one or more processors to perform various functionalities as discussed herein. The code may be stored in a non-transitory computer readable storage medium (e.g., a memory, magnetic storage, optical storage, solid-state storage, etc.). The computer readable storage medium may be part of a computer readable storage device, such as an electronic circuit, a semiconductor device, a semiconductor memory device, a read only memory (ROM), a flash memory, an erasable programmable read only memory (EPROM); a floppy diskette, a CD-ROM, an optical disk, a hard disk, or other storage device. The code may be downloaded via computer networks such as the Internet, Intranet, etc. for storage on the computer readable storage medium. The code may be executed by any of a wide variety of centralized or distributed data processing architectures. The programmed instructions of the code may be implemented as a number of separate programs or subroutines, or they may be integrated into a number of other aspects of the systems described herein. The components of the computing systems discussed herein may be connected using wired and / or wireless connections. In some examples, the wireless connections may use wireless communication protocols such as Bluetooth, near-field communication (NFC),Infrared Data Association (IrDA), home radio frequency (HomeRF), IEEE 802.11 , Digital Enhanced Cordless Telecommunications (DECT), and wireless medical telemetry service (WMTS).

[0100] Various general-purpose computer systems may be used to perform one or more processes, methods, or functionalities described herein. Additionally or alternatively, various specialized computer systems may be used to perform one or more processes, methods, or functionalities described herein. In addition, a variety of programming languages may be used to implement one or more of the processes, methods, or functionalities described herein.

[0101] While certain embodiments and examples have been described above and shown in the accompanying drawings, it is to be understood that such embodiments and examples are merely illustrative and are not limited to the specific constructions and arrangements shown and described, since various other alternatives, modifications, and equivalents will be appreciated by those with ordinary skill in the art.

Claims

WHAT IS CLAIMED:

1. A computer-assisted system comprising: a repositionable structure; a probe holder coupled to the repositionable structure, the probe holder supporting a sensing probe; an instrument holder coupled to the repositionable structure, the instrument holder supporting an instrument and including an instrument driver configured to control the instrument; and a control system comprising one or more processors, where in the control system is operably coupled to the repositionable structure, the probe holder, and the instrument holder the control system is configured to: obtain, by the sensing probe, images of a target volume of a subject; reconstruct a three-dimensional (3D) representation of the target volume from the images of the target volume; segment the 3D representation of the target volume to identify one or more objects, boundaries, or features in the 3D representation; determine a determined pathway including an insertion site and a treatment site; and control the repositionable structure and the instrument holder to advance the instrument along the determined insertion pathway.

2. The computer- assisted system of claim 1, wherein the sensing probe comprises at least one of an ultrasound probe, photoacoustics probe, microscope, optical imaging probe, video camera, infrared camera, handheld x-ray system, electronic palpation device, impulse-radar detector, focused ultrasound probe, gamma probe, perfusion imager, optical coherence tomography device.

3. The computer- assisted system of claim 1, wherein to determine the instrument insertion pathway, the control system is configured to: determine two or more candidate insertion pathways and select, by the system or via a user input, the determined insertion pathway from the two or more candidate insertion pathways.

4. The computer- assisted system of claim 3, wherein to select the determined insertion pathway, the control system is configured to: select a pathway based upon at least one of pathway length, a number of voxels the pathway traverses in the 3D representation, avoidance of identified objects, an amount of curvature associated with the pathway, an instrument type of the instrument, a needle type for a needle in which the instrument is inserted, and one or more operator preferences.

5. The computer-assisted system of claim 1, wherein the 3D representation includes a plurality of voxels including (i) voxels with free space not including tissue, and (ii) voxels including tissue, and wherein the control system determines the determined insertion pathway based on a number of voxels in free space versus a number of voxels with tissue.

6. The computer- assisted system of claim 3, wherein at least two of the two or more candidate insertion pathways have a same insertion site.

7. The computer-assisted system of claim 3, wherein at least two of the two or more candidate insertion pathways have a different insertion site.

8. The computer-assisted system of claim 4, wherein the operator preferences include one or more of user-identified restricted regions of tissue or anatomical objects, a user-identified maximum path curvature, a user preference for a straight path, a user-identified insertion site preference, and a user-identified insertion site restriction.

9. The computer-assisted system of any of claims 1 to 8, wherein the control system is further configured to: determine a restricted region representative of a 3D volume not to be traversed by any pathway, and determine the determined insertion pathway that does not intersect the restricted region.

10. The computer-assisted system of claim 9, wherein the control system is configured to: automatically determine the restricted region from the one or more objects, boundaries, or features in the 3D representation.

11. The computer-assisted system of any of claims 1 to 8, wherein the control system is further configured to: obtain preoperative imaging data including one or more of CT scan data, MRI scan data, cone beam CT data, or ultrasound data; and merge the preoperative imaging data and 3D representation to provide a combined image of the target volume.

12. The computer-assisted system of claim 11, wherein the control system is further configured to: adapt an insertion pathway associated with the preoperative imaging data to the combined image of the target volume.

13. The computer-assisted system of any of claims 1 to 8, further comprising: a pressure sensor configured to measure pressure exerted against the instrument, and wherein the control system is configured to: receive pressure data indicative of pressure on the instrument; and determine a current instrument position or trajectory from the pressure data.

14. The computer-assisted system of claim 13, wherein the control system is further configured to: update the instrument insertion trajectory based on the current instrument position or trajectory, and the pressure data.

15. The computer-assisted system of any of claims 1 to 8, wherein the control system is further configured to: control at least one of the repositionable structure and the probe holder to maintain the instrument within a field of view (FOV) of the sensing probe as the instrument moves along the insertion pathway; and obtain images from the sensing probe during instrument insertion to track advancement of the instrument.

16. The computer-assisted system of claim 15, wherein to control the repositionable structure, the control system is configured to: command the repositionable structure to move according to a first set of movements which further causes simultaneous movement of both the probe holder and the instrument holder according to the first set of movements.

17. The computer-assisted system of claim 15, wherein the control system is further configured to: reposition the repositionable structure and move the sensing probe to a new position; identify the position of the target volume relative to the new position of the sensing probe; reposition the sensing probe such that at least a portion of the target volume is within the FOV of the sensing probe; and obtain, via the sensing probe, one or more images of the target volume in the FOV of the sensing probe.

18. The computer-assisted system of claim 17, wherein to identifying the position of the target volume relative to the new position of the sensing probe, the control system is configured to: receiving a user input identifying the location of the target volume.

19. The computer-assisted system of claim 17, wherein to obtain one or more images of the target volume in the FOV of the sensing probe the control system is further configured to:control the probe holder to sweep the FOV of the sensing probe across the target volume; and capture a plurality of images using the sensing probe while performing the sweep.

20. The computer-assisted system of any of claims 1 to 8, wherein the control system is further configured to: control the instrument driver to pause advancement of the instrument; perform an update real-time probe sweep; and update the 3D representation according to the update real-time probe sweep.

21. The computer-assisted system of claim 20, wherein the update real-time probe sweep is performed periodically in response to the instrument advancing a predefined distance along the insertion pathway.

22. The computer-assisted system of claim 21, wherein the predefined distance is 3cm.

23. The computer-assisted system of claim 20, further comprising: a pressure sensor configured to measure pressure exerted against the instrument, and wherein the update real-time probe sweep is performed in response to unexpected pressure data from the pressure sensor.

24. The computer-assisted system of any of claims 1 to 8, wherein the control system is further configured to provide, by the processor, a needle type recommendation based on the instrument insertion pathway.

25. The computer-assisted system of claim 24, wherein the needle type recommendation includes one or more of a needle size, needle width, needle length, needle diameter, and needle tip.

26. The computer-assisted system of any of claims 1 to 8, wherein to advance the instrument along the insertion pathway, the control system is configured to: control at least one of a needle rotation and a needle vibration.

27. The computer-assisted system of any of claims 1 to 8, wherein to segment the 3D representation of the target volume the control system is further configured to: identify a treatment site.

28. The computer-assisted system of any of claims 1 to 8, wherein the control system is further configured to: receive a user identification of a treatment site from the 3D representation of the target volume.

29. The computer-assisted system of and of claims 1 to 8, further comprising: a probe pressure sensor operatively coupled to the sensing probe, the probe pressure sensor configured to measure pressure exerted against the probe, and wherein the control system is further configured to: control the probe holder to maintain the exerted pressure and position against the subject.

30. The computer-assisted system of any of claims 1 to 8, wherein the control system is configured to: control movement of the probe holder and the instrument in tandem such that a distal end of the instrument is maintained in a FOV of the sensing probe during movement of the instrument.

31. The computer-assisted system of claim 30, wherein to maintain the distal end of the instrument in the FOV of the sensing probe the control system is configured to: maintain the position of the distal end of the instrument within a 2D imaging plane of the sensing probe.

32. The computer-assisted system of any of claims 1 to 8, wherein the control system is configured to: during image scanning of the target volume, control the movement of the repositionable structure and the probe holder to maintain (i) a 2D imaging plane depth, and (ii) pressure of the sensing probe against the target.

33. The computer-assisted system of any of claims 1 to 8, wherein the control system is further configured to: extract the instrument along the instrument insertion pathway.

34. The computer-assisted system of claim 33, wherein the control system is further configured to: control the probe holder to maintain a distal end of the instrument to be in the FOV of the sensing probe while the instrument is extracted along the insertion pathway.

35. The computer-assisted system of claim 33, wherein the control system is further configured to: advance a second instrument along the insertion pathway.

36. The computer-assisted system of any of claims 1 to 8, wherein: the computer-assisted system is used to perform a multi-stage process using a first instrument for a first stage and a second instrument for a second stage, and the control system is further configured to: identify a plurality of insertion paths for inserting instruments into the subject to provide treatment to the treatment site for each stage of the multi-stage process.

37. The computer-assisted system of claim 36, wherein each stage of the multi-stage process utilizes a same insertion site.

38. The computer-assisted system of claim 36, wherein at least two stages of the multi-stage process utilize different insertion sites.

39. The computer-assisted system of claim 36, wherein the plurality of insertion paths comprise at least one of (i) different insertion sites, and (ii) a different insertion path to the treatment site.

40. The computer-assisted system of claim 36, wherein the first instrument and second instrument are the same instrument.

41. The computer-assisted system of any of claims 1 to 8, wherein: the repositionable structure comprises a plurality of links and a plurality of joints, the instrument holder is configured to receive an instrument and coupled to a distal link of the plurality of links, and the probe holder is configured to receive a sensing probe having a field of view and coupled to the distal link, the probe holder.

42. The computer-assisted system of any of claims 1 to 8, wherein the repositionable structure has at least four degrees of freedom of motion.

43. A method for performing ultrasound guided robotic percutaneous instrument insertion, the method comprising: obtaining, by a sensing probe, images of a target volume of a subject; reconstructing, by a control system, a three-dimensional (3D) representation of the target volume from the images of the target volume; segmenting, by the control system, the 3D representation of the target volume to identify one or more objects, boundaries, or features in the 3D representation of the target volume; determining, by the control system, a determined instrument insertion pathway including an insertion site and a treatment site; and controlling, by the control system, a repositionable structure and an instrument holder to advance an instrument along the determined insertion pathway, wherein a position of the sensing probe is supported by a probe holder that is coupled to the repositionable structure.

44. The method of claim 43, wherein the sensing probe comprises at least one of an ultrasound probe, photoacoustics probe, microscope, optical imaging probe, video camera, infrared camera, handheld x-ray system, electronic palpation device, impulse-radar detector, focused ultrasound probe, gamma probe, perfusion imager, optical coherence tomography device.

45. The method of claim 43, wherein determining the determined instrument insertion pathway includes; determining, by the control system, two or more candidate insertion pathways and selecting a determined insertion pathway from the two or more candidate insertion pathways.

46. The method of claim 45, wherein selecting the determined insertion pathway includes: selecting, by the control system, the determined pathway based upon at least one of pathway length, a number of voxels the pathway traverses in the 3D representation, avoidance of identified objects, an amount of curvature associated with the pathway, an instrument type of the instrument, a needle type for a needle in which the instrument is inserted, and one or more operator preferences.

47. The method of claim 43, wherein: the 3D representation includes a plurality of voxels including (i) voxels with free space not including tissue, and (ii) voxels including tissue, and determining the determined instrument insertion pathway comprises: determining, by the control system, the determined instrument insertion pathway based on a number of voxels in free space versus a number of voxels with tissue.

48. The method of claim 45, wherein at least two of the two or more candidate insertion pathways have a same insertion site.

49. The method of claim 45, wherein at least two of the two or more insertion pathways have a different insertion site.

50. The method of claim 46, wherein the operator preferences include one or more of user- identified restricted regions of tissue or anatomical objects, a user-identified maximum path curvature, a user preference for a straight path, a user-identified insertion site preference, and a user-identified insertion site restriction.

51. The method of any of claims 43 to 50, further comprising: determining, by the control system, a restricted region representative of a 3D volume not to be traversed by the two or more pathways, wherein the determining the determined instrument insertion pathway further comprises: determining, by the control system, two or more candidate pathways that do not intersect the restricted region.

52. The method of claim 51, further comprising: automatically determining, by the control system, the restricted region from the one or more objects, boundaries, or features in the 3D representation.

53. The method of any of claims 43 to 50, further comprising obtaining, by the control system, preoperative imaging data including one or more of CT scan data, MRI scan data, cone beam CT data, or ultrasound data; and merging, by the control system, the preoperative imaging data and 3D representation to provide a combined image of the target volume.

54. The method of claim 53, further comprising: adapting, by the control system, a proposed insertion pathway associated with the preoperative imaging data to the combined image of the target volume.

55. The method of any of claims 43 to 50, further comprising: receiving, from a pressure sensor configured to measure pressure exerted against the instrument, pressure data indicative of pressure on the instrument; and determining, by the control system, a current instrument position or trajectory from the pressure data.

56. The method of claim 55, further comprising: updating, by the control system, the determined insertion trajectory based on the current instrument position or trajectory, and the pressure data.

57. The method of any of claims 43 to 50, further comprising: controlling, by the control system, at least one of the repositionable structure and the probe holder to keep the instrument within a field of view of the sensing probe as the instrument moves along the insertion pathway; and obtaining, by the control system and from the sensing probe, images during instrument insertion to track advancement of the instrument.

58. The method of claim 57, wherein controlling the repositionable structure comprises: controlling, by the control system, the repositionable structure to move according to a first set of movements which further causes simultaneous movement of both the probe holder and the instrument holder according to the first set of movements.

59. The method of any of claims 43 to 50, further comprising: controlling, by the control system, an instrument driver to pause advancement of the instrument; performing, by the control system, an update real-time probe sweep; and updating, by the control system, the 3D representation according to the update real-time probe sweep.

60. The method of claim 59, wherein the update real-time probe sweep is performed periodically in response to the instrument advancing a predefined distance along the insertion pathway.

61. The method of claim 60, wherein the predefined distance is 3cm.

62. The method of claim 59, wherein the update real-time probe sweep is performed in response to unexpected pressure data from a pressure sensor.

63. The method of any of claims 43 to 50, further comprising: providing, by the control system, a needle recommendation based on the instrument insertion pathway.

64. The method of claim 63, wherein the needle type recommendation includes one or more of a needle size, needle width, needle length, needle diameter, and needle tip.

65. The method of any of claims 43 to 50, wherein advancing the instrument along the insertion pathway comprises: controlling, by the control system, at least one of a needle rotation and a needle vibration.

66. The method of any of claims 43 to 50, wherein segmenting the 3D representation of the target volume comprises: identifying, by the control system, a treatment site.

67. The method of any of claims 43 to 50, further comprising: receiving, by the control system, a user identification of a treatment site from the 3D representation of the target volume.

68. The method of any of claims 43 to 50, further comprising: repositioning, by the control system, the repositionable structure to move the sensing probe to a new position; identifying, by the control system, the position of the target volume relative to the new position of the sensing probe; repositioning, by the control system, the sensing probe such that at least a portion of the target volume is within the FOV of the sensing probe; and obtaining, via the sensing probe, one or more images of the target volume in the FOV of the probe.

69. The method of claim 68, wherein identifying the position of the target volume relative to the new position of the sensing probe comprises: receiving, by the control system, a user input identifying the position of the target volume.

70. The method of claim 68, wherein obtaining one or more images of the target volume in the FOV of the sensing probe comprises: controlling, by the control system, the probe holder to sweep the FOV of the sensing probe across the target volume; and capturing, using the sensing probe, a plurality of images while performing the sweep.

71. The method of any of claims 43 to 50, further comprising: receiving, from a probe pressure sensor operatively coupled to the sensing probe, an exerted pressure on the sensing probe; and controlling, by the control system, the probe holder to maintain the exerted pressure and position against the subject.

72. The method of any of claims 43 to 50, further comprising: controlling, by the control system, the movement of the probe holder and the instrument is performed in tandem such that a distal end of the instrument is maintained in a FOV of the sensing probe during movement of the instrument.

73. The method of claim 72, wherein maintaining the distal end of the instrument in the FOV of the sensing probe comprises: maintaining, by the control system, the position of the distal end of the instrument within a 2D imaging plane of the sensing probe.

74. The method of claim 71 , further comprising:during image scanning of the target volume, controlling, by the control system, the movement of the rcpositionablc structure and the probe holder to maintain (i) a 2D imaging plane depth, and (ii) exerted pressure of the sensing probe against the target.

75. The method of any of claims 43 to 50, further comprising: extracting, by the control system, the instrument along the instrument insertion pathway.

76. The method of claim 75, further comprising: controlling, by the control system, the probe holder to maintain a distal end of the instrument to be in the FOV of the sensing probe while the instrument is extracted along the insertion pathway.

77. The method of claim 75, further comprising: advancing, by the control system, a second instrument along the insertion pathway.

78. The method of any of claims 43 to 50, wherein the control system is configured to perform a multi-stage process using a first instrument for a first stage and a second instrument for a second stage, wherein the method further comprises: identifying, by the control system, a plurality of candidate insertion paths for inserting instruments into the subject to provide treatment to the treatment site for each stage of the multistage process.

79. The method of claim 78, wherein each stage of the multi-stage process has a same insertion site.

80. The method of claim 78, wherein at least two stages of the multi-stage process have different insertion sites.

81. The method of claim 78, wherein the plurality of candidate insertion paths comprise at least one of (i) different insertion sites, and (ii) a different insertion path to the treatment site.

82. The method of claim 78, wherein the first instrument and second instrament are the same instrument.

83. The method of any of claims 43 to 50, wherein: the repositionable structure comprises a plurality of links and a plurality of joints, the instrument holder is configured to receive an instrament and is coupled to a distal link of the plurality of links, and the probe holder is configured to receive a sensing probe having a field of view and coupled to the distal link, the probe holder.

84. The method of any of claims 43 to 50, wherein the repositionable structure has at least four degrees of freedom of motion.

85. A computer-readable media storing instructions that, when executed by a processor, cause a system to perform the method of any of claims 43-84.

Images