Workflow for ultrasound-guided percutaneous needle robot

Abstract

Systems and methods for inserting an ultrasound-guided robotic percutaneous instrument into a subject are described. A method may include (i) obtaining an image of a target volume of a subject, the image obtained by a sensing probe physically positioned by a probe holder coupled to a repositionable structure; (ii) reconstructing a three-dimensional (3D) representation of the target volume from the image of the target volume; (iii) segmenting the 3D representation of the target volume to identify one or more objects, boundaries or features in the 3D representation of the target volume; (iv) determining an instrument insertion path comprising an insertion site and a treatment site; and (v) controlling the repositionable structure and the instrument holder to advance the instrument along the insertion path.

Description

[0001] Cross-references to related applications

[0002] This application claims priority and benefit to U.S. Provisional Patent Application No. 63 / 580,860, entitled "WORKFLOW FOR AN ULTRASOUND GUIDEDPERCUTANEOUS NEEDLE ROBOT", filed September 6, 2023, and U.S. Provisional Patent Application No. 63 / 580,868, entitled "ROBOTIC SURGICAL MANIPULATOR SYSTEM", also filed September 6, 2023. The entire contents of each of these two U.S. Provisional Patent Applications are expressly incorporated herein by reference. Technical Field

[0003] The disclosed examples relate to the control of a guided robot. Specifically, the disclosed examples relate to systems and methods for analyzing image data and determining an insertion path for a medical device to reach the treatment area of ​​a subject. Background Technology

[0004] Minimally invasive medical techniques aim to reduce the amount of tissue damaged during medical procedures, thereby reducing patient recovery time, discomfort, and harmful side effects. Such techniques can be performed through natural openings in the patient's anatomy or through one or more surgical incisions. Through these natural openings or incisions, physicians can insert minimally invasive medical instruments (including surgical, diagnostic, therapeutic, and / or biopsy instruments) to reach the target tissue location. A wide variety of medical instruments, including needles and probes of various lengths, sizes, and flexibility, can be used to reach the target volume. For example, some medical instruments may be rigid and only capable of traversing a linear insertion path, while others may be able to bend to reach the therapeutic volume along one or more curves of the insertion path. Furthermore, the path may change or vary during insertion due to patient movement or to compensate for other errors.

[0005] Image data of the subject is used to plan the path toward the region of interest and to ensure proper navigation during the procedure. As an example, image data can be analyzed to separate traversable blood vessels from other objects in the image that could potentially harm the patient if ruptured by an invasive medical device. Ultrasound imaging is a tool frequently used to perform imaging to plan the insertion path of instruments. Ultrasound imaging allows clinicians to visualize organs and tissues in the subject and identify regions of interest that may include abnormalities or target organs or tissues. These types of imaging techniques are used when planning to perform biopsies and deliver drugs to patients.

[0006] However, performing ultrasound imaging and subsequent insertion path planning is a time-consuming, cumbersome, and ergonomic process that is a burden on clinicians or operators. For example, to perform ultrasound imaging during instrument insertion, the operator must observe the ultrasound image on the screen while simultaneously controlling the insertion path of the medical instrument (e.g., needle, probe, etc.) with one hand and applying the ultrasound probe to the patient's area with the other. This often results in slow instrument insertion because the movement of the insertion device and the probe are constantly manually updated to ensure the instrument follows the desired path. Furthermore, the accuracy of insertion and imaging depends heavily on the individual operator's skill and technique.

[0007] Therefore, there is a need for more accurate methods and systems for determining medical device insertion paths so that the insertion path can be further actively observed during device insertion. Summary of the Invention

[0008] The following is a brief summary of the various examples described herein and is not intended to identify key or important elements or to outline the scope of the claims.

[0009] In some examples, a computer-aided system is provided. This system may include: (i) a repositionable structure; (ii) a probe holder coupled to the repositionable structure; (iii) a probe holder supporting a sensing probe; (iv) an instrument holder coupled to the repositionable structure, the instrument holder supporting an instrument and including an instrument driver configured to control the instrument; and (v) a control system including one or more processors, wherein the control system is operatively coupled to the repositionable structure, the probe holder, and the instrument holder. The control system is configured to: (1) acquire an image of the target volume of the subject from the sensing probe; (2) reconstruct a three-dimensional (3D) representation of the target volume from the image of the target volume; (3) segment the 3D representation of the target volume to identify one or more objects, boundaries, or features in the 3D representation; (4) determine an instrument insertion path including an insertion site and a treatment site; and (5) control the repositionable structure and the instrument holder to advance the instrument along the insertion path.

[0010] In another example, a method for performing ultrasound-guided robotic percutaneous instrument insertion is provided. The method includes: (1) obtaining an image of a target volume of a subject by a sensing probe; (2) reconstructing a three-dimensional (3D) representation of the target volume from the image of the target volume by a control system; (3) segmenting the 3D representation of the target volume by the control system to identify one or more objects, boundaries, or features in the 3D representation of the target volume; (4) determining an instrument insertion path including an insertion site and a treatment site by the control system; and (5) controlling a repositionable structure and an instrument holder by the control system to advance the instrument along the insertion path, wherein the position of the sensing probe is supported by a probe holder coupled to the repositionable structure.

[0011] In another example, a non-transitory computer-readable medium on which instructions are stored is provided. When executed by one or more processors, the instructions cause one or more processors to: (1) obtain an image of the target volume of the subject by a sensing probe; (2) reconstruct a three-dimensional (3D) representation of the target volume from the image of the target volume by a control system; (3) segment the 3D representation of the target volume by the control system to identify one or more objects, boundaries, or features in the 3D representation of the target volume; (4) determine an instrument insertion path including an insertion site and a treatment site by the control system; and (5) control a repositionable structure and an instrument holder by the control system to advance the instrument along the insertion path, wherein the position of the sensing probe is supported by a probe holder coupled to the repositionable structure.

[0012] It should be understood that both the foregoing general description and the following detailed description are explanatory and illustrative in nature and are intended to provide an understanding of this disclosure, without limiting its scope. In this regard, additional aspects, features, and advantages of this disclosure will be apparent to those skilled in the art from the following detailed description. Attached Figure Description

[0013] Figure 1 These are schematic diagrams of robot-assisted manipulator systems based on some examples.

[0014] Figure 2A It is a perspective view based on some example manipulator systems.

[0015] Figure 2B It is based on some examples Figure 2A Alternative perspective view of the control system.

[0016] Figure 2C It is based on some examples Figure 2A Alternative perspective view of the control system.

[0017] Figure 3 This is a perspective view of an instrument retainer for a manipulator system, based on some examples.

[0018] Figure 4 These are diagrams illustrating high-frequency and low-frequency vibration mechanisms for instrument actuators used in instrument holders, based on some examples.

[0019] Figure 5 These are schematic diagrams of manipulator systems based on some examples.

[0020] Figure 6 This is an example flowchart based on some examples of an example method for performing percutaneous instrument insertion using an ultrasound-guided robot.

[0021] Figure 7 This is a diagram illustrating the implementation of inserting an ultrasound-guided robot percutaneously into a target area according to some examples.

[0022] Figure 8 This is a schematic diagram of a target area with multiple insertion paths for performing percutaneous instrument insertion by an ultrasound-guided robot, based on some examples.

[0023] Figure 9 This is an example flowchart of an example method for performing multi-stage treatment via ultrasound-guided robotic percutaneous instrument insertion, based on some examples.

[0024] Examples of this disclosure and its advantages can be better understood by referring to the following detailed description. It should be understood that the same reference numerals are used to identify one or more of the same elements shown in the drawings, which are illustrated for illustrative purposes and not for limiting the scope of this disclosure. Detailed Implementation

[0025] In the following description, specific details are set forth in relation to some examples conforming to this disclosure. Numerous specific details are set forth to provide a thorough understanding of the examples. However, it will be apparent to those skilled in the art that some examples can be practiced without some or all of these specific details. The specific examples disclosed herein are intended to be illustrative and not restrictive. Those skilled in the art can implement other elements, though not specifically described herein, that fall within the scope and spirit of this disclosure. Furthermore, to avoid unnecessary repetition, one or more features shown and described in association with one example may be incorporated into other examples unless otherwise specifically described or if one or more features would render the example inoperable. In some cases, well-known methods, processes, components, and circuits have not been described in detail to avoid unnecessarily obscuring aspects of the examples.

[0026] This disclosure describes various instruments and instrument parts based on their states in three-dimensional space. As used herein, the term "position" refers to the location of an object or part of an object in three-dimensional space (e.g., three translational degrees of freedom along Cartesian x, y, and z coordinates). As used herein, the term "orientation" refers to the rotational placement of an object or part of an object (e.g., one or more rotational degrees of freedom, such as roll, pitch, and yaw). As used herein, the term "pose" refers to the position of an object or part of an object in at least one translational degree of freedom and the orientation of the object or part of an object in at least one rotational degree of freedom (e.g., up to six total degrees of freedom). As used herein, the term "shape" refers to the set of poses, positions, and / or orientations measured along an object. As used herein, the term "distal" refers to a position closer to the procedural site, and the term "proximal" refers to a position further away from the procedural site. Thus, when an instrument is designed to perform a process, the distal portion or distal end of the instrument is closer to the proximal portion or proximal end of the instrument than the proximal portion or proximal end of the instrument.

[0027] This disclosure occasionally relates to the application of the disclosed techniques to "patients" undergoing a "medical procedure." It should be understood that these references are not intended to limit the application of the disclosed techniques to medical settings. For example, the described techniques can be applied to facilitate physician training, equipment testing and / or calibration, and / or other settings. Therefore, any reference to the term "patient" is for illustrative purposes only, and it is also contemplated that the described techniques be applied to "subjects" in general.

[0028] This disclosure describes various types of image data. In some embodiments, the image data includes 2D images and / or 3D images. Although the term "pixel" is used herein to refer to a specific point within the image data, the term "pixel" should not be construed as implying that the corresponding image data is a 2D image. In embodiments where the image data is 3D image data, the term "pixel" should be understood as referring to a "voxel" of the 3D image.

[0029] Systems and methods are provided that utilize a manipulator arm and a device retainer capable of moving about several degrees of freedom to allow for complex positioning and control. In some examples, the systems and methods control needle insertion and, for example, track the movement of the needle along the insertion path using an ultrasound probe. In these examples, the manipulator arm may have at least four degrees of freedom (e.g., three pivots about a vertical axis and a distal pivot about a horizontal axis), while the retainers for the needle and ultrasound probe may have at least three degrees of freedom (e.g., x-axis, z-axis, pitch, and / or yaw).

[0030] In other or additional examples, the system and method control the movement of the manipulator and instrument retainer based on a remote center around the needle insertion point at the skin level. In these examples, the control system determines the insertion point at the skin and the direction vector of needle insertion, and controls the movement of the manipulator arm and instrument retainer to position the instrument driver so that its insertion axis is aligned with the insertion point along the direction vector. The control system can then operate the instrument driver to drive the needle along the insertion axis to the insertion point at the skin level, and control the movement of the manipulator arm and instrument retainer about the insertion point as a remote center in response to the needle reaching that insertion point.

[0031] The instrument driver for the needle can be configured to impart two degrees of freedom to the needle, including insertion along the insertion axis and rotation about the insertion axis. The system and method may also include a stylus coupled to and driven by the instrument driver via the lumen of the needle. The instrument driver may also impart two degrees of freedom to the stylus, including insertion along the insertion axis and rotation about the insertion axis.

[0032] In the above or alternative examples, the needle driver includes both a high-frequency vibration mechanism and a low-frequency vibration mechanism coupled to the needle. The high-frequency vibration mechanism can be used to assist needle insertion through tissue and to aid in ultrasound visualization of the needle. In one example, the high-frequency vibration mechanism includes a stack of piezoelectric components housed within a housing coupled to the needle. The low-frequency vibration mechanism can be used to reduce pain during needle insertion. In one example, the low-frequency vibration mechanism can be an eccentric motor or linear actuator coupled to the needle. The needle driver can serve as a component of the instrument retainer throughout the system. The system may also include a manipulator arm and a probe retainer.

[0033] The aspects of this disclosure may be part of a computer-aided teleoperation manipulator system (sometimes referred to as a robot-aided manipulator system or a robot system). A manipulator system may include one or more manipulators that can operate with the aid of an electronic controller (e.g., a computer) to move one or more instruments and control the function of one or more instruments when coupled to the manipulator.

[0034] Figure 1 An embodiment of a robot-assisted manipulator system for use with the tools described herein is shown. The manipulator system can be used, for example, in surgical, diagnostic, therapeutic, biopsy, or non-medical procedures, and is generally indicated by reference numeral 100. Figure 1As shown, the robot-assisted manipulator system 100 may include one or more manipulator assemblies 102 for operating one or more medical device systems 104 while performing various procedures on a patient P positioned on a table T in a medical environment 101. For example, the manipulator assembly 102 may drive the movement of a catheter or end effector, apply treatment to target tissue, and / or manipulate control members. The manipulator assembly 102 may be a teleoperated assembly, a non-teleoperated assembly, or a hybrid teleoperated and non-teleoperated assembly, having selectable degrees of freedom of motion that can be motorized and / or teleoperated, and selectable degrees of freedom of motion that can be de-motorized and / or non-teleoperated. An operator input system 106, which may be internal or external to the medical environment 101, typically includes one or more control devices for controlling the manipulator assembly 102. The manipulator assembly 102 supports the medical device system 104 and may optionally include multiple actuators or motors that drive inputs on the medical device system 104 in response to commands from the control system 112. The actuator may optionally include a drive system that, when coupled to the medical device system 104, can advance the medical device system 104 into a natural orifice or surgically created anatomical orifice. Other drive systems can move the distal end of the medical device in multiple degrees of freedom, which may include three linear degrees of motion (e.g., linear motion along the X, Y, Z Cartesian axes) and three rotational degrees of motion (e.g., rotation about the X, Y, Z Cartesian axes). The manipulator assembly 102 may support various other systems for irrigation, treatment, or other purposes. Such systems may include fluid systems (including, for example, reservoirs, heating / cooling elements, pumps, and valves), generators, lasers, demodulators, ultrasonic gel reservoirs and / or dispensers, and ablation components.

[0035] The robot-assisted manipulator system 100 also includes: a display system 110 for displaying images or representations of surgical sites; and a medical device system 104, the images or representations being generated by an imaging system 109, which may include an imaging system such as an ultrasound imaging system. The display system 110 and the operator input system 106 can be configured such that an operator can remotely control the medical device system 104 and the operator input system 106 using remote presence sensing. A graphical user interface can be displayed on the display system 110 and / or on a separately planned workstation display system.

[0036] In some examples, the ultrasound imaging system components of imaging system 109 may be integrally or removably coupled to medical device system 104. However, in some examples, a separate imaging device (such as an endoscope, 3D video, or infrared camera) attached to a separate manipulator assembly may be used with medical device system 104 to image a surgical site. Ultrasound imaging system 109 may be implemented as hardware, firmware, software, or a combination thereof, interacting with or otherwise executed by one or more computer processors, which may include the processor of control system 112.

[0037] The robot-assisted manipulator system 100 may also include a sensor system 108. The sensor system 108 may include a position / positioning sensor system (e.g., an actuator encoder or electromagnetic (EM) sensor system) and / or a shape sensor system (e.g., a fiber optic shape sensor) to determine the position, orientation, rate, velocity, pose, and / or shape of the medical device system 104. The sensor system 108 may also include temperature sensors, pressure sensors, force sensors, or contact sensors, etc.

[0038] The robot-assisted manipulator system 100 may further include a control system 112. The control system 112 includes at least one memory 116 and at least one computer processor 114 for controlling the medical device system 104, the operator input system 106, the sensor system 108, and the display system 110. The control system 112 also includes programming instructions (e.g., a non-transitory machine-readable medium storing the instructions) to perform processes using the robot-assisted manipulator system, including navigation, steering, imaging, engaging feature deployment or retraction, applying treatment to target tissue (e.g., via energy application), etc.

[0039] Optionally, the control system 112 may also include a virtual visualization system to provide navigation assistance to the operator O when controlling the medical device system 104 during an image-guided surgical procedure. Virtual navigation using the virtual visualization system can be based on a reference to an acquired preoperative or intraoperative dataset of the anatomical access. The virtual visualization system processes images of the surgical site imaged using imaging techniques such as computed tomography (CT), magnetic resonance imaging (MRI), fluoroscopy, thermal imaging, ultrasound, optical coherence tomography (OCT), thermal imaging, impedance imaging, laser imaging, nanotube X-ray imaging, etc. The control system 112 can use the preoperative images to locate target tissue (using visual imaging techniques and / or by receiving user input) and create a preoperative plan that includes determining a first location for performing treatment. The preoperative plan may include, for example, planned dimensions for expanding the expandable device, treatment duration, treatment temperature, and / or multiple deployment locations. Other configurations of the teleoperation manipulator system are also envisioned, such as systems configured for multi-port or single-port procedures. For example, the embodiments described herein can be compared with da Surgical systems (such as, da) Surgical system Surgical system or The surgical systems described herein can be used in conjunction with real-time CT guidance (on a CT table), real-time cone-beam CT and X-ray image guidance (in an interventional angiography room), and with an X-ray table and / or operating table.

[0040] Figures 2A to 2C An example embodiment of a manipulator system 200 that can be used as part of a manipulator system 100 is shown. The manipulator system 200 includes a repositionable structure 202 comprising a plurality of links 204 and a plurality of joints 206. The system 200 also includes an instrument holder 208 configured to receive an instrument 210 (e.g., a needle, cannula, etc.) and a probe holder 212 configured to receive a sensing probe 214 having a field of view (e.g., an ultrasound probe). The instrument holder 208 and the probe holder 212 are coupled to one or more of the links 204. With this configuration, the repositionable structure 202 has a plurality of mechanical degrees of freedom, through which the instrument holder 208 and the probe holder 212 can be positioned and controlled to scan a patient with the sensing probe 214 and insert the instrument 210 into the patient. In some examples, the instrument holder 208 has at least three degrees of freedom, and the probe holder 212 has at least three degrees of freedom. In some examples, the repositionable structure 202 has at least four mechanical degrees of freedom.

[0041] The control system 216 of the manipulator system 200 Figure 5 The control system 216 is operatively coupled to the repositionable structure 202, the instrument holder 208, and the probe holder 212 to control the operation of the instrument holder 208 and the movement of the repositionable structure 202, the instrument holder 208, and the probe holder 212. In some examples, the control system 216 may correspond to the control system 112 described above.

[0042] The control system 216 is configured to move the instrument 210 along the insertion path and, as the instrument 210 moves along the insertion path, to move the sensing probe 214 so that the instrument 210 remains within the field of view of the sensing probe 214. The repositionable structure 202 is configured such that movement of the repositionable structure 202 moves both the instrument holder 208 and the probe holder 212. As shown, both the instrument holder 208 and the probe holder 212 are coupled to a distal link 204a of the repositionable structure 202 such that, when either the link 204a or the joint 206 moves, the repositionable structure 202 moves both the instrument 210 and the sensing probe 214 via the instrument holder 208 and the probe holder 212.

[0043] In other words, the control system 216 is configured to operate the instrument holder 208 to move the instrument 210 along the insertion path at least by moving the repositionable structure 202 according to a first set of movements and by moving the instrument holder 208 according to a second set of movements. The control system 216 is also configured to move the sensing probe 214 at least by moving the repositionable structure 202 according to a first set of movements and by moving the probe holder 212 according to a third set of movements, so as to keep the instrument 210 within the field of view of the sensing probe 214 as the instrument 210 moves along the insertion path.

[0044] In some examples, the repositionable structure 202 includes a support assembly 218, which includes a base 220 and a support column 222. An arm 224 is connected to the support column 222. The arm 224 and, in some examples, the support column 222 can serve as a manipulator assembly 102. The arm 224 includes a link 204 and a joint 206. As shown, the arm 224 may include a shoulder joint 206a, an elbow joint 206b, and a wrist joint 206c capable of pivoting about a vertical axis. The arm 224 may also include a distal joint 206d capable of pivoting about a horizontal axis. Thus, the arm 224 has four degrees of mechanical freedom, but additional or fewer links 204 and joints 206 can be used to provide a desired number of degrees of mechanical freedom.

[0045] As shown, a drive 226 can be provided between the support column 222 and the arm 224. The drive 226 can be operated to move the arm 224 along the Z-axis (e.g., height) of the support column 222 for additional degrees of motion of the repositionable structure 202.

[0046] The probe holder 212 includes: a drive 228 operable to move the sensing probe 214 along the X-axis; a drive 230 for moving the sensing probe 214 along the Z-axis; and an articulation 232 for pivoting the sensing probe 214 about a pitch axis. This configuration gives the probe holder 212 three degrees of freedom of movement. In some examples, a pressure sensor 234 may be operatively coupled to the probe holder 212, allowing the control system 216 to use data from the pressure sensor 234 to maintain a desired pressure on the patient using the sensing probe 214. The pressure sensor 234 may be part of the sensing probe 214, part of the probe holder 212, or a separate sensor coupled to the probe holder 212, etc.

[0047] The instrument retainer 208 includes: an actuator 236 operable to move the instrument retainer 208 along an X-axis; a joint 238 for pivoting the instrument retainer 208 about a yaw axis; and a joint 240 for pivoting the instrument retainer 208 about a pitch axis. Furthermore, the instrument retainer 208 may include an instrument actuator 242 comprising an insertion drive 244 for moving the instrument 210 along the insertion axis X and, in some examples, a rotation drive 246 for rotating the instrument 210 about the insertion axis X. This configuration gives the instrument retainer 208 four or five degrees of freedom of motion. In some examples, a pressure sensor 248 may be operatively coupled to the instrument actuator 242, allowing the control system 216 to monitor the pressure at the tip of the instrument 210 (e.g., during puncture of the patient's skin and / or during insertion) to adjust for tissue pressure and collect insertion data. This can be helpful when the instrument 210 is transferred from soft tissue to hard tissue (e.g., a tumor nodule) and from hard tissue to soft tissue. For example, pressure sensor 248 can be a force sensor.

[0048] With the above configuration, the instrument holder 208 can move the instrument 210 along the insertion path. Moving the instrument 210 along the insertion path can include both: for the straight portion of the insertion path, using the insertion drive 244 to align the insertion axis X to drive it in and out along the insertion axis X; and when the insertion path includes a non-linear portion, using the rotation drive 246 to rotate the instrument 210 (e.g., when using an instrument with a beveled tip) so that when the instrument 210 is moved using the insertion drive 244, bending along the insertion path is subsequently or simultaneously caused.

[0049] As shown, both the instrument holder 208 and the probe holder 212 can be coupled to the distal link 204a of the repositionable structure 202. Furthermore, X-axis drives 228 and 236 can interact with the distal link 204a to move the probe holder 212 and the instrument holder 208, respectively, along the distal link 204a. For example, the X-axis drives 228 and 236 can be rack and pinion drives utilizing a common rack extending along the distal link 204a.

[0050] Instrument retainer 208 may include one or more vibration mechanisms to aid instrument insertion. Therefore, instrument retainer 208 may include a high-frequency vibration mechanism 250 coupled to instrument 210. High-frequency vibration of instrument 210 can aid in visualization of instrument 210 using sensing probe 214 (e.g., under ultrasound). High-frequency vibration can also reduce the insertion force required for instrument insertion and tissue deformation caused by dragging. Alternatively or additionally, instrument retainer 208 may include a low-frequency vibration mechanism 252 coupled to instrument 210. Low-frequency vibration of instrument 210 can reduce patient pain during instrument insertion by numbing the local area around the insertion path. Vibration of the instrument can also aid in enhanced visualization of instrument 210 in ultrasound imaging. Some instruments 210 (such as needles) may be coated with microbubbles or other elements to enhance visualization and detection of instrument 210 in ultrasound images.

[0051] Furthermore, the vibration of the instrument 210 combined with the rotation of the instrument 210 via the rotation drive 246 can help the instrument 210 penetrate freely floating / moving objects (e.g., lymph nodes or similar unfixed structures inside tissues), rather than unintentionally pushing the object away as is possible with conventional systems.

[0052] In some examples, the frequency range of the high-frequency vibration mechanism 250 may be about 27 kHz to about 44 kHz (which may be advantageous for visibility under ultrasound vision), between about 20 kHz and about 50 kHz (which may be advantageous for reducing insertion force), or about 34 kHz; and the frequency range of the low-frequency vibration mechanism 252 may be about 75 Hz to about 250 Hz (which may be advantageous for pain reduction) or about 166 Hz. The vibration mechanisms 250 and 252 may be part of the instrument retainer 208, coupled to the instrument retainer 208, or separate components directly coupled to the instrument 210, separate from the instrument retainer 208.

[0053] In one example, the high-frequency vibration mechanism 250 may include a plurality of piezoelectric elements 254 housed within a housing 256 coupled to the instrument 210. If desired, the housing 256 may be radially coupled to a proximal mount 258 of the instrument 210, such that the piezoelectric elements 254 are offset relative to the insertion axis X and the instrument 210.

[0054] In one example, the instrument retainer 208 may include a distal instrument guide 260 that supports the instrument 210 adjacent to the insertion positioning. The instrument guide 260 may have any suitable form. For example, the instrument guide 260 may be side-mounted or top-mounted and define the path through which the instrument 210 is to be inserted. A low-frequency vibration mechanism 252 may be coupled to the instrument guide 260 to vibrate the instrument 210 via the instrument guide 260. In some examples, the low-frequency vibration mechanism 252 may be an eccentric motor or a linear actuator.

[0055] In some embodiments, device 210 may define a lumen extending longitudinally through the length of device 210. With this configuration, in some examples, device 210 may be used to deliver fluid at a treatment location. Additionally or alternatively, system 200 may include a second device 262 (e.g., a stent) inserted through the lumen of device 210. The second device 262 can be used for any desired purpose. In some examples, the second device 262 may have a solid cross-section and complementary bevels to provide a solid integral tip between the first and second devices 210, 262. The second device 262 may be a biopsy stent or needle, an anchoring stent, etc. Furthermore, the second device 262 may also be a miniature imaging device, endoscope, OCT, optical imaging laser, etc.

[0056] As described above, the instrument actuator 242 may further include an insertion drive 264 for moving the second instrument 262 along the insertion axis X within the lumen of the first instrument 210, and a rotation drive 266 for rotating the second instrument 262 about the insertion axis X in some examples. This configuration gives the instrument retainer 208 an additional one or two degrees of freedom of motion. In one example, the insertion drives 244, 264 may be rack and pinion drives utilizing a common rack.

[0057] If desired, the high-frequency vibration mechanism 268 can be coupled to the second device 210 to provide similar benefits as described above. For example, the high-frequency vibration mechanism 268 can be coupled to the proximal mount 270 of the second device 262. The high-frequency vibration mechanism 268 can be any suitable mechanism, including multiple piezoelectric elements, ultrasonic transducers, etc., as described above.

[0058] like Figure 5As shown, control system 216 can be configured to control the movement and operation of components of control system 100. These include support assembly drive 226, joint 206 of arm 224, drive 228, 230 and joint 232 of probe holder 212, drive and joint 236, 238, 240, 244, 246, 264, 266, and vibration mechanisms 250, 252, 268 (if included).

[0059] In some examples, the control system 216 may be configured to control the movement of the repositionable structure 202 (e.g., drive 226 and joint 206) and the instrument retainer 208 (e.g., drive 236 and joints 238, 240) to selectively move and manipulate around a remote center during the procedure. For example, the remote center may correspond to an insertion point along the insertion path into the skin of a patient. This configuration allows the system 100 to move as needed during the procedure to insert the instrument 210 along the insertion path while minimizing or avoiding damage at the insertion point at the skin level. In one example, the control system 216 may be configured to control the movement of the repositionable structure 202 and the instrument retainer 208 around the remote center to deflect the insertion path, such as by rotating the instrument 210 with rotation drive 246.

[0060] As described above, the control system 216 can be configured to control the movement of the repositionable structure 202 and the instrument holder 208 about a remote center, which corresponds to the instrument insertion point in the patient at the skin site. The movement about the remote center imposes a set of constraints on at least one of the following: one or more degrees of freedom of motion of the repositionable structure 202 or one or more degrees of freedom of motion of the instrument holder 208.

[0061] Furthermore, the control system 216 can be configured to control the movement of the repositionable structure 202 and the probe holder 212 to perform an initial scan of the target area within the patient using the sensing probe 214. The initial scan can be used to determine the insertion path. For example, the control system 216 can utilize the initial scan to determine the instrument insertion point and the direction vector of instrument insertion.

[0062] To allow system 100 to complete a full scan, an initial scan can be performed without imposing a set of constraints on a set of degrees of freedom of motion of the repositionable structure 202 or the instrument holder 208, which are imposed during movement around a remote center. Therefore, the remote center function of system 100 can be disabled for the initial scan. In the case of an ultrasound scan, the sensing probe 214 should be in physical contact with the patient to deliver ultrasound imaging energy, with a range of pressure between the sensing probe 214 and the patient. To achieve this, system 100 can utilize various degrees of freedom to maintain pressure, such as that measured by pressure sensor 234 coupled to instrument driver 242. In an embodiment, an ultrasound gel can be used to improve contact between the sensing probe 214 and the patient to help deliver imaging energy to the patient.

[0063] Subsequently, the remote center function can be activated in response to the device 210 reaching the insertion point, after which the control system 216 controls the movement of the repositionable structure 202 and the device holder 208 around the remote center. The insertion point at the skin can be determined based on data from the pressure sensor 234, the length of the device 210 driven forward by the insertion drive 244, etc. Furthermore, control over the movement around the remote center can be maintained until the device 210 is removed from the patient.

[0064] In some examples, the control system 216 may control the operation of the instrument holder 208 and the probe holder 212 to move both the instrument 210 along the insertion path and the sensing probe 214 to keep the instrument 210 within the field of view as it moves along the insertion path. This functionality may include the control system 216 controlling the movement of the repositionable structure 202 around a remote center and controlling the movement of the probe holder 212 to keep the instrument 210 within the field of view as it moves along the insertion path.

[0065] In an example where the sensing probe 214 is an ultrasound probe with a 2D field of view, the control system 216 can be configured to move the sensing probe 214 to keep the tip of the instrument 210 within the ultrasound 2D slice at all times during insertion.

[0066] In some examples, system 100 can be configured to insert multiple instruments 210 into a patient by decoupling the inserted instruments and loading subsequent instruments 210. Accordingly, instrument driver 242 may include one or more slots 272 to allow instruments 210 to be unloaded from instrument driver 242. Alternatively, instrument driver 242 may include releasable chucks to allow instruments to be decoupled from instrument driver 242. Furthermore, in examples with a high-frequency vibration mechanism, housing 256 may also include slots 274 or releasable connectors for radial coupling.

[0067] Figure 6 This is a flowchart of a method 600 for performing ultrasound-guided robotic percutaneous instrument insertion, based on some examples. Method 600 can use... Figure 1 The method 600 can be performed by the robot-assisted manipulator system 100, the manipulator system 200 of Figure 2, or by another manipulator system. The method 600 can be performed by a control system that executes instructions stored in one or more computer-readable media (e.g., non-volatile memory). Figure 5 The control system 216) is executed by one or more processors or by another processor. Figure 7 and Figure 8 This is a diagram of the target volume 700 of subject 702, which illustrates an example implementation of method 600. For clarity, reference will be made to... Figure 1 Figure 2 Figure 7 and Figure 8 Method 600 is described by its components.

[0068] At box 602, the system acquires one or more images of a target volume of 700. For example... Figure 7 As shown, sensing probe 214 is positioned relative to or against subject 702 to obtain an image of target volume 700. The image data can be preoperative or intraoperative image data. In some embodiments, the image data is 3D image data, such as computed tomography (CT) image data, cone-beam CT (CBCT) image data, positron emission tomography (PET) image data, ultrasound image data, or magnetic resonance imaging (MRI) image data. In examples, sensing probe 214 may include one or more of an ultrasound probe, photoacoustic probe, microscope, optical imaging probe, video camera, infrared camera, handheld X-ray system, electronic palpation device, pulse radar detector, focused ultrasound probe, gamma probe, perfusion imaging system, and optical coherence tomography device. In an example, sensing probe 214 includes an ultrasound probe with a field of view (FOV) 715 for capturing images of an object (such as object 705) within the FOV 715.

[0069] At box 604, the system reconstructs a 3D representation of the target volume 700. The 3D representation of the target volume 700 includes multiple voxels and may include voxels in free space excluding tissue, voxels including the tissue of subject 702, and voxels that may include a portion of both free space outside subject 702 and the tissue of subject 702. System 100 uses both the FOV of the ultrasound probe and information and data relating to the sensing probe 214 relative to the ultrasound probe or relative to other image FOVs and the position and orientation of the scan to construct the 3D representation. The method also includes segmenting the 3D representation at box 606 to identify one or more objects, boundaries, or features in the 3D representation of the target volume. Examples of identifiable objects and features include, but are not limited to, one or more tissues, organs, fluids, blood vessels, bones, medical structures (e.g., implanted devices, stents, medical instruments, etc.). In implementation, segmenting the 3D representation may also include a processor or control system 216 identifying the treatment site 712 of subject 702.

[0070] At frame 608, control system 216 determines the instrument insertion path. Control system 216 determines an insertion site 710 for inserting the instrument into subject 702 and further identifies treatment site 712 during segmentation or from additional processing of the 3D representation. Treatment site 712 is a location within subject 702 where instrument 210 or at least a portion of instrument 210 (such as the tip of a needle or the distal end of an instrument) will be reached to provide treatment. Treatment may include: delivering medication to treatment site; providing treatment to treatment site; performing a biopsy at treatment site; drawing blood; delivering electronic leads, microrobots, or microsensors; delivering energy; providing gene therapy; delivering PICC lines; delivering dialysis fistulas; delivering hydrogels; delivering implants; placing needle electrodes; extracting tissue; delivering tissue; measuring tissue; and / or destroying tissue. Treatment site 712 may include organs, tissues, tumors, cancer, blood vessels, living tumors, epidermis, ligaments, muscles, sinuses, brain, pain sources or regions, nerves, lymphatic vessels, or any part or subpart thereof. In some implementations, the method may include receiving a user-identified treatment site from a 3D representation of the target volume. For example, the user may designate a specific area as the treatment site, or designate an object, boundary, feature, organ, etc., as the treatment site 712. The user may provide such designation via a user interface (such as a keyboard, mouse, or touchscreen), via a network, etc.

[0071] To determine one or more of the following, (i) instrument insertion path, (ii) insertion site 710, and / or (iii) treatment site 712, the control system 216 may determine two or more potential insertion paths and select the insertion path from two or more candidate paths. For example, as Figure 8As shown, the control system can determine a first insertion path 720a having a first insertion portion 710a, a second insertion path 720b having a second insertion portion 710b, and a third insertion path 720c having a third insertion portion 710c. Each of the first insertion path, the second insertion path, and the third insertion paths 720a to 720c provides a path to the treatment site 712.

[0072] When determining the insertion path, the control system 216 can select the insertion path based on the path length into the subject 702 or the number of voxels traversed by the path in the 3D representation. For example, an insertion path may include voxels with free space, voxels with both free space and tissue, and three voxels with tissue to reach the treatment site 712. Another second path may not include voxels with free space, but includes four voxels with tissue to reach the treatment site 712. Due to the reduced distance of the entire insertion path, the determined path can be a second path with a total of four voxels. Conversely, the determined insertion path can be a first path with a total of five voxels, wherein only three voxels are in the tissue of the subject 702, and this first path can be selected to reduce the distance of the insertion path within the subject. The determined path can be determined by the number of voxels with free space in the 3D representation, or the determined insertion path can be determined by the number of voxels in free space and the number of voxels with tissue.

[0073] Determining the insertion path and site can be based on the location of organs and tissues, and can also be designed to avoid identified objects, such as bones, blood vessels, organs, and tissue areas. Although in Figure 8 The path shown is a linear insertion path, but the insertion path can include one or more curves or arcs (see reference). Figure 7 (Further description), and the path determination can be based on the amount of curvature or the direction of curvature of the insertion path. The control system 216 can determine the path based on the instrument type (e.g., needle, endoscope, implant, catheter, PICC line, stylus) or the type of needle into which the instrument is inserted.

[0074] Users can also provide user preferences, and the path determination can be based on these preferences. For example, a user can select an insertion path from multiple candidate insertion paths. Users can provide specific tissues, organs, or areas within the patient's body to be avoided during device insertion. Similarly, users can define one or more restricted areas in 3D representation to exclude any insertion path from traversing these restricted areas. User preferences may include the maximum or minimum distance of the insertion path into subject 702, the maximum or minimum curvature of the insertion path, a preference for straight paths, a user-identified area for a specific insertion site or one or more insertion sites, restrictions on the insertion site or prohibited insertion site, and / or a user-defined treatment site 712.

[0075] When determining an insertion path or two or more candidate insertion paths, the control system may also determine a restricted region in the 3D representation. The restricted region indicates a region in the 3D representation, and also a region of the subject 702, that will not be traversed by the determined insertion path and / or two or more candidate insertion paths. The restricted region is a 3D volume, and the control system 216 may determine a candidate insertion path as a path that does not intersect with or traverse any volume within the restricted region. The processor may automatically determine the restricted region based on one or more objects, boundaries, or features in the 3D representation.

[0076] As shown, each potential insertion path 720a to 720c can have its own independent corresponding insertion portion 710a to 710c. Conversely, two or more candidate insertion paths 720a to 720c can have the same insertion portion. For example, the fourth curved insertion path 720d can have the same insertion portion 710a as the first insertion path 720a. Therefore, a single insertion portion 710a can be used for more than one insertion path.

[0077] like Figure 7As shown, the insertion path 720 may include one or more curves or arcs. To insert the device 210 along a curved path, the device 210 may be inserted into the subject 702 at a first position 730a. While the device 210 is being inserted into the patient, the distal end of the device may be imaged by the sensing probe 214 to monitor the position of the device 210 along the insertion path 720. At some point during insertion, the device 210 may be repositioned to a second position 730b at an angle relative to the first position 730, causing the device to bend along the first arc 721a. At another point during insertion, the device 210 may again be repositioned to a third position 730c at an angle relative to the first and second positions 730a and 730b, causing the device 210 to bend along the second arc 721b to reach the treatment site 712. Therefore, the insertion path may include one or more curves or arcs to reach the treatment site, avoid restricted areas, avoid objects, reduce the number of voxels traversed, etc. In addition, the sensing probe 214 can continuously image the FOV 715 to observe the position of the distal end of the instrument, thereby providing feedback for controlling the position of the instrument and achieving a specific insertion path depth and / or path curve.

[0078] In the example, method 600 may optionally include acquiring preoperative imaging data at box 610 and merging the preoperative imaging data and a 3D volume representation to provide a combined image of the target volume 700. The preoperative image data may include one or more of CT scan data, MRI scan data, cone-beam CT data, and / or ultrasound data. The combined image of the target volume may allow imaging of some tissues, cavities, and structures, or allow for better imaging of areas of the patient. For example, ultrasound imaging does not penetrate gaseous areas, such as in the lungs. Ultrasound imaging can image the surface of the pneumopulmonary lungs, and when combined with a CT scan, the combined image shows the lung surface and most of the air-filled cavities of the lung structures. In another example, bone may obscure ultrasound imaging of a portion of an organ behind the bone; when combined with an MRI scan, the resulting image allows imaging of the organ obscured by the skeletal structure.

[0079] Preoperative MRI or CT scans can provide higher image quality and potentially reveal additional features or targets that might be difficult to image using ultrasound. Furthermore, preoperative imaging sets may have a larger field of view and can be analyzed by the operator to pre-select targets such as tumors. Matching preoperative FOV imaging data with ultrasound image datasets can guide ultrasound probes or sensors toward targets that may be difficult to reach. Data or image fusion can be performed using rigid or non-rigid registration (e.g., vessel-based registration). Because the preoperative dataset is not updated with real-time anatomical changes, ultrasound scans can provide up-to-date, real-time information. For one example, during an ablation procedure, preoperative scans can be combined with postoperative 3D ultrasound volumes to determine if the tumor has been completely ablated.

[0080] At box 612, the control system 216 or the user can adjust and determine the instrument insertion path based on a combined image of the target volume. For example, preoperative data can also provide insights into tissue density, objects, features, or boundaries that the original 3D representation might not capture. Therefore, the instrument insertion path can be updated or otherwise adjusted based on information collected from a combined image of two different images and / or a 3D representation of the target region 700.

[0081] In the example, method 600 may further include receiving pressure data from a pressure sensor 248 coupled to instrument driver 242, wherein the pressure data indicates the pressure applied to the instrument. Control system 216 may then determine the current instrument position or trajectory based on the received pressure data. For example, control system 216 may determine that the instrument has reached a boundary between different types of tissue or an organ surface based on changes in the pressure data, and may also determine the position of the instrument along the insertion path. Furthermore, control system 216 may determine from the pressure data that the instrument position is not along the desired insertion path, and the insertion trajectory may be updated to compensate for the current position of the instrument. The pressure data and the current position or trajectory of the instrument may be used to update the insertion trajectory or to adjust control of the instrument to compensate for any deviation from the desired insertion trajectory.

[0082] System 100 can continuously perform ultrasound scans to image and analyze patient movement due to breathing by imaging the distance and position between the target and the needle tip or instrument 210, in order to correct the path or trajectory of instrument 210 based on patient breathing. System 100 can determine patient movement from functions such as breathing, heartbeat, digestion, or other biological movements or functions. Furthermore, patient breathing can be measured by placing sensors on the patient's body, extracting from an ECG, determining during estimation, and having the patient take a deep breath or providing a sample of breathing patterns, from a pressure sensor 234 coupled to sensing probe 214, or by other means. Additionally, the system can move instrument 210 or needle only during certain respiratory or movement phases (e.g., only during exhalation).

[0083] At box 614, method 600 includes controlling a repositionable structure 202 and an instrument retainer 208 to advance instrument 210 along a defined or identified insertion path. Instrument 210 is advanced until it reaches treatment site 712 and administers treatment. The instrument is then removed from subject 702 along the insertion path. In an implementation using a second instrument 262 (such as a stylus), the stylus can be retracted separately from instrument 210 (such as a needle) while the needle remains in the patient to create a channel within the needle for delivering additional treatment. In the example, instrument 210 can also be rotated or vibrated to advance instrument 210 along the insertion path to help reduce induced pain, reduce the required insertion force, move from one tissue to another, or insert instrument 210 into denser tissue, harder tissue or object, rolling or movable object or target (e.g., lymph node), etc. The pressure sensor 234 coupled to the instrument retainer 208 can be used to provide a closed feedback loop, and the needle insertion path or trajectory can be updated or further determined based on the force derived from the pressure sensor feedback loop.

[0084] The probe holder 212 can also be controlled to reposition the probe to perform one or more update scans or image sweeps of the target area 700 during insertion of the instrument 210. In the example, method 600 may further include controlling at least one of the repositionable structure 202 and the probe holder 212 to hold the position of the instrument 210 within the FOV 715 of the sensing probe 214. The repositionable structure 202 and the probe holder 212 together control the position and orientation of the sensing probe 214, and thus, the sensing probe 214 can move and reorient during insertion of the instrument 210 to actively image the position of the instrument 210 as it moves along the insertion path. When controlling the repositionable structure 202, method 600 may include moving the repositionable structure 202 according to a first set of movements that cause simultaneous movement of both the probe holder 212 and the instrument holder 208. Once repositioned, sensing probe 214 can then acquire images of device 210 (e.g., a portion of the device, the distal end of the device, or another portion of the device) to track and provide feedback on the position and trajectory of device 210 during advancement of device 210 along the insertion path. Sensing probe 214 can be actively repositioned as device 210 is inserted, wherein sensing probe 214 acquires real-time images to track the position and / or trajectory of device 210. In a particular example, method 600 may include cooperatively moving probe holder 212 and device holder 212 such that the distal end of device 210 is held within the FOV 715 of sensing probe 214 during movement of device 210. A control system may synchronously move sensing probe 214 and device 210 to hold the distal end of device 210 within the 2D imaging plane of sensing probe 214. In an embodiment, the distal end is held within the 2D imaging plane as it is inserted from the insertion site toward the treatment site or retracted from the treatment site toward the insertion site. Furthermore, the movement of the repositionable structure 202 and the probe holder 208 can be controlled to maintain the 2D planar imaging depth, while also maintaining the sensing probe 214 applying a given pressure to the subject 702 or the target during scanning of the target volume 700.

[0085] Method 600 may include performing one or more updated image sweeps (also referred to herein as “updating real-time probe sweeps”). For example, method 600 may also include controlling instrument driver 242 to pause advance of instrument 210. The repositionable structure 202 and / or probe holder 212 may then be controlled to perform updated image sweeps of target region 700 to obtain multiple images representing the real-time condition of the target region. Updated image sweeps may be performed by sweeping the same region with the same pressure between sensing probe 214 and subject 205, or by performing updated image sweeps with different pressures for subject 205 or at different angles from the viewpoint of target region 702. Control system 216 may then construct an updated 3D representation of the target volume from the acquired images. The updated 3D representation may be used to update a previous 3D representation, which may be stored in memory or provided concurrently with the updated 3D representation for comparison. The updated 3D representation can reflect the current position and / or trajectory of the instrument 210, as well as any movement of the organ and tissue relative to the insertion path and the instrument 210. The insertion path can then be updated based on the updated 3D representation.

[0086] Updated image sweeps can be performed periodically based on predefined parameters or settings. For example, an updated image sweep can be performed in response to the instrument advancing a predefined distance along the insertion path. Predefined distances can be on the order of millimeters, less than 1 cm, approximately 2 cm, approximately 3 cm, between 3 and 5 cm, 10 cm, or greater than 10 cm. Updated probe sweeps can be performed based on given time intervals (e.g., every half second, every second, every 5 seconds, every 10 seconds, every 30 seconds, etc.). In some examples, updated image sweeps can be performed based on pressure data received from pressure sensor 248. The pressure data may include one or more unexpected pressure values ​​indicating that the instrument may not be at the desired insertion path location, or that the instrument has encountered a restricted area, tissue, or organ. Therefore, the repositionable structure 202 and the instrument actuator 242 can stop the insertion of instrument 210 to perform an updated image sweep to update the insertion path based on an updated 3D representation (and / or verify the current insertion path).

[0087] In implementation, method 600 may further include repositioning the repositionable structure at block 616 and moving the sensing probe to a new position to image different portions of the target volume 700, different orientations of the target volume 700, different angles or fields of view of the target 705, or different areas of the subject 702. At block 618, control system 216 identifies the position of the target volume 700 relative to the new position of sensing probe 214. Control system 216 may store previous spatial coordinates in memory and compare the previous spatial coordinates with the new repositioned spatial coordinates to determine the position of the target volume 7090 relative to the new position of sensing probe 214. Control system 216 may map the relative position of the target volume 700 based on the known position of sensing probe 214, or through the repositioning vector between the old position of sensing probe 214 and the repositioned position and orientation. In the example, to identify the position of the target volume relative to the new position of the sensing probe, control system 216 may receive user input to identify the positioning of the target volume 700, such as via a user interface or by providing the spatial coordinates of the new position of the target volume or sensing probe.

[0088] At block 620, probe holder 212 repositions sensing probe 214 such that at least a portion of target volume 700 is within the field of view (FOV) 715 of sensing probe 214. At block 622, sensing probe acquires one or more images of target volume 700 from the new position and orientation of sensing probe 214. Sensing probe 214 can be controlled to perform sweeping within the FOV 215 of sensing probe 214 to acquire multiple images of the target volume. Control system 216 can then reconstruct a new 3D reconstruction of the target volume based on the multiple images obtained from the sweeping of sensing probe 214.

[0089] The method may also include receiving pressure data from a pressure sensor 248 operatively coupled to the sensing probe 214, indicating the pressure applied to the probe. Based on the pressure data indicating the pressure applied to the probe 214, the control system 216 may control the probe holder 212 to maintain the desired pressure of the sensing probe 214 on the subject 702.

[0090] In the example, method 600 may further include controlling the repositionable structure 202 and the device holder 208 to remove the device 210 along the device insertion path. During removal, the controller may control the repositionable structure 202 and the probe holder 212 to hold the distal end of the device 210 within the field of view (FOV) of the sensing probe 214. After the device 210 is removed from the subject 702, the device 210 may be replaced by another device, and the controller may control the repositionable structure 202 and the device holder 208 to insert the second device into the subject along the device insertion path. The second device may be inserted to further provide treatment to the treatment site 712 of the target 705, or to observe the treatment site 712, or to extract tissue or a portion of the target 705.

[0091] The method 600 described above can be implemented to perform a multi-stage process by using a first instrument to perform the first stage of the treatment process and using a second instrument to perform the second stage of the treatment process. Figure 9 This is an example flowchart of an example method 900 for performing ultrasound-guided robotic percutaneous instrument insertion to perform multi-stage treatment.

[0092] Method 900 includes acquiring multiple images of a target volume 700 with a target 705 at box 902 for providing multi-stage treatment. Sensing probe 214 acquires the multiple images and reconstructs a 3D representation of the target volume 700 at box 904. At box 906, control system 216 segments the 3D representation to identify objects, boundaries, and features within the 3D representation. At box 908, control system 216 then determines one or more instrument insertion paths from the 3D representation and the identified objects, boundaries, and / or features.

[0093] At frame 910, the repositionable structure 202 and instrument holder 208 are inserted into the first instrument along the first-stage insertion path. The first instrument provides first-stage treatment to treatment site 712. First-stage treatment can provide multi-stage treatment to a target area or target or patient. For example, multi-stage treatment may include first-stage treatment including imaging the target or target area using an ultrasound probe at the needle tip, and then replacing the ultrasound probe with a drug delivery catheter. Many electrotherapy treatments for tumors require multiple electrodes to be placed around the tumor volume to treat the entire tumor, which also requires multiple stages of electrode delivery. Another multi-stage treatment may include placing a catheter at the target site and then delivering analgesics through the catheter. Furthermore, multiple biopsies can be performed as multi-stage treatment to retrieve sufficient tissue using a smaller needle. Additionally, multi-stage treatment may include repositioning or recalculating the needle trajectory. For example, the first stage may include inserting the needle through the skin and into the first two to three centimeters of tissue. The needle trajectory can be reassessed at this point to determine whether it still relates to the planned trajectory along the path to the target. If unrelated, system 100 can adjust the angle of the needle to align the path towards the target and continue inserting the needle further.

[0094] At frame 912, the repositionable structure 202 and the instrument holder 208 remove the first instrument along the first-stage insertion path. The first instrument is then replaced with a second instrument, or the second instrument with the second instrument is inserted into the subject 702.

[0095] In the example, method 900 can be performed using a single insertion path or multiple insertion paths. For a single insertion path, at block 914, the repositionable structure 202 and instrument holder 208 insert a second instrument along the first-stage insertion path. The second instrument is used to provide a second treatment to treatment site 712. The second treatment may include performing one or more of the following: delivering medication to the treatment site; providing treatment to the treatment site; performing a biopsy at the treatment site; drawing blood; delivering electronic leads, microrobots, or microsensors; delivering energy; providing gene therapy; providing PICC lines; providing dialysis fistulas; delivering hydrogels; delivering implants; placing needle electrodes; extracting tissue; delivering tissue; measuring tissue; and / or destroying tissue. At block 916, the repositionable structure 202 and instrument holder 208 remove the second instrument along the first insertion path. Additionally, a needle cannula may remain in the first insertion path, and other instruments (such as biopsy tools or energy delivery tools) may be inserted through the needle cannula to perform additional treatments.

[0096] Furthermore, multiple needles can be inserted into a patient to provide one or more treatments. For example, a first needle can be inserted into a tumor to deliver energy to it. The needle can remain in the inserted tumor and be decoupled from system 100. System 100 can then be loaded with a second needle. The second needle can then be inserted into the patient or a tumor adjacent to the first needle, according to the treatment plan. This is particularly useful in delivering energy-based treatments, where a single needle is used on an ellipsoidal volume to treat a volume smaller than the tumor volume, where multiple needles need to be delivered to cover the entire tumor, which may be up to ten needles in current technology.

[0097] In other examples, the multi-stage treatment method 900 can utilize multiple insertion paths to perform multiple stages of treatment. Multiple insertion paths can utilize paths traversing the fewest voxels in a 3D representation. Utilizing multiple insertion paths allows treatment to be delivered to different areas or regions of a target, for example, performing multiple tumor biopsies on different parts of a tumor. Furthermore, the insertion sites of the multiple insertion paths can be determined by the control system 216 or by input from the user. At block 918, the repositionable structure 202 and instrument holder 208 insert a second instrument into a second insertion path different from the first insertion path. Figure 8 As shown, the first insertion path can be a first insertion path 720a, and the second insertion path can be a second insertion path 720b. Therefore, the second insertion path 720b has a second insertion site 710b that is different from the first insertion site 710a. Both the first insertion path and the second insertion paths 720a and 720b provide paths to the first and second instruments to provide treatment to the same or nearby treatment site 712 of the target 705. At block 920, the repositionable structure 202 and the instrument holder 208 remove the second instrument along the second insertion site 720b.

[0098] In the example, method 900 may further include inserting a third device during multi-stage treatment to provide an additional treatment stage. The third device may then be inserted along a first insertion path 720a, a second insertion path 720b, or a third insertion path different from the first and second insertion paths 720a and 720b. For some multi-stage treatments, it may be necessary to insert the first and second devices through corresponding insertion paths to provide simultaneous treatment at treatment site 712 of target 705. For example, the first treatment may include providing active anesthesia to the treatment site, while the second treatment performs a biopsy at or near treatment site 712 on target 705. Furthermore, the first and second stages may utilize the same insertion site having different insertion paths to the same treatment site or to two different treatment sites. In another example, the first and second devices are identical devices that can be used to perform multiple stages of treatment, such as providing two doses of medication or performing two biopsies. Furthermore, the same instrument can be used in the first and second stages with different insertion sites to provide treatment to different areas of the target 705, or to different sides of the treatment site 712, or to provide treatment from different angles of the treatment site 712.

[0099] In some examples, the control system 216 may also provide recommendations for instrument types based on a determined instrument insertion path. For example, the control system 216 may provide recommendations for needle types for a given insertion path. A straight insertion path may allow for thicker or more rigid needle types, while a curved insertion path may require thinner needles or needles made of softer materials. Furthermore, various needle tips (e.g., sharp blunt tips, tapered, conical, beveled, etc.) can be used to traverse different types of tissues and perform different types of treatments. The control system 216 may suggest needle recommendations that include one or more of the following: needle size, needle width, needle length, needle diameter, needle tip, or needle material.

[0100] One or more components of the embodiments discussed in this disclosure (such as control system 112) may be implemented in software to execute on one or more processors of a computer system. The software may include code that, when executed by one or more processors, configures one or more processors to perform the various functions discussed herein. The code may be stored in a non-transitory computer-readable storage medium (e.g., memory, magnetic storage device, optical storage device, solid-state storage device, etc.). The computer-readable storage medium may be part of a computer-readable storage device, such as electronic circuitry, semiconductor devices, semiconductor memory devices, read-only memory (ROM), flash memory, erasable programmable read-only memory (EPROM); floppy disk, CD-ROM, optical disk, hard disk, or other storage device. The code may be downloaded via a computer network (such as the Internet, intranet, etc.) to be stored on the computer-readable storage medium. The code may be executed by any of a variety of centralized or distributed data processing architectures. The programming instructions of the code may be implemented as multiple separate programs or subroutines, or they may be integrated into multiple other aspects of the system described herein. Components of the computing system discussed herein may be connected using wired and / or wireless connections. In some examples, wireless connectivity can use wireless communication protocols such as Bluetooth, Near Field Communication (NFC), Infrared Data Association (IrDA), HomeRF, IEEE 802.11, Digital Enhanced Cordless Telecommunications (DECT), and Wireless Medical Telemetry Service (WMTS).

[0101] Various general-purpose computer systems can be used to perform one or more procedures, methods, or functions described herein. Additionally or alternatively, various special-purpose computer systems can be used to perform one or more procedures, methods, or functions described herein. Furthermore, various programming languages ​​can be used to implement one or more of the procedures, methods, or functions described herein.

[0102] While certain embodiments and examples have been described above and shown in the accompanying drawings, it should be understood that such embodiments and examples are merely illustrative and not limited to the specific constructions and arrangements shown and described, as those skilled in the art will understand various other alternatives, modifications and equivalents.

Claims

1. A computer-aided system, comprising: Repositionable structure; A probe holder, coupled to the repositionable structure, supports a sensing probe; An instrument holder coupled to the repositionable structure, the instrument holder supporting the instrument and including an instrument driver configured to control the instrument; as well as A control system comprising one or more processors, wherein the control system is operatively coupled to the repositionable structure, the probe holder, and the instrument holder, the control system being configured to: An image of the target volume of the subject is obtained by the sensing probe; Reconstruct a three-dimensional (3D) representation of the target volume from an image of the target volume; The 3D representation of the target volume is segmented to identify one or more objects, boundaries, or features in the 3D representation; Determine the path including the insertion site and the treatment site; and Control the repositionable structure and the instrument retainer to advance the instrument along the defined insertion path.

2. The computer-aided system according to claim 1, wherein, The sensing probe includes at least one of the following: an ultrasound probe, a photoacoustic probe, a microscope, an optical imaging probe, a video camera, an infrared camera, a handheld X-ray system, an electronic palpation device, a pulse radar detector, a focused ultrasound probe, a gamma probe, a perfusion imaging system, and an optical coherence tomography device.

3. The computer-aided system according to claim 1, wherein, To determine the instrument insertion path, the control system is configured to: Two or more candidate insertion paths are determined, and the determined insertion path is selected by the system or via user input from the two or more candidate insertion paths.

4. The computer-aided system according to claim 3, wherein, In order to select the determined insertion path, the control system is configured to: The path is selected based on at least one of the following: path length, number of voxels traversed by the path in the 3D representation, avoidance of identified objects, curvature associated with the path, instrument type of the instrument, needle type of the needle into which the instrument is inserted, and one or more operator preferences.

5. The computer-aided system according to claim 1, wherein, The 3D representation includes multiple voxels, which include (i) voxels with free space that do not contain tissue and (ii) voxels containing tissue, and wherein the control system determines the determined insertion path based on the number of voxels in free space and the number of voxels containing tissue.

6. The computer-aided system according to claim 3, wherein, At least two of the two or more candidate insertion paths have the same insertion site.

7. The computer-aided system according to claim 3, wherein, At least two of the two or more candidate insertion paths have different insertion sites.

8. The computer-aided system according to claim 4, wherein, The operator preferences include one or more of the following: user-identified restricted area of ​​the tissue or anatomical object, user-identified maximum path curvature, user preference for straight paths, user-identified insertion site preference, and user-identified insertion site restrictions.

9. The computer-aided system according to any one of claims 1 to 8, wherein, The control system is also configured to: Define the confined region of a 3D volume that will not be traversed by any path; and Determine the insertion path that does not intersect with the restricted area.

10. The computer-aided system according to claim 9, wherein, The control system is configured to: The restricted area is automatically determined from the one or more objects, the boundary, or the feature in the 3D representation.

11. The computer-aided system according to any one of claims 1 to 8, wherein, The control system is also configured to: Obtaining preoperative imaging data, including one or more of CT scan data, MRI scan data, cone-beam CT data, or ultrasound data; and The preoperative imaging data and the 3D representation are combined to provide a combined image of the target volume.

12. The computer-aided system according to claim 11, wherein, The control system is also configured to: The insertion path associated with the preoperative imaging data is adapted to the combined image of the target volume.

13. The computer-aided system according to any one of claims 1 to 8, further comprising: A pressure sensor configured to measure the pressure applied to the instrument, and wherein the control system is configured to: Receive pressure data indicating the pressure on the instrument; and The current instrument position or trajectory is determined from the pressure data.

14. The computer-aided system according to claim 13, wherein, The control system is also configured to: The instrument insertion trajectory is updated based on the current instrument position or trajectory and the pressure data.

15. The computer-aided system according to any one of claims 1 to 8, wherein, The control system is also configured to: Control at least one of the repositionable structure and the probe holder to hold the instrument within the field of view (FOV) of the sensing probe as the instrument moves along the insertion path; as well as Images are acquired from the sensing probe during instrument insertion to track the advancement of the instrument.

16. The computer-aided system according to claim 15, wherein, In order to control the repositionable structure, the control system is configured to: The command to repositionable structure moves according to the first set of movements, which further causes the probe holder and the instrument holder to move simultaneously according to the first set of movements.

17. The computer-aided system according to claim 15, wherein, The control system is also configured to: Reposition the repositionable structure and move the sensing probe to the new position; Identify the position of the target volume relative to the new position of the sensing probe; Reposition the sensing probe so that at least a portion of the target volume is within the FOV of the sensing probe; as well as One or more images of the target volume within the field of view (FOV) of the sensing probe are obtained via the sensing probe.

18. The computer-aided system according to claim 17, wherein, In order to identify the position of the target volume relative to the new position of the sensing probe, the control system is configured to: Receive user input to identify the location of the target volume.

19. The computer-aided system according to claim 17, wherein, In order to obtain one or more images of the target volume within the field of view (FOV) of the sensing probe, the control system is further configured to: Control the probe holder to sweep the FOV of the sensing probe across the target volume; and Multiple images are captured using the sensing probe while the sweep is being performed.

20. The computer-aided system according to any one of claims 1 to 8, wherein, The control system is also configured to: Control the instrument driver to pause the propulsion of the instrument; Perform real-time probe scan updates; and The 3D representation is updated based on the real-time probe sweep.

21. The computer-aided system according to claim 20, wherein, In response to the instrument advancing a predefined distance along the insertion path, the updated real-time probe sweep is performed periodically.

22. The computer-aided system according to claim 21, wherein, The predefined distance is 3 cm.

23. The computer-aided system according to claim 20, further comprising: A pressure sensor, configured to measure the pressure applied to the instrument, and The real-time probe sweep update is performed in response to unwanted pressure data from the pressure sensor.

24. The computer-aided system according to any one of claims 1 to 8, wherein, The control system is also configured to have the processor provide needle type recommendations based on the instrument insertion path.

25. The computer-aided system according to claim 24, wherein, The recommended needle types include one or more of the following: needle size, needle width, needle length, needle diameter, and needle tip.

26. The computer-aided system according to any one of claims 1 to 8, wherein, In order to advance the instrument along the insertion path, the control system is configured to: Control at least one of needle rotation and needle vibration.

27. The computer-aided system according to any one of claims 1 to 8, wherein, In order to segment the 3D representation of the target volume, the control system is further configured to: Identify the treatment site.

28. The computer-aided system according to any one of claims 1 to 8, wherein, The control system is also configured to: The user identifier of the receiving treatment site is derived from the 3D representation of the target volume.

29. The computer-aided system according to any one of claims 1 to 8, further comprising: A probe pressure sensor, operatively coupled to the sensing probe, is configured to measure the pressure applied to the probe. The control system is further configured to: The probe holder is controlled to maintain the pressure applied to the subject and the position relative to the subject.

30. The computer-aided system according to any one of claims 1 to 8, wherein, The control system is configured to: The probe holder and the instrument are controlled to move in coordination such that the distal end of the instrument is held within the field of view (FOV) of the sensing probe during the movement of the instrument.

31. The computer-aided system according to claim 30, wherein, In order to keep the distal end of the device within the field of view (FOV) of the sensing probe, the control system is configured to: The distal end of the device is positioned within the 2D imaging plane of the sensing probe.

32. The computer-aided system according to any one of claims 1 to 8, wherein, The control system is configured to: During image scanning of the target volume, the movement of the repositionable structure and the probe holder is controlled to maintain (i) the 2D imaging plane depth and (ii) the pressure of the sensing probe on the target.

33. The computer-aided system according to any one of claims 1 to 8, wherein, The control system is also configured to: The instrument is removed along the instrument insertion path.

34. The computer-aided system according to claim 33, wherein, The control system is also configured to: The probe holder is controlled to hold the distal end of the instrument within the field of view (FOV) of the sensing probe as the instrument is removed along the insertion path.

35. The computer-aided system according to claim 33, wherein, The control system is also configured to: The second instrument is advanced along the insertion path.

36. The computer-aided system according to any one of claims 1 to 8, wherein: The computer-aided system is used to perform a multi-stage process using a first instrument for the first stage and a second instrument for the second stage, and The control system is also configured to: Multiple insertion paths are identified to insert the device into the subject, thereby providing treatment to the treatment site for each stage of the multi-stage process.

37. The computer-aided system according to claim 36, wherein, Each stage of the multi-stage process utilizes the same insertion site.

38. The computer-aided system according to claim 36, wherein, At least two stages in the multi-stage process utilize different insertion sites.

39. The computer-aided system according to claim 36, wherein, The plurality of insertion paths includes at least one of the following: (i) different insertion sites and (ii) different insertion paths to the treatment site.

40. The computer-aided system according to claim 36, wherein, The first instrument and the second instrument are the same instrument.

41. The computer-aided system according to any one of claims 1 to 8, wherein: The repositionable structure includes multiple links and multiple joints. The instrument retainer is configured to receive the instrument and coupled to the distal link of the plurality of links, and The probe holder is configured to accommodate a sensing probe with a field of view and is coupled to the distal link and the probe holder.

42. The computer-aided system according to any one of claims 1 to 8, wherein, The repositionable structure has at least four degrees of freedom of motion.

43. A method for performing ultrasound-guided robotic percutaneous instrument insertion, the method comprising: An image of the target volume of the subject is obtained using a sensing probe; The control system reconstructs a three-dimensional (3D) representation of the target volume from an image of the target volume; The control system segments the 3D representation of the target volume to identify one or more objects, boundaries, or features in the 3D representation of the target volume; The control system determines a specific instrument insertion path, including the insertion site and the treatment site. as well as The control system controls the repositionable structure and the instrument holder to advance the instrument along the defined insertion path, wherein the position of the sensing probe is supported by a probe holder coupled to the repositionable structure.

44. The method according to claim 43, wherein, The sensing probe includes at least one of the following: an ultrasound probe, a photoacoustic probe, a microscope, an optical imaging probe, a video camera, an infrared camera, a handheld X-ray system, an electronic palpation device, a pulse radar detector, a focused ultrasound probe, a gamma probe, a perfusion imaging system, and an optical coherence tomography device.

45. The method according to claim 43, wherein, Determining the instrument insertion path includes: The control system determines two or more candidate insertion paths and selects one of the two or more candidate insertion paths to determine the insertion path.

46. ​​The method according to claim 45, wherein, Selecting the insertion path includes: The determined path is selected by the control system based on at least one of the following: path length, number of voxels traversed by the path in the 3D representation, avoidance of identified objects, curvature associated with the path, instrument type of the instrument, needle type of the needle into which the instrument is inserted, and one or more operator preferences.

47. The method according to claim 43, wherein: The 3D representation includes multiple voxels, which include (i) voxels with free space that do not contain tissue and (ii) voxels that contain tissue. Determining the instrument insertion path includes: The control system determines the insertion path of the instrument based on the number of voxels in free space and the number of voxels with tissue.

48. The method according to claim 45, wherein, At least two of the two or more candidate insertion paths have the same insertion site.

49. The method according to claim 45, wherein, At least two of the two or more insertion paths have different insertion sites.

50. The method of claim 46, wherein, The operator preferences include one or more of the following: user-identified restricted area of ​​the tissue or anatomical object, user-identified maximum path curvature, user preference for straight paths, user-identified insertion site preference, and user-identified insertion site restrictions.

51. The method according to any one of claims 43 to 50, further comprising: The control system determines a confined region representing a 3D volume that will not be traversed by the two or more paths. Determining the instrument insertion path further includes: The control system determines two or more candidate paths that do not intersect with the restricted area.

52. The method according to claim 51, further comprising: The restricted area is automatically determined by the control system from one or more objects, boundaries, or features in the 3D representation.

53. The method according to any one of claims 43 to 50, further comprising: The control system obtains preoperative imaging data, which includes one or more of CT scan data, MRI scan data, cone-beam CT data, or ultrasound data; and The control system combines the preoperative imaging data and the 3D representation to provide a combined image of the target volume.

54. The method of claim 53, further comprising: The control system adapts the proposed insertion path associated with the preoperative imaging data to the combined image of the target volume.

55. The method according to any one of claims 43 to 50, further comprising: Pressure data indicating the pressure on the instrument is received from a pressure sensor configured to measure the pressure applied to the instrument. as well as The control system determines the current instrument position or trajectory from the pressure data.

56. The method of claim 55, further comprising: The control system updates the determined insertion trajectory based on the current instrument position or trajectory and the pressure data.

57. The method according to any one of claims 43 to 50, further comprising: The control system controls at least one of the repositionable structure and the probe holder to keep the instrument within the field of view of the sensing probe as the instrument moves along the insertion path; as well as During instrument insertion, the control system acquires images from the sensing probe to track the advancement of the instrument.

58. The method according to claim 57, wherein, Controlling the repositionable structure includes: The repositionable structure is controlled by the control system to move according to the first set of movements, which further causes the probe holder and the instrument holder to move simultaneously according to the first set of movements.

59. The method according to any one of claims 43 to 50, further comprising: The control system controls the instrument driver to pause the propulsion of the instrument; The control system performs real-time probe scanning updates. as well as The 3D representation is updated by the control system based on the updated real-time probe scan.

60. The method according to claim 59, wherein, The updated real-time probe sweep is performed periodically in response to the instrument advancing a predefined distance along the insertion path.

61. The method according to claim 60, wherein, The predefined distance is 3 cm.

62. The method according to claim 59, wherein, The updated real-time probe sweep is performed in response to unwanted pressure data from the pressure sensor.

63. The method according to any one of claims 43 to 50, further comprising: The control system provides needle recommendations based on the instrument insertion path.

64. The method according to claim 63, wherein, The recommended needle types include one or more of the following: needle size, needle width, needle length, needle diameter, and needle tip.

65. The method according to any one of claims 43 to 50, wherein, Advancing the instrument along the insertion path includes: The control system controls at least one of needle rotation and needle vibration.

66. The method according to any one of claims 43 to 50, wherein, The 3D representation of the segmented target volume includes: The control system identifies the treatment site.

67. The method according to any one of claims 43 to 50, further comprising: The user identifier of the treatment site is obtained from the 3D representation of the target volume by the control system.

68. The method according to any one of claims 43 to 50, further comprising: The control system repositions the repositionable structure to move the sensing probe to a new position. The control system identifies the new position of the target volume relative to the sensing probe; The control system repositions the sensing probe so that at least a portion of the target volume is within the field of view (FOV) of the sensing probe. as well as One or more images of the target volume within the field of view (FOV) of the sensor are obtained via the sensor probe.

69. The method according to claim 68, wherein, Identifying the new position of the target volume relative to the sensing probe includes: The control system receives user input to identify the location of the target volume.

70. The method according to claim 68, wherein, Obtaining one or more images of the target volume within the FOV of the sensing probe includes: The control system controls the probe holder to sweep the field of view (FOV) of the sensing probe across the target volume; and Multiple images are captured using the sensing probe while the sweep is being performed.

71. The method according to any one of claims 43 to 50, further comprising: The pressure applied to the sensing probe is received from a probe pressure sensor that is operatively coupled to the sensing probe. as well as The probe holder is controlled by the control system to maintain the pressure applied to the subject and its position relative to the subject.

72. The method according to any one of claims 43 to 50, further comprising: The movement of the probe holder and the instrument is coordinated and controlled by the control system, such that the distal end of the instrument is held within the field of view (FOV) of the sensing probe during the movement of the instrument.

73. The method according to claim 72, wherein, Holding the distal end of the device within the field of view (FOV) of the sensing probe includes: The control system maintains the position of the distal end of the device within the 2D imaging plane of the sensing probe.

74. The method of claim 71, further comprising: During image scanning of the target volume, the movement of the repositionable structure and the probe holder is controlled by the control system to maintain (i) the 2D imaging plane depth and (ii) the pressure exerted by the sensing probe on the target.

75. The method according to any one of claims 43 to 50, further comprising: The control system removes the instrument along the instrument insertion path.

76. The method of claim 75, further comprising: The control system controls the probe holder to hold the distal end of the instrument within the field of view (FOV) of the sensing probe when the instrument is removed along the insertion path.

77. The method of claim 75, further comprising: The second instrument is advanced along the insertion path by the control system.

78. The method according to any one of claims 43 to 50, wherein, The control system is configured to perform a multi-stage process using a first instrument for a first stage and a second instrument for a second stage, wherein the method further includes: The control system identifies multiple candidate insertion paths to insert the device into the subject, thereby providing treatment to the treatment site for each stage of the multi-stage process.

79. The method according to claim 78, wherein, Each stage in the multi-stage process has the same insertion site.

80. The method according to claim 78, wherein, At least two stages in the multi-stage process have different insertion sites.

81. The method according to claim 78, wherein, The plurality of candidate insertion paths includes at least one of the following: (i) different insertion sites and (ii) different insertion paths to the treatment site.

82. The method according to claim 78, wherein, The first instrument and the second instrument are the same instrument.

83. The method according to any one of claims 43 to 50, wherein: The repositionable structure includes multiple links and multiple joints. The instrument retainer is configured to receive the instrument and coupled to the distal link of the plurality of links, and The probe holder is configured to accommodate a sensing probe with a field of view and is coupled to the distal link and the probe holder.

84. The method according to any one of claims 43 to 50, wherein, The repositionable structure has at least four degrees of freedom of motion.

85. A computer-readable medium storing instructions that, when executed by a processor, cause a system to perform the method according to any one of claims 43 to 84.

Images