Ultrasound-guided positioning and insertion of a percutaneous cannulation device
The hand-held cannulation device addresses the limitations of fixed-frame automated devices by using real-time imaging and sensor data to adjust cannula trajectory, improving accuracy and convenience in venipuncture.
Patent Information
- Application Number
- JP2021520123
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2018-10-09
- Filing Date
- 2019-10-09
- Publication Date
- 2024-11-21
- Estimated Expiration
- 2039-10-09
Smart Images

Figure 0007588069000008 
Figure 0007588069000009 
Figure 0007588069000010
Abstract
Description
[Technical field]
[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of the filing date of U.S. Provisional Application No. 62 / 743,283, entitled “Ultrasound Guided Alignment And Insertion Of Percutaneous Cannulating Instrument,” filed October 9, 2018, the disclosure of which is incorporated herein by reference.
[0002] (STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT) This invention was made with Government support under Contract No. R01EB020036 awarded by the National Institutes of Health, Contract No. EB018191 awarded by the National Institutes of Health, Contract No. T32 GM008339 awarded by the National Institutes of Health, Contract No. DGE-0937373 awarded by the National Science Foundation, and Contract No. P200A150131 awarded by the U.S. Department of Education. Accordingly, the U.S. Government may have certain rights in this invention. [Background technology]
[0003] The process of venipuncture, inserting a cannula into a vein, for example to draw a blood sample or deliver intravenous (IV) fluids, is one of the most common clinical routines performed in medicine. Often, finding a suitable venipuncture site can be difficult, especially in patients with darker skin or a high body mass index. Even after the target vein is identified, the process of inserting the cannula into the vessel can also be difficult. Failed insertion attempts are common. Sometimes, the failure is caused by unexpected movement of the vein, such as pressure applied to the vein by the needle tip causing the vein to deform, shift, or rotate. Other times, the failure is caused by unexpected movement of the patient or the clinician holding the cannulation tool. An unsuccessful needle introduction can cause complications, such as increased pain, internal bleeding, and leakage of IV fluids into extravascular tissues.
[0004] Various imaging techniques based on near-infrared (NIR) light or ultrasound (US) have been introduced to assist clinicians in visualizing veins. However, studies have shown that these imaging techniques alone do not significantly improve first-stick success rates compared to manual procedures. To improve venipuncture insertion accuracy, devices have been developed that combine NIR and US imaging techniques with automated robotic systems that use image data to guide the cannula to the center of the vessel.
[0005] 1 is an outline side view of an exemplary prior art automated cannulation system 100. The automated cannulation system 100 includes a positioning unit 110 attached to an ultrasound imaging unit 120. The positioning unit 110 may include a robotic arm 112 configured to control the positioning of a cannula 115, such as a needle, and may have multiple degrees of freedom for manipulating the cannula 115. The ultrasound imaging unit 120 may further include a probe 122 attached to an end of the ultrasound imaging unit 120, whereby the ultrasound imaging unit 120 is configured to acquire US images of a portion of a patient (not shown) positioned opposite the probe 122. The probe 122 may be made of a flexible material designed to bend and conform to the patient's imaging target area (e.g., the patient's arm). In some embodiments, the probe 122 may further include a gel compartment 125 configured to contain ultrasound gel for enhancing acquisition of US images.
[0006] 2, a conventional automated cannulation system 100 is used to insert a needle into a patient's arm 201. The patient's arm 201 may be positioned on a table 210 to ensure patient comfort and stability of the arm 201. A cannula 115 can be positioned diagonally above the patient's arm 201 such that the tip of the cannula 115 is aligned with a cannulation location 225 directly below the imaging probe 122.
[0007] 3 is an exterior side view of an exemplary conventional positioning unit 110 having four degrees of freedom. The positioning unit 110 includes a robotic arm 112 having a base 302, a first extension member 304 extending from the base 302, a second extension member 306 connected to and extending from the first extension member 304, and a cannulation tool receptacle 308 for receiving a cannula 115. The positioning unit 110 of FIG. 3 has four degrees of freedom: (1) a degree of freedom to rotate the base 302 about a focal point 312 in the xy plane; (2) a degree of freedom to rotate the first extension member 304 about a focal point 314 in the xz plane (more precisely, the plane formed by the z axis and a specific axis of the first extension member); (3) a degree of freedom to rotate the second extension member 306 about a focal point 316 in the xz plane (more precisely, the plane formed by the z axis and a specific axis of the second extension member); and (4) a degree of freedom to translate the cannula 115 forward or backward along its axis 318.
[0008] Imaging technology in the exemplary automated device and other known devices can track patient movement, including movement of the patient's target location (e.g., a vein beneath the patient's skin). These devices can also adjust their trajectory for precise insertion at the target location. However, for the device to perform such precise tracking and repositioning, it must be secured to a benchtop or other stationary platform, as shown in FIG. 2. The benchtop provides the necessary stability to ensure that the cannulation instrument does not move unintentionally, but only moves when guided by the positioning unit.
[0009] This need to secure a venipuncture device in place prior to use can be inconvenient. First, not all clinical environments are equipped with a benchtop or other surface on which the device can be placed. Moreover, it can take a clinician a significant amount of time to place and remove the device. Ultimately, while automated venipuncture devices offer several advantages over traditional handheld venipuncture methods, such as from the standpoint of accuracy, some clinicians find automated devices inconvenient, and some clinical settings may not even be suitable for use with automated devices. Summary of the Invention
[0010] The present disclosure relates to a handheld automated cannulation device that provides the convenience and portability associated with being handheld while at the same time providing the improved accuracy associated with other automated devices. In order for an automated device to be used handheld, it was necessary to design a control scheme that could respond quickly and efficiently to real-time sensor data while not having to rely on a fixed reference frame external to the device itself.
[0011] One aspect of the disclosure relates to a device for directing a cannula towards a subcutaneous target location in a patient. The device may include an imaging probe that continuously generates imaging data of the target location, a positioning unit, and a processor. The positioning unit may include a plurality of motors configured to manipulate a cannula attached to the positioning unit. The positioning unit may further include at least one cannula position sensor configured to generate cannula attitude data indicative of a position and orientation of the cannula based on movement of the motors. The positioning unit may further include at least one device position sensor configured to generate device attitude data indicative of an orientation and position of the device.
[0012] The processor may be configured to receive imaging data from the imaging probe, receive cannula orientation data from the at least one cannula position sensor, receive device orientation data from the at least one device position sensor, identify a target location from the imaging data, and determine a trajectory for steering the cannula towards the target orientation based on the imaging data, the cannula orientation data, and the device orientation data. The processor may be further configured to instruct the positioning unit to steer the cannula according to the determined trajectory. Furthermore, the processor may be configured to identify that the determined trajectory has deviated from the target location, and to update the determined trajectory to a modified trajectory based on the imaging data, the cannula orientation data, and the device orientation data, and to instruct the positioning unit to steer the cannula according to the modified trajectory.
[0013] In some embodiments, the imaging probe may be an ultrasound probe. The imaging data generated by the imaging probe may be any one of two-dimensional imaging data, three-dimensional volumetric imaging data, and bi-plane imaging data. In some embodiments, the processor may be further configured to identify and determine a pose of a target vessel underlying the patient's skin based on the imaging data. In some embodiments, the device may be configured to acquire imaging data of the target location from a plurality of angles. The processor may be further configured to construct three-dimensional imaging data from the received imaging data acquired from the plurality of angles, determine a three-dimensional pose of a target location underlying the patient's skin from the imaging data acquired from the plurality of angles, or both.
[0014] In some examples, the imaging data may include video data of the target location acquired at different times, and the processor may be configured to determine a trajectory for steering the cannula towards the target pose based on a comparison between previous and subsequent frames of the video data.
[0015] In some embodiments, the positioning unit may be configured to manipulate the cannula in at least two degrees of freedom.
[0016] In some examples, the processor may be configured to identify when the determined trajectory has deviated from the target position based on device attitude data received from at least one device position sensor, such as a camera, an electromagnetic sensor, a gyroscope, an accelerometer, or any combination thereof.
[0017] In some embodiments, the device may comprise a force sensor configured to generate force data indicative of a measure of the force exerted on the cannula. The processor may be configured to receive the force data from the force sensor, update the cannula attitude data to a corrected position and orientation based on the received force data, and instruct the cannula positioning unit to manipulate the cannula according to the corrected position and orientation. Additionally or alternatively, the processor may be configured to receive the force data from the force sensor, obtain a force profile from the force data, and classify the obtained force. Examples of force classifications include a successful cannulation event, a vessel rotation event, an undershoot event, and an overshoot event. The processor may be able to classify the force data into one or more of these classifications. Furthermore, in some embodiments, in the absence of a successful cannulation event, the processor may be configured to instruct the positioning unit to manipulate the cannula according to the classified event. In some examples, the device may further comprise a blood flush sensor configured to generate blood flush data indicative of the flow of blood into the cannula. The processor may be configured to receive the blood flush data from the blood flush sensor and to instruct the cannula positioning unit to manipulate the cannula further based on the blood flush data.
[0018] In some examples, each motor may be associated with a corresponding motor sensor that may be configured to generate cannula posture data indicative of a position and orientation of the cannula based on movement of the corresponding motor.
[0019] Another aspect of the disclosure relates to a method for guiding a cannula towards a subcutaneous target location in a patient, the method being executable by a processor coupled to an imaging probe for continuously generating imaging data of the target location, and a positioning unit comprising a plurality of motors configured to manipulate a cannula attached to the positioning unit, each motor associated with at least one cannula position sensor configured to generate cannula attitude data indicative of a position and orientation of the cannula based on movement of the corresponding motor. The positioning unit may further comprise at least one device position sensor configured to generate device attitude data indicative of an orientation and position of the device. The method may include receiving imaging data from an imaging probe, receiving cannula orientation data from at least one cannula position sensor, receiving device orientation data from at least one device position sensor, identifying a target position from the imaging data, determining a trajectory for steering the cannula towards the target orientation based on the imaging data, the cannula orientation data and the device orientation data, instructing a positioning unit to steer the cannula according to the determined trajectory, identifying that the determined trajectory has deviated from the target position, updating the determined trajectory to a modified trajectory based on the imaging data, the cannula orientation data and the device orientation data, and instructing the positioning unit to steer the cannula according to the modified trajectory.
[0020] In some embodiments, the method may further include identifying and determining a position of a target vessel underlying the patient's skin based on the imaging data, and instructing the positioning unit to steer the cannula based on the determined position of the target vessel.
[0021] In some embodiments, receiving imaging data from the probe may include receiving imaging data from a plurality of angles, and the method may further include constructing three-dimensional imaging data from the received imaging data acquired from the plurality of angles. In some embodiments, the method may include determining a pose of a subcutaneous target location on the patient from the imaging data acquired from the plurality of angles.
[0022] In some embodiments, instructing the positioning unit to manipulate the cannula may include providing instructions for manipulating each of two or more degrees of freedom of the positioning unit. In some embodiments, identifying that the determined trajectory has misaligned with the target position may be based on device attitude data received from a device position sensor.
[0023] In some embodiments, determining the cannula posture data may be based on imaging data using at least one of a computer vision algorithm or an electromagnetic sensor.
[0024] In some examples, the processor may be coupled to a force sensor that generates force data indicative of a measure of the force exerted on the cannula. The method may further include receiving the force data from the force sensor, updating the cannula positioning data to a corrected position and orientation based on the received force data, and instructing the cannula positioning unit to manipulate the cannula according to the corrected position and orientation. Additionally or alternatively, the method may further include obtaining a force profile from the force data and classifying the obtained force profile (e.g., successful cannulation event, vessel rotation event, undershoot event, overshoot event). Furthermore, in some cases, in the absence of a successful cannulation event, the method may include instructing the positioning unit to manipulate the cannula according to the classified event.
[0025] Yet another aspect of the present disclosure relates to a device for inserting a needle into a blood vessel of a patient. The device may include an ultrasound probe configured to image a portion of the blood vessel located within a field of view of the ultrasound probe, a mechanical arm attached to the ultrasound probe, and a processor for controlling the operation of one or more motors included in the mechanical arm. The needle may be attachable to the mechanical arm. The mechanical arm may include one or more motors configured to manipulate the position and orientation of the needle. The processor may be configured to define a frame of reference for controlling the one or more motors based on spatial parameters of the blood vessel. The mechanical arm may be configured to insert the needle at an angle into the portion of the blood vessel located within a field of view of the ultrasound probe. The processor for controlling the operation of the one or more motors may be automated based at least in part on image data from the ultrasound probe, and the device may be configured to be handheld.
[0026] In some embodiments, the motor of the mechanical arm may be configured to manipulate at least a depth of needle insertion and an angle of needle insertion. A processor may be communicatively coupled to the probe and configured to determine the depth of insertion based at least in part on data received from the probe. The angle of insertion may be determined based on the depth of insertion.
[0027] In some embodiments, the device may further comprise an accelerometer and a gyroscope configured to provide data indicative of the device's motion and orientation, respectively, to the processor. The processor may control operation of one or more motors, which may be automated based in part on data from the accelerometer and gyroscope.
[0028] Yet another aspect of the disclosure relates to a device for directing a cannula toward a subcutaneous target location on a patient, the device may include at least one motor configured to steer the cannula along its length toward the target, a force sensor configured to generate force data indicative of a measure of force exerted on the cannula, a blood flush sensor configured to generate blood flush data indicative of blood flow into the cannula, and a processor configured to receive the force data from the force sensor, receive the blood flush data from the blood flush sensor, determine the occurrence of a successful cannulation event based on a combination of the force data and the blood flush data, and direct the at least one motor to cease steering the cannula toward the target upon the occurrence of a successful cannulation event.
[0029] In some embodiments, the device may further comprise a cannula position sensor configured to generate cannula posture data indicative of a position of the cannula based on movement of the at least one motor. The processor may be configured to determine a successful cannulation event based on a combination of the force data, the blood flush data, and the cannula posture data.
[0030] In some examples, the device may further comprise an imaging probe to acquire ultrasound imaging data. The processor may be configured to determine a likelihood of cannulation failure based on a combination of the force data, the blood flush data, the ultrasound imaging data, and the cannula orientation data, and to determine adjustments to the cannula that reduce the likelihood of a cannulation failure event based on the combination of the force data, the blood flush data, the ultrasound imaging data, and the cannula orientation data.
[0031] In some examples, the processor may be configured to obtain a force profile from the received force data, classify the obtained force profile as either a successful cannulation event or a failed cannulation event, and instruct one or more motors to operate the cannula in a reverse direction upon the occurrence of a failed cannulation event. [Brief description of the drawings]
[0032] [Figure 1] 1 is a conventional automatic cannulation device. [Diagram 2] FIG. 2 is another view of a conventional cannulation device in operation. [Diagram 3] 2 is an exemplary positioning unit of the conventional cannulation device of FIG. 1; [Figure 4] FIG. 1 is a block diagram of a cannulation device according to one aspect of the present disclosure. [Diagram 5] FIG. 1 is a diagram of a vessel and coordinate reference frame according to one aspect of the present disclosure. [Figure 6A] 1 is a graphical representation of force measurements over time according to one aspect of the present disclosure. [Figure 6B] 1 is a graphical representation of force measurements over time according to one aspect of the present disclosure. [Figure 6C] 1 is a graphical representation of force measurements over time according to one aspect of the present disclosure. [Figure 6D] 1 is a graphical representation of force measurements over time according to one aspect of the present disclosure. [Figure 7] 1 is a flow chart of a protocol according to one aspect of the present disclosure. [Figure 8] FIG. 1 is an exploded perspective view of an electromagnetic attachment mechanism according to one aspect of the present disclosure. [Figure 9A] 13 is a flow chart of an additional protocol according to an aspect of the present disclosure. [Figure 9B] 13 is a flow chart of an additional protocol according to an aspect of the present disclosure. [Figure 9C] 13 is a flow chart of an additional protocol according to an aspect of the present disclosure. [Figure 10A] FIG. 1 is a perspective view of a cannulation device according to one aspect of the present disclosure. [Figure 10B] FIG. 10B is an exploded perspective view of the cannulation device of FIG. 10A. [Figure 10C] FIG. 10B is a perspective view of a portion of the cannulation device of FIG. 10A. [Figure 10D] FIG. 10B is a perspective view of a portion of the cannulation device of FIG. 10A. [Figure 10E] FIG. 10B is a perspective view of a portion of the cannulation device of FIG. 10A. [Figure 10F] FIG. 10B is a perspective view of a portion of the cannulation device of FIG. 10A. [Figure 10G] FIG. 10B is a perspective view of a portion of the cannulation device of FIG. 10A. [Figure 10H] FIG. 10B is a perspective view of a portion of the cannulation device of FIG. 10A. [Figure 10I] FIG. 10B is a perspective view of a portion of the cannulation device of FIG. 10A. [Figure 11A] FIG. 13 is a perspective view of another cannulation device according to an aspect of the present disclosure. [Figure 11B] FIG. 11B is a perspective view of a portion of the cannulation device of FIG. [Figure 11C] FIG. 11B is a perspective view of a portion of the cannulation device of FIG. [Figure 11D] FIG. 11B is a perspective view of a portion of the cannulation device of FIG. [Figure 11E] FIG. 11B is a perspective view of a portion of the cannulation device of FIG. [Figure 11F] FIG. 11B is a perspective view of a portion of the cannulation device of FIG. [Figure 11G] FIG. 11B is a perspective view of a portion of the cannulation device of FIG. [Figure 12] FIG. 13 is a block diagram of a workflow for operating a cannulation device according to one aspect of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0033] 4 is a block diagram illustrating a control scheme for a handheld cannulation device 400 according to the present disclosure. The cannulation device 400 comprises a main processor 410, a microcontroller 420, a display 430, and multiple motor controllers 440a, 440b each controlling various axes of freedom of a cannulation tool 450. The main processor 410 may be any processing unit known in the art having sufficient processing power to receive, process, and output image data at video frame rates. The microcontroller 420 may be an integrated chip or other processing unit known in the art having sufficient processing power to control a robotic arm, including processing and relaying sensor input signals and controlling motor operation.
[0034] The microcontroller 420 may be configured to receive any one or a combination of analog and digital input signals from various sensors, process the input signals, and relay the input signals to the main processor via a controller area network (CAN) connection. In some cases, the main processor 410 may include a different connection, such as a universal serial bus (USB) connection, in which case the sensor data may be relayed via a CAN-to-USB connection. Conversely, instructions relayed from the main processor 410 through the microcontroller 420 may be provided via a USB-to-CAN connection.
[0035] The display 430 may be configured to provide a two-dimensional or three-dimensional image of the cannulation target location and the cannulation tool. A cannulation trajectory, such as a predicted trajectory of the cannulation tool to reach the target vessel, may be superimposed on the image. The image may be updated or refreshed in real time or at the video frame rate. Additionally, the display may be configured to allow the clinician to identify locations within the images presented thereon, e.g., to identify a target location.
[0036] The display 430 may further be configured to initialize image processing routines. For example, automatic segmentation of a vessel boundary may be initialized by interaction between a user and the device 400 via the display 430. For example, the ultrasound transducer may be moved until the target vessel is centered within the ultrasound image (the display may include crosshairs indicating the image center). Once centered, processing software within the device may be manually initiated to perform a search for segmentation starting from the center of the image. After the initial segmentation, the target may be automatically tracked.
[0037] A similar method of initializing the automatic segmentation algorithm can be used when the probe is oriented longitudinally. In this situation, the vessels may be aligned so that they are parallel to the longitudinal image. The segmentation algorithm can then identify the vessels from the longitudinal image and automatically track the identified vessels.
[0038] The center of the blood vessel can be detected manually or automatically based on the ultrasound image. In the case of automatic detection, the image segmentation algorithm may be applied for both longitudinal and transverse orientations of the ultrasound probe. In the transverse orientation, where the grayscale B-mode ultrasound image shows a cross-section of the blood vessel, the algorithm may detect elliptical structures in the image and may initialize rough segmentation contours for potential blood vessel structures based on the detected elliptical structures. These initial contours may be refined iteratively. In the longitudinal orientation, where the B-mode ultrasound image is parallel to the blood vessel, the algorithm may instead detect elongated tubular structures with expected diameters in the range of 1-5 mm and may perform a similar process of contour initialization and refinement for the detected tubular structures.
[0039] Additionally, the B-mode image segmentation algorithm may be complemented by image information extracted from color Doppler imaging mode, where velocity (rather than tissue backscatter) is detected by the probe. Blood flow can be directly visualized by color Doppler imaging, since blood exhibits directional motion and velocity while the surrounding tissue is mostly stationary. Color Doppler image data may be combined with the B-mode intensity image data in several different ways (masking, contour initialization, weighted sum, etc.) to improve the accuracy of the segmentation algorithm.
[0040] Once the vessel center coordinates are identified, an image-based tracking algorithm can be used to track the vessel on a frame-by-frame basis, allowing the device to detect tissue motion in real time and adjust the instrument pose accordingly. Tracking the vessel in this way also reduces the need to perform vessel segmentation from scratch in each frame, thus reducing computation time.
[0041] The cannulation device 400 further comprises a number of sensors which provide input signals to the main processor 410 and the microcontroller 420. The sensors may include an imaging probe 462, such as an ultrasound probe, a force sensor 464 to sense the force applied to the motors when driving the cannulation tool, position encoders 466a and 466b for motor controllers 440a and 440b to monitor needle manipulation by the device's motors, and a position sensor 468 to track changes in translation and orientation of the cannulation device 400. Because the cannulation device is handheld, the position sensor 468 is needed to track if the cannulation device is deliberately or accidentally moved from its initial position.
[0042] The main processor 410 and the microcontroller 420 are programmed with software instructions to carry out the operations and protocols outlined in this disclosure. For example, the microcontroller may be programmed to receive analog input signals from each of the force sensors, position encoders, and position sensors, process the received input signals into a format acceptable to the main processor, and then transmit the reformatted input signal data to the main processor for further processing. As a further example, the main processor may be programmed to receive and process imaging data, such as to identify a target location on the patient and determine a trajectory to guide the cannulation tool to the target location. The main processor may further determine and continuously track a relative position of the cannulation tool to the target location based on the received input signals, and may update the determined trajectory based on changes in the relative position.
[0043] The processor may calculate the relative positions during cannulation and display these to the operator. For example, the processor may automatically estimate the final position of the cannulation instrument tip (e.g., where the tip will be during insertion) based on the current pose of the instrument as it is being manipulated / positioned. It may also calculate an offset distance between the estimated tip position and the target position. Similarly, the expected trajectory of the instrument (e.g., the path the instrument tip will follow during insertion) may be continuously estimated from the current pose of the instrument and displayed. The estimated position in the ultrasound image may be continuously updated and the updated position displayed. In some cases, the ultrasound image and the estimated tip position may be further updated while being continuously calculated while the insertion is being performed. The processor may also update the coordinates of the vascular target, for example to compensate for movements due to the tissue, the patient and the ultrasound probe. The updated position of the vascular target is indicated to the instrument positioning unit to adjust the pose and trajectory of the instrument tip to minimize the offset distance between the instrument tip and the vascular target and ensure constant alignment with the vascular target. In yet another embodiment, a signal may be given to the user when the estimated tip position is accurately aligned with a vessel or other target position in 3D space. This signal indicates that cannulation can be performed. Depending on the clinical situation, cannulation can be assumed to be performed only when the estimated tip and target positions are aligned within a preset error tolerance. Such a mechanism serves as a safety feature to prevent erroneous, inaccurate or unsafe cannulation attempts.
[0044] The imaging probe 462 may be an ultrasound (US) imaging unit capable of detecting target vessels or tissues below the surface of the patient's skin. The US imaging unit may be capable of collecting US images in real time or at video frame rates (e.g., 30 or 60 frames per second or more). In some imaging units, the images may be two-dimensional. In other units, the images may be two-directional. In yet other units, the images may be three-dimensional. Exemplary applications of these imaging types are described in more detail below. Preferably, the frequency of the ultrasound probe is in the range of about 6 MHz to about 18 MHz. However, other frequencies may be used if it is determined that a higher image quality is obtained. For example, deeper tissue structures may require lower ultrasound frequencies, while structures closer to the surface may warrant higher frequencies.
[0045] In some embodiments, the US imaging unit may further include any one or any combination of color Doppler, power Doppler, spectral Doppler, tissue Doppler, vector Doppler, and other ultrasound-based imaging modes. These imaging techniques may be used to detect the presence and direction of blood flow in blood vessels so that suitable blood vessels can be selected for puncture. Additionally or alternatively, the imaging probe 462 may be paired with other secondary imaging units, such as visible light imaging (e.g., camera), infrared (IR) or near-infrared (NIR) imagers, multispectral imagers, structured light or depth imaging, optical tomography techniques, laser Doppler, etc. Also, other imaging techniques, such as photoacoustic or optoacoustic imaging, may be employed.
[0046] The force sensor 464 may comprise a material having properties that change in response to an applied force, such as a piezoelectric material or a force sensitive resistor, which may be embedded in the cannulation device in contact with the cannulation tool. In one example of operation, as the cannulation tool moves along a linear degree of freedom to advance the cannula, for example by sliding along a ball bearing slider, both normal and frictional forces act on the cannulation tool, causing it to press against the material of the force sensor 464. The force sensor may generate an analog signal, which is then digitized by an analog-to-digital converter, such as the microcontroller 420, and relayed to the main processor 410. The processor 410 may use the received signal data to monitor the amount of force applied to the material.
[0047] Additionally or alternatively, the force sensor may be a current sensor configured to measure the current flowing through the windings of the motor of the cannulation device. The current may increase proportionally to the load on the motor, meaning that a measurement of the current may indicate an increase in the force exerted on the cannulation tool, such as, for example, in puncturing the target vessel wall. Again, the current measurement may be an analog measurement, which is converted to a digital signal and relayed to the processor 410 for further processing. For example, a series of current measurements over time may be recorded by the processor and then filtered (e.g., by a first order low pass digital filter) to obtain an indication of the force on the cannulation tool. Taking the needle insertion force as an example, the force measured may be the sum of several forces, such as, but not limited to, kinetic and viscous friction, elastic deformation of soft tissue, and cutting of tissue.
[0048] The position encoders 466a, 466b may be of various types depending on the type of degree of freedom controlled by the motor corresponding to each encoder. For example, if a motor controls a rotational degree of freedom of the positioning unit, the corresponding encoder may be a rotary encoder or a potentiometer. In a further embodiment, if a motor controls a linear degree of freedom of the positioning unit, the corresponding encoder may be a linear encoder reading a linear translation. Either relative or absolute encoders may be used.
[0049] The position sensor 468 may include a gyroscope and / or an accelerometer to track the movement and tilt of the cannulation device. In one embodiment, the gyroscope may be responsible for tracking the orientation of the device, while the accelerometer is responsible for tracking the overall movement of the device. Other combinations of sensors, such as multiple gyroscopes or multiple accelerometers, may be utilized to achieve similar results. In some cases, an inertial measurement unit may be used to track the movement. In other cases, a stereo camera may be used to improve tracking. Additionally or alternatively, an electromagnetic tracking device may be used to monitor the movement of the device. Additionally or alternatively, image data (e.g., camera image frames, ultrasound image frames) may be used to track as the device translates or tilts. In such cases, the images may be processed using a computer vision program so that changes in the image frames may indicate the displacement or tilt of the device.
[0050] The position sensor 468 can be utilized for several purposes. An exemplary application of the motion sensor is to form a 3D image from 2D probe imaging data. If the cannulation device includes a 2D imaging probe, such as a two-dimensional ultrasound imaging unit, the device may sweep around the target location on the patient. In this manner, images of the target location can be acquired from multiple angles. The images are provided to a main processor and can then be effectively stitched together to obtain a composite 3D image. The use of 3D images to track the target vessel is advantageous compared to 2D images for reasons further described below.
[0051] Another exemplary use of the position sensor 468 is to provide a gravity vector that may be overlaid with the probe imaging data to aid in the interpretation of the imaging results. In some cases, this can be useful in identifying the patient's anatomy.
[0052] Locating the position sensor 468 in a particular area of the cannulation device 400 does not limit the sensor's ability to track component movement in other areas of the device. For example, if the position sensor is a gyroscope and accelerometer attached to the imaging probe, the position sensor can be used to directly measure the attitude and acceleration of the probe. The directly monitored data can then be processed by the main processor to indirectly calculate changes in attitude and displacement of the cannulation tool based on the known kinematic attitude of the cannulation tool relative to the probe, as well as changes in the rotational attitude and displacement of the probe. Similarly, a gyroscope and accelerometer attached to the cannulation tool may be used to indirectly calculate the attitude and displacement of the imaging probe.
[0053] In other embodiments, the probe and cannulation tool may each be equipped with its own position sensor, and measurements from the position sensors may be compared to one another to confirm agreement, to provide further accuracy to the measurements, or both.
[0054] It should be understood that because the cannulation device 400 is handheld and includes an imaging probe, the image data received by the main processor is not taken from a fixed position or angle. Therefore, the objects and locations identified in the image data cannot be characterized using an absolute world coordinate system. Instead, a dynamic reference coordinate system must be used to relate the images to the relative position of the cannulation tool. Because the cannulation tool and the imaging probe are connected to each other, moving the cannulation device causes the cannulation tool and the imaging probe to move in predictable coordination with each other.
[0055] One challenge that arises from using a handheld cannulation device is the need to define the orientation of the cannulation tool relative to the device and the imaging probe. Because the cannulation device is handheld and subject to being moved or tilted, a fixed coordinate system cannot be used. Instead, the orientation of the cannulation tool may be based on one or more sensors that can track the movement of the cannulation tool relative to the device.
[0056] In some cases, the pose data of the cannulation tool may be obtained from processing data received from an electromagnetic tracker to track the movement of the cannulation tool. Additionally or alternatively, image data (e.g., camera image frames, ultrasound image frames) may be used to track the pose of the cannulation tool as it advances towards the target location. In such cases, the images may be processed using a computer vision program. Additionally or alternatively, the pose data of the cannulation tool may be obtained from position encoders for each motor. Each joint of the robot arm's kinematic chain (such as the degree of freedom chain shown in FIG. 3) can be defined by a separate pose transformation starting from the robot origin and ending at the tip of the tool.
[0057] The pose of the cannulation tool is calculated from the encoder measurements of each degree of freedom or joint and can be represented by a rotation and translation offset relative to the past degree of freedom or joint. This representation can be in the form of, for example, a vector describing the rotation and a translation vector describing the offset. Alternatively, an Euler vector representation may be used to describe the rotation. Alternatively, the rotation and translation representations can be combined into a single homogeneous transformation matrix. Whatever the representation, the relative pose of each degree of freedom or joint can be combined in turn to obtain the current pose between the origin and the tip of the cannulation tool.
[0058] For example, in the case of a four degree of freedom system, a four-dimensional vector or a 4×4 matrix containing the position encoder measurements for each degree of freedom may be employed to represent the current pose of the cannulation tool. As the motors of the robotic arm manipulate the cannulation tool, updated encoder information may be provided to one or both of the main processor and the microcontroller, and the transformation chain may be updated based on the updated encoder information.
[0059] As explained above, the relative coordinate systems of the imaging unit and the robot arm remain fixed relative to each other even when the motors move the cannulation tool. In this regard, the N joints (joint 1 to joint N ) can be characterized as follows: JPEG0007588069000001.jpg496
[0060] In this way, a reference frame may be defined in terms of the attitude and orientation of the device, or in particular the main part of the device. This reference frame (device) may then be used to relate other parts of the system to each other. For example, an imaging probe and a cannulation tool may be related to each other in the following way: JPEG0007588069000002.jpg576
[0061] Encoder-based pose measurements may be subject to some residual errors. Residual errors are typically caused by mechanical inaccuracies, motor backlash, calibration errors, etc. Therefore, additional sensors may be integrated into the cannulation device to reduce the residual errors from these measurements. For example, a position sensor (such as that of FIG. 4) may be used to correct the residual errors. In an exemplary arrangement, the position sensor may comprise one or both of a gyroscope mounted on either the robot arm or the imaging probe to estimate the absolute 3D pose of the cannulation device, and an absolute position / orientation sensor mounted near the distal joint of the cannulation tool to estimate the absolute 3D pose of the cannulation tool. The outputs from these two position sensors may be combined (e.g., in the main processor) to estimate a relative tool pose that is not affected by mechanical errors. The transformation used to arrive at the 3D pose of the cannulation tool relative to the surrounding environment (e.g., the patient, the patient's skin, and the target position, collectively referred to as the "world") may be obtained based on the gyroscope measurements and may be characterized as follows: JPEG0007588069000003.jpg781
[0062] Another challenge posed by the use of a handheld device is the need to characterize the 3D pose of the target vessel relative to the coordinate system of the cannulation device, as opposed to characterizing the vessel's pose based on a fixed coordinate system. That way, if the cannulation device moves, its movement can be easily tracked relative to the target vessel. Furthermore, because the cannulation tool is tracked relative to the cannulation device, the manipulation of the cannulation tool can also be easily tracked relative to the target vessel.
[0063] This may be accomplished by relying entirely or primarily on image data from an imaging probe. There are several ways to perform this characterization, depending on the particular image data available. For clarity, reference is made to FIG. 5, which illustrates a particular 3D coordinate axis useful for characterizing vascular pose.
[0064] In a first embodiment, a single two-dimensional (2D) ultrasound image can be acquired in a transverse orientation, from which the center of the vessel can be seen through a single cross-sectional plane of the vessel, including its depth from the surface of the patient's skin. These are shown in FIG. v and Z v To confirm that the target location is suitable for puncture, the vessel diameter (d v ) can be measured. When using color Doppler, blood flow velocity can be quantified. The drawback of 2D images is that the blood vessel orientation parameter (θ in Figure 5) is not known. v and φ v ) cannot be determined. Therefore, it is assumed that the blood vessel is horizontal, i.e., parallel to the X-axis in FIG. 5. Based on this assumption, the preset insertion angle φ of the cannulation tool is v An angle (e.g., 25 degrees) is provided and the instrument is inserted in a linear trajectory at a preset angle. If the ultrasound probe is not aligned with the vessel center or has become displaced from the vessel center, it can be realigned by translating it along the transverse direction (Y).
[0065] In a second embodiment, a single 2D ultrasound image can be acquired in a longitudinal orientation. In this case, the vessel depth (Z v ) and the blood vessel orientation parameter φ v Both are visualized in the ultrasound image. It is also possible to visualize whether the blood vessel is straight or curved. v can be selected anywhere within the field of view of the 2D ultrasound image. The vascular target location may be determined by an automated process, for example, based on computer analysis of the 2D image. Alternatively, the vascular target location may be selected manually. As with the first embodiment, color Doppler can be used to confirm the presence of sufficient blood flow through the vessel. A drawback of this embodiment is that it is not possible to confirm whether the 2D imaging plane accurately cuts through the center of the vessel. In other words, the parameter Y in FIG. vThe insertion must be assumed to be in the center of the vessel, which cannot be confirmed from a single image slice. This drawback can be overcome by having a human monitor the insertion process.
[0066] In a third example, multiple 2D images can be acquired in a common orientation, such as a transverse orientation. This may be accomplished via a manual "sweep" of a handheld imaging probe. The longitudinal vessel orientation may be determined using a "sweep" of the probe along the X-axis while keeping the probe in a transverse orientation. The sweep may be relatively short, such as only about 10 mm. Each 2D image acquired during the sweep may be processed and stacked to construct a 3D image from the 2D slices. Then, θ v and φ v The vascular posture parameters, including: may be calculated from the constructed 3D image. In this regard, the insertion angle for vascular cannulation may be changed based on the 3D image.
[0067] In a fourth embodiment, multiple 2D ultrasound images can be acquired in a longitudinal orientation. As in the third embodiment, this can be accomplished using a "sweep" of the probe. The sweep is performed by scanning the center (Y v ) may be relatively short, such as only about 5 mm or less, to construct a 3D image from 2D image slices that precisely identify the
[0068] In a fifth embodiment, "bi-planar" ultrasound images can be obtained as opposed to relying on 2D images. This requires the use of a bi-planar probe capable of simultaneously acquiring both transverse and longitudinal images. The simultaneously acquired images can then be processed together to determine the necessary parameters of the target vessel, including vessel length, depth and orientation, as well as to identify the vessel center. A trajectory can then be determined that can position the tip of the cannulation tool directly over the vessel center and align with its axis during insertion and puncture. In this embodiment, a single bi-planar image may be sufficient, but acquiring multiple bi-planar images can improve the overall accuracy of the calculations.
[0069] In a sixth embodiment, a 3D volumetric ultrasound image can be obtained, which provides complete information about the center, orientation and curvature of the vessel, from which an optimal trajectory can be determined.
[0070] When a 3D volumetric image is acquired, one or more 2D or bidirectional slices may be extracted from the volume at any orientation for display. For example, an oblique slice containing the center of a vessel in a longitudinal orientation may be determined from the 3D volume and displayed as a single 2D image frame. As the probe moves, the orientation of the oblique slice may be updated so that the vessel is always displayed in the resulting 2D image as long as the vessel is somewhere within the 3D field of view of the imaging probe. Similarly, the oblique slice can be updated to always display the shaft of an instrument as long as the shaft is somewhere within the 3D imaging volume. Additionally, multiple slices can be displayed as separate images, e.g., an image slice containing a vessel and another slice containing an instrument can be displayed separately.
[0071] In addition to the above examples, secondary imaging techniques and devices may be combined to provide the missing information. For example, camera-based imaging may be used. This may be particularly useful when ultrasound imaging cannot provide all of the required 3D pose parameters (as is the case with both longitudinal and transverse 2D ultrasound).
[0072] For example, camera-based imaging may be used to determine the orientation of the imaging probe, the cannulation device, or both, relative to the patient's skin surface. One or more cameras may be attached to the cannulation device (e.g., on a sidewall of the imaging probe) and pointed toward the imaging probe to track features of the skin surface as the device moves over the patient's skin. With proper camera calibration, the camera images can be used to determine the 3D coordinates of the skin surface relative to the probe. However, because the target vessel is subcutaneous, it may still be desirable to determine the orientation of the target vessel relative to the patient's skin surface based on the imaging probe data.
[0073] Since the camera moves with the cannulation device, its coordinates must also move in order to properly characterize the patient or device motion. The transformation used to arrive at the 3D pose of the patient's skin based on the camera image data can be characterized as follows: JPEG0007588069000004.jpg674
[0074] Additionally or alternatively, camera-based imaging may be used to track changes in the orientation of the cannulation tool when it is within the camera's field of view. The 3D pose of the cannulation tool based on the camera image data can be characterized as follows: JPEG0007588069000005.jpg678
[0075] Additionally or alternatively, camera-based imaging may be used to visualize blood vessels near the surface of the patient's skin. Certain wavelengths of light, particularly near-infrared wavelengths (approximately 700-1000 nm), are strongly absorbed by hemoglobin in blood from the skin surface to a depth of 4 mm. This allows for high-contrast images of blood vessels to be obtained in a top-down view (as seen from the skin surface). The 3D pose of the target blood vessel based on the camera image data can be characterized as follows: JPEG0007588069000006.jpg682
[0076] Additionally, camera-based imaging may be used to confirm blood flash in the cannula during puncture. Confirmation of successful puncture has traditionally been achieved by visualizing blood flash in the hub of the cannula. However, this is only possible when the cannula hub is in view. Camera imaging with display of the camera image on a monitor provides another means of bringing the cannula hub into view, provided, of course, that the cannula hub is within the camera's field of view.
[0077] Camera-derived parameters can add redundancy to parameters obtained using an imaging probe, such as an ultrasound probe. In some cases, camera-derived parameters may be used to fill in missing information not available from the imaging probe data. However, the resolution of cameras is often an order of magnitude lower than that of ultrasound, and therefore ultrasound should preferably remain the primary imaging modality for this technology.
[0078] Putting the imaging data and coordinate transformations together above, the 3D vessel pose relative to the cannulation tool pose can finally be characterized as follows: JPEG0007588069000007.jpg689
[0079] This relationship between vessel posture and instrument posture is not tied to a particular fixed reference frame and can therefore easily adapt as the cannulation device moves around over the target location or the target location moves around beneath the cannulation device.
[0080] In some situations, it may be beneficial to first identify the appropriate vessel or characterize its 3D pose before the cannulation procedure begins. For example, an initial imaging procedure may be useful to determine whether cannulation is necessary first, to determine a rough location for puncture, or to determine the vessel's health and / or readiness for cannulation. In such cases, it may be advantageous to perform the imaging scan without the need to fix the cannulation instrument to the imaging device, e.g., to minimize the bulk of the device during imaging. Thus, the imaging unit and the instrument positioning unit may include mechanisms to allow the cannulation instrument to be detached from the main handheld device. However, if all units are attached to the main device, the coordinate transformation remains the same as above.
[0081] Yet another challenge created by using a handheld device is the need to determine the desired trajectory of the cannulation tool without relying on a fixed coordinate system. Trajectory determination may begin with determining the depth of the blood vessel below the patient's skin surface in contact with the imaging probe (Z in FIG. 5). v ) In addition, the direction of the blood vessel at the time of cannulation (coordinates [θ v , φ v ]) may be determined. The length of the cannulation tool may be known, either as a preset value or as a manually entered value. For some operations, such as blood sampling or intravenous cannulation, the cannulation tool may be adjusted to a given insertion angle (θ n-v ) for example, for vessels less than 4 mm below the skin surface, the default insertion angle may be about 15 degrees, for vessels 4-8 mm deep, the default insertion angle may be about 20 degrees, and for vessels greater than 8 mm deep, the default insertion angle may be about 30 degrees. These values are based on recommendations in clinical practice, but may be further adjusted based on the length of the cannulation tool.
[0082] In some cases, the cannulation tool rotates about a vertical axis (θ v ) which may be beneficial if the vessel is not exactly parallel to the longitudinal x-axis. The routing of the cannulation tool allows the cannulation tool to be rotated so that it is aligned with the vessel.
[0083] In other cases, the main processor may be configured to consider additional factors (e.g., age of the patient, anatomical location of the target vessel) to determine the trajectory to the target vessel. These factors may be entered manually.
[0084] Once the desired trajectory is established and defined in the coordinate reference frame of the cannulation device, cannulation can begin. During the insertion process, the target vessel and its center may be tracked based on image data acquired continuously, frame by frame, from the imaging probe using an image tracking algorithm. "Continuous" data may mean data received at a frame rate suitable for videography. Alternatively, "continuous" may mean data acquired at a rate faster than the repetitive instructions given to the positioning unit of the cannulation device, as described below. As mentioned above, this "continuous" data acquisition allows for real-time detection of motion of both the patient and the cannulation device, and real-time adjustment of the cannulation tool's attitude accordingly. Tracking the vessel based on frame-to-frame changes is preferred as it reduces the amount of computation required by the main processor compared to making an original decision for each frame. Other forms of image data, such as camera imaging data, may also be used for tracking the motion. Any one or any combination of the cannulation tool, the cannulation device, the patient's skin, and the target vessel may be tracked.
[0085] The instrument positioning unit may be iteratively adjusted until the cannulation instrument is properly aligned to the target location. The iterative adjustments may be made to the cannulation instrument by actuators, each actuator individually controlling a degree of freedom (DOF) of the positioning unit. The target location, e.g., the center of the target vessel, may be tracked at each iteration of the image processing routine. When the target location moves, the coordinates of the target location may be updated according to the coordinate reference frame. The trajectory and pose of the cannulation instrument may then be adjusted to take into account the updated target location coordinates, and this adjustment may be included in the iteration of the coordinate update. In this way, the instrument positioning unit may operate in a closed-loop manner based on feedback based on successive images to align the instrument to the target tissue.
[0086] For example, if the device is being manually moved along the patient's forearm, this movement may change the depth of the target location, which may result in an adjustment of the end point of the cannulation tool. Similarly, if the cannulation device is in the middle of an insertion process and it is detected that the target vessel has rotated or translated horizontally, the positioning unit may adjust the orientation of the cannulation tool to take into account the new coordinates of the target vessel.
[0087] Depending on the DOF available in the positioning unit and the robot arm, there are several ways to adjust the trajectory of the cannulation instrument. For clarification, we refer again to Figure 5, which shows the specific 3D coordinate axes useful for defining the changes in the 3D pose of the cannulation instrument.
[0088] In the first embodiment, the positioning unit has an insertion distance (d n ) (e.g., the back and forth movement of the cannulation tool). In this configuration, assuming the trajectory of the cannulation tool passes through a portion of a blood vessel, DOF1 can be iteratively adjusted until the tool punctures the blood vessel at an intersection along the trajectory. When cannulating a particular portion of the blood vessel, the trajectory may be changed by reorienting the positioning unit according to the US image.
[0089] In the second embodiment, the positioning unit has two degrees of freedom. DOF1 is the same as above. DOF2 is the insertion depth (Z n ) (e.g. skin depth when the cannulation tool tip reaches the target location). In this configuration, both DOFs can be iteratively adjusted, so that if the trajectory of the cannulation tool tip becomes misaligned with the target due to, e.g., tissue or probe movement, the height of the positioning unit can be adjusted during the insertion process.
[0090] In the third embodiment, the positioning unit has three degrees of freedom. DOF1 and DOF2 are the same as above. DOF3 is the off-axis position (Y n) (e.g., lateral movement of the cannulation tool). In this configuration, the positioning device aligns the tip of the cannulation tool with the transverse center of the blood vessel (Y v ,Z v ) in the puncture. In this regard, the positioning device can compensate for off-axis (Y,Z) movements that may occur due to probe movement during puncture, tissue movement, or vessel rotation. This configuration example may be beneficial when the probe is imaged sideways, such that off-axis movements can be automatically detected and iteratively corrected from the probe images. These transverse or off-axis adjustments may thus be useful to correct for lateral movements by either the user or the patient.
[0091] In the fourth embodiment, the positioning unit has four degrees of freedom. DOF1, DOF2 and DOF3 are the same as above. DOF4 is the insertion angle (φ n ) (e.g., the angle at which the cannulation tool is tilted downward relative to the patient's skin surface). In this configuration, the positioning unit can independently control the insertion angle to determine the depth (Z v ), direction (φ v ) or both. This configuration also allows for precise cannulation along non-linear curved trajectories. For example, the device may describe an insertion trajectory that starts with a relatively steep angle (e.g., 30 degrees) and then gradually decreases to a relatively shallow angle (e.g., 10 degrees) before reaching the target location. A relatively steep insertion angle at the beginning may be advantageous to reduce the risk of vessel rotation. A relatively shallow insertion angle at the end may be advantageous to reduce the risk of penetrating the underside of the vessel wall of the target vessel. Such adjustments may be particularly important when introducing instruments into pediatric or elderly patients with small and fragile vessels, as well as high BMI patients with very deep vessels.
[0092] In the fifth embodiment, the positioning unit has five degrees of freedom. DOF1, DOF2, DOF3 and DOF4 are the same as above. DOF5 is the vertical axis (θ n ) around the vessel orientation θ v If the target vessel moves such that is no longer parallel to the longitudinal x-axis, the orientation of the cannulation tool, θ n is the blood vessel direction θ v θ n Iterative adjustments to
[0093] In the sixth embodiment, the positioning unit has six degrees of freedom. DOF1, DOF2, DOF3, DOF4 and DOF5 are the same as above. DOF6 is the longitudinal translation (X n ) (e.g., lateral movement and advancement and retraction along an axis perpendicular to the vessel depth axis). In this configuration, the positioning device may be capable of centering the cannulation tool to any position longitudinally along the vessel. This direction (X v ) is selected, the position of the instrument tip (X n ) can be set and moved iteratively to align with the target position.
[0094] Compatibility between each DOF and the corresponding US imaging orientation is an important aspect in ensuring that the device is usable. For example, it may be advantageous to combine the above 3DOF configuration with a transverse US probe orientation rather than a longitudinal orientation. In contrast, n DOF = X n When replacing DOF, it may be advantageous to use a longitudinal orientation rather than a lateral orientation. n ) and orientation (φ n ) combination of DOFs allows for nonlinear insertion trajectories, which can be adapted to a primarily longitudinal probe.
[0095] In some embodiments, the DOF of the positioning unit may include one or more remote centers of motion (RCMs) located around the imaging axis of the imaging unit (e.g., in an ultrasound imaging unit, the imaging axis of the ultrasound transducer). In other words, the axes of the RCMs may correspond to the axes of the imaging unit, so that any rotational manipulation of the cannulation tool around the RCMs results in a manipulation such that the estimated tip position of the cannulation tool is centered below (e.g., within the field of view) of the imaging unit upon insertion into the target vessel. The RCMs, in addition to the DOF of insertion depth (z-axis), may be such that different vessels at different depths can be targeted within a single field of view of the imaging unit without requiring device movement.
[0096] A force sensor (e.g., as described in connection with FIG. 4) may further provide continuous feedback regarding the force exerted on the cannulation tool. This feedback may help to confirm puncture of the target vessel wall. In an ideal case, the puncture detected by the force sensor is T vessel_to_instrument This should occur when the movement is ≈0, meaning that the tip of the cannulation tool has just reached the coordinates of the target location.
[0097] Additionally or alternatively, feedback from the force sensor may be used to control the attitude and insertion trajectory of the cannula immediately prior to puncturing the vessel. Immediately prior to puncturing the vessel, force sensor data may be indicative of vessel motion, such as vessel rotation imposed by pressure applied from the tip of the cannula. Vessel rotation and other forms of vessel motion can result in displacements on the order of sub-millimeters. Yet these small displacements can be the difference between successful puncture, puncture of the vessel sidewall, or complete deviation from the target vessel.
[0098] In order to correct for vessel motion, it is necessary to translate the force sensor feedback into the type of vessel motion, which can be done by providing the force sensor data to either a main processor or a microprocessor and then comparing the force sensor data to pre-stored force data profile information to characterize the vessel motion.
[0099] 6A-6D show various force sensor profiles associated with various blood vessel movements. FIG. 6A shows a successful puncture profile, characterized by a gradual increase in force on the cannulation tool over a few seconds, followed by a sharp increase in force just prior to puncture, followed by a slight decrease in force for another few seconds just after puncture. In a successful puncture profile, it can be assumed that the blood vessel did not move during the insertion and puncture process.
[0100] 6B shows the profile of the cannula overshooting through the back wall of the target vessel. The profile is characterized by a sudden increase in force, followed by a sudden decrease in force, followed by a sudden re-increase in force, followed by a slower and gradual decrease in force. Given the overshoot profile, it can be determined that the cannula and / or vessel have become out of position. The cannulation tool may then be repositioned to ensure successful puncture at the center of the vessel.
[0101] 6C shows the profile of the cannula resting against the front wall of the target vessel. The profile is characterized by, among other things, a sudden increase in force followed by an absence of force. Given the undershoot profile, the cannulation instrument may be repositioned to ensure successful puncture at the center of the vessel.
[0102] Finally, FIG. 6D shows a profile of a cannula that has deviated along the sidewall of a target vessel. A common reason for contact along the vessel sidewall is rotation of the vessel caused by initial contact between the cannula and the vessel. The profile is characterized, among other things, by the absence of a sudden increase in force, indicating that the vessel wall has moved out of position and has not been punctured at all. Given the missed puncture profile, the cannulation instrument may be repositioned to ensure successful puncture at the center of the vessel on further attempts.
[0103] In addition to the force sensor feedback, other feedback sensor information can be used to characterize a successful puncture. For example, a fluid flush detector may be used to detect the appearance of fluid at the hub of the cannulation tool (e.g., blood flush of a needle inserted into a vein). The force data may be combined with the detection of the fluid flush to determine the success of the intended puncture. The combination of the fluid flush data and the force sensor feedback may be sufficient to guide an operator of a handheld venipuncture device to a successful puncture, even in devices with a limited number of degrees of freedom. For example, in the case of a device with a single degree of freedom, the combination of the fluid flush data and the force sensor feedback may be sufficient to indicate to the device when to stop the advancement of the cannulation tool or when the tool needs to be retracted or withdrawn due to a failed attempt.
[0104] As a further example, for devices with no degrees of freedom, the combination of fluid flush data and force sensor feedback may be provided to the clinician via a display to improve the clinician's ability to manually successfully advance the instrument towards the target vessel and not overshoot the vessel. However, automating the insertion of the instrument may be beneficial for several reasons. First, automation facilitates reduced movement on the part of the user as the instrument is being inserted. Additionally, it allows for increased control over the speed at which the instrument is inserted. Differences in insertion speed may affect the puncture force exerted on the instrument, so that a higher speed may exert less force and a lower speed may exert more force. By controlling the insertion speed, including ensuring a sufficiently slow insertion, more accurate force measurements may be collected.
[0105] FIG. 7 is a flow chart of an exemplary protocol 700 for performing an automated method of aligning the trajectory of a cannulation device to a target location.
[0106] In block 702, an imaging probe is used to scan the area surrounding the target vessel. As discussed above, the scan may include acquiring a single 2D image in one direction, multiple 2D images in one direction, a single 2D image in multiple directions, or other combinations of 2D images. Additionally, in some cases, the imaging probe is attached to a handheld cannulation device so that the probe can be swept over the area surrounding the target location, and the imaging data can then be stitched together to obtain a 3D image of the target.
[0107] A target vessel is identified within the scanned region at block 704. The target vessel may be identified manually or with an imaging device such as an ultrasound imager or a camera.
[0108] In block 706, the cannulation device processor determines a final position of the tip of the cannulation tool. The final position is typically a center point within the target vessel. The final position may be defined in terms of the vessel geometry, including the vessel's spatial coordinates [x,y,z], and further including the vessel's polar coordinates of rotation [θ,φ].
[0109] In block 708, the processor determines a trajectory to guide the cannulation tool to the final position. The trajectory is based on the current attitude and position of the cannulation tool as indicated by position encoders corresponding to the motors of the positioning unit that manipulate the cannulation tool. Furthermore, different aspects of the trajectory may be controlled by separate motors, each motor itself corresponding to different degrees of freedom (DOF) for manipulating the cannulation tool. The various DOF axes and the effects of manipulating the cannulation tool along such axes are described above.
[0110] In block 710, the processor may instruct the positioning unit to steer the cannulation tool towards a final location, which may include aligning and positioning the cannulation tool at the target location, steering the cannulation tool along a trajectory towards the target location, or both.
[0111] In some embodiments, the instructions in block 710 may be given iteratively, meaning that the positioning unit moves the cannulation tool incrementally towards the target location, but not necessarily completely to the target location. Iterative positioning allows the processor to receive more imaging data, process the newly received imaging data, and make corrections to the target location, trajectory, or both, before the positioning unit continues manipulating the cannulation tool. At this point, in block 712, the operation may continue with the processor receiving additional images or other sensor data, such as feedback data from a force sensor or a position sensor. Next, in block 714, the processor determines whether either or both of the cannulation device (e.g., cannulation tool, imaging probe) or the patient (e.g., the patient's skin, the patient's target vessel) have moved in a manner inconsistent with the instructions given to the positioning unit. If no movement is detected, the operation may continue to the next iteration of instructions to the positioning unit in block 710, or may continue to block 720, described below.
[0112] Conversely, if movement is detected, in block 716, a new target position, a new trajectory, or both may be determined. In order to avoid starting from scratch, the detected movement may be characterized in terms of previously collected data, and in particular in terms of coordinates that defined the previously collected data. Then, in block 718, the processor may instruct the positioning unit to realign the cannulation instrument to the new target position, to steer the cannulation instrument along the new trajectory, or both. In some cases, the processor may be configured to determine the axis or axes along which the movement occurred, and then provide instructions to certain motors of the positioning unit that control the movement along the determined axes. As in block 710, instructions may be provided iteratively. Operation may then continue to block 712.
[0113] In block 720, the processor may determine whether the tip of the cannula of the cannulation tool has reached the target location. If the target location has not been reached, the operation may continue to block 710 with further iterative movement towards the target location. However, if the target location has been reached, the operation may optionally continue to block 722 where the processor instructs the positioning unit to stop moving the cannulation tool. The processor may determine when the target has been reached by tracking the target and tool pose and further analyzing the force sensor information. By iteratively tracking the position, orientation or both of both the tool tip and the center of the target, the processor may confirm that the tool has reached the target when the two tracked values converge. Additionally, the force sensor may provide an instantaneous, either-or confirmation of target penetration.
[0114] In some embodiments, the determination of whether the target penetration was successful may be binary, whereby in the event of success the cannulation tool stops moving and in the event of failure the trajectory may be adjusted to retry the cannulation attempt. Alternatively, the likelihood of success or failure may be determined and the trajectory adjusted to increase the likelihood of success and / or decrease the likelihood of failure. In this manner, adjustments may be made to the trajectory of the cannulation tool even during the middle of an insertion attempt before determining whether the insertion was successful.
[0115] The determination during insertion may be made using a machine learning algorithm programmed to predict the success or failure of target puncture. The algorithm may be trained on logged data recorded from past puncture attempts of the same or similar devices. The logged data may include indications of the success or failure of the attempt and other collected data such as, but not limited to, image data (e.g., ultrasound information for tracking vascular movement), force sensor feedback, cannula attitude data, device attitude data, etc. The same or similar data may then be collected from the device's sensors and probes during the current session and used to predict the likelihood of success or failure. The program may be further configured to analyze how likely success or failure is due to manipulation of the cannulation tool (e.g., inserting the cannulation tool deeper, moving the tool side to side, changing the tool pitch, etc.). The program may further determine whether any of the analyzed adjustments most advantageously improve the likelihood of success (or minimize the likelihood of failure), and the processor may control the manipulation of the cannulation tool accordingly.
[0116] In some cases, machine learning algorithms use multivariate learning processes such as classification, regression, clustering, etc. to categorize training data and then determine the appropriateness of the class or classification that the current data falls into. In other cases, machine learning algorithms may use neural networks to process many raw inputs into a single probability output of success / failure.
[0117] To illustrate the machine learning based routine, the following exemplary application is described. During insertion but prior to vascular puncture, a vessel may begin to rotate out of position on the insertion path. Vessel rotation can be detected using one or more sensors including ultrasound imaging and force sensors. However, it may be difficult to characterize vessel rotation with complete certainty until after the fact. Thus, rather than recognizing vessel rotation after the fact, a machine learning algorithm can predict in advance that the vessel is likely to rotate out of trajectory, and the trajectory of the cannulation instrument with the target vessel may be immediately adjusted to minimize the chance that vessel rotation actually occurs.
[0118] The above examples generally describe the attachment of a cannulation tool. The tool may be, for example, a needle or a catheter, depending on the particular application at a given time. Furthermore, the cannulation device may be designed to receive several different types of cannulation tools. For example, the robotic arm of the device may be designed to receive either a needle or a catheter. Furthermore, the manipulators for each of the needle and catheter may be further designed to correspond to the same robotic arm, so that one tool can be easily removed and the other tool can be attached. The tools may be attached and removed using an electromagnet-based attachment component. An example of an electromagnet-based attachment component is shown in the exploded view of FIG. 8, using the attachment of a blood collection needle 802 for illustration. The blood collection needle 802 is generally cylindrical in shape and has a first segment 804 having a relatively large diameter and a second segment 806 having a relatively small diameter, tapering from the first segment to the second segment. The adapter 810 is configured to clip onto the needle 802 at the first segment 804 and the second segment 806, respectively. Specifically, the adapter 810 comprises an upper member 811 having a flat upper surface 812 and two clips 814 and 816 attached to the lower surface. The clips 814 and 816 are adapted to engage the corresponding segments 804 and 806 of the needle 802, respectively. The middle portion 825 of the upper member is hollow and a steel plate 835 is inserted into the middle portion 825. The steel plate is configured to engage with an electromagnet provided on a robotic arm (not shown). This design can be utilized to allow quick attachment and detachment of various cannulation tools, which allows easy switching between using a single cannulation device, for example, from catheter insertion to needle insertion.
[0119] The mounting components allow the handheld device to be easily compatible with conventional needle and / or catheter insertion devices. For example, angle adjustments of 0-70 degrees, injection forces of 0-15N, and insertion depths of 0-100mm may all still be possible depending on the type of vascular access being performed.
[0120] In the above embodiment where an ultrasound transducer is used, a gel mounting component may be used that is attached to the head of the transducer and provides an acoustic coupling source when the transducer is in contact with the skin. The gel mounting component may consist of a clip that holds the acoustic coupling gel. Additionally, the acoustic coupling gel may be formed to contain an anesthetic designed to be released via ultrasound energy, thereby numbing the insertion site prior to puncture. The gel mounting component may also be used to stabilize the blood vessel or other tissue target. Additionally, the gel mounting component may be flexed to conform to the geometry of the patient's arm. Finally, the ultrasound gel mounting component and the instrument mounting component of FIG. 8 may be incorporated into a single disposable housing unit. In this way, the ultrasound and instrument mounting components may be mounted to the handheld device in one motion. For example, the two components may be fixed in a pre-set geometry within the disposable housing unit such that when the device is positioned and lowered on the disposable housing unit, the instrument mounting component engages the positioning unit and the ultrasound gel mounting component engages the ultrasound imaging unit.
[0121] The above examples generally describe device protocols using out-of-plane imaging approaches to track the insertion of a cannulation tool. However, similar methods and protocols are applicable to in-plane imaging techniques where the cannulation tool is present in the image even before reaching the target location. Figures 9A-9C are flow charts of three example protocols 900a, 900b, and 900c for performing various automated methods of aligning the trajectory of a cannulation device to a target location depending on whether the imaging approach is in-plane or out-of-plane, and depending on the particular inputs available to track the alignment. Each example relies on two-dimensional ultrasound imaging as the primary source of image data, however, the two-dimensional imaging in the example protocol 900a of Figure 9A is in-plane, whereas the two-dimensional imaging in the example protocols 900b and 900c of Figures 9B and 9C is out-of-plane, respectively. Additionally, protocol 900c relies on data received from a gyroscope and a secondary imaging source (eg, a camera), whereas protocol 900b does not.
[0122] 9A, protocol 900a begins with several manual preparation steps, including priming the device (902), clipping a desired cannulation tool to the device (904), mounting a gel attachment component to the device (906), and placing the device in a predetermined position to image the target area (e.g., over the target area) (908). In some cases, the target area may be manually scanned to the appropriate insertion site (910). During the aforementioned imaging process, two-dimensional ultrasound images may be taken by the device (912). As discussed above, these images may be taken continuously, e.g., in real time, to provide feedback regarding the movement of the target or device in the imaged target area.
[0123] To ensure tracking of the vessel in subsequent images, the vessel (e.g., vein) may be segmented in the ultrasound image. The device may check (914) whether the vessel in the captured image is properly segmented, and if not, may wait for additional scans of the target area to be acquired and processed (910, 912). If the vessel is properly segmented, the device user may attempt to hold the device in place instead of scanning a new target location (916), and operation may continue with calculation of the center coordinate of the segmented vessel (918). The coordinate data (934) may be stored for later use.
[0124] The vessel coordinates may be used to control a motor that positions the instrument to properly align the instrument trajectory with the target location, e.g., the center of the vessel. This includes Y-axis alignment of the instrument (e.g., axis Y n 9, and the actuation of motors (920) that control the Z-axis or insertion depth positioning of the instrument (e.g., along axis Z n 9, along with one or both of the above-mentioned DOFs (DOF 1, 2). The motor movements (920, 924) may be tracked by corresponding position encoders (922, 926).
[0125] A trajectory may be calculated (928) for moving the instrument to the vessel center, and the calculated trajectory may be checked (930) for alignment with the vessel center. If the alignment is incorrect, operation may return to controlling the motors (920, 924) to bring the instrument back into alignment and recalculate a trajectory (928) for moving the instrument to the target location. As explained above, this tracking may be done continuously in case either the target vessel or the device move unexpectedly. However, for clarity, tracking of trajectory alignment is shown only once in the protocol.
[0126] If the trajectory is properly aligned, kinematics software (936) included in the processor may begin to move the instrument along its insertion axis (938) based on the vessel center coordinates (934) and also based on image data collected from the ultrasound image indicating the position of the instrument tip (932). The instrument may be moved iteratively (940), periodically checking whether the tip has reached the vessel center, for example based on ultrasound imaging. If the instrument tip has not yet reached the vessel center, the instrument may be moved further forward along its insertion axis (938). Otherwise, if it is determined that the instrument has reached the vessel center, or more generally the target location, the operation may continue with the designated function of the cannulation instrument (e.g., blood withdrawal) (948).
[0127] Optionally, the decision to initiate a specified operation of the cannulation tool may be further based on force data from a force sensor (944). For example, the processor may determine (942) whether the cannulation tool has properly punctured the blood vessel based on the force sensor data (944). For example, the force sensor data may be classified based on a number of profiles (946), and if the profile of the force data matches a profile indicating proper puncture, the operation may continue with the specified function of the cannulation tool (e.g., blood withdrawal) (948). Additionally or alternatively, the processor may determine (942) whether the cannulation tool has properly punctured the blood vessel based on the fluid flush data (944), e.g., feedback from a blood flush sensor in the case of blood vessel cannulation. For example, if the fluid flush data indicates the presence of a fluid flush in the cannulation tool, this may be an indication of successful cannulation, and then the operation may continue with the specified function of the cannulation tool (e.g., blood withdrawal) (948). Otherwise, if the force sensor data falls into a different type of profile, such as vessel rotation or vessel wall overshoot, or if no fluid flush or any combination of feedback data is detected, the operation may continue with moving the instrument back along its insertion axis (938).In other cases, successful cannulation may be further based on cannula pose information, which may be image data (e.g., from an ultrasound imager) or motor feedback data.
[0128] 9B, protocol 900b is similar to protocol 900a, except that ultrasound imaging (912) is performed out-of-plane rather than in-plane. As a result, the image of the target location does not feature the cannulation instrument until later in the insertion process. Once the tip of the instrument is eventually in the image (952), its location is detected (950) and may be used as an indication of whether the instrument has reached the target location so that a cannulation operation (e.g., blood withdrawal) (948) can begin.
[0129] Additionally, in protocol 900b, the vessel center coordinates are defined in terms of the vessel's Y and Z axes (958), as opposed to the X and Z axes described in protocol 900a. The Y and Z axis coordinates are then stored (954) as inputs to the kinematics software (936) that is used to drive (938) the cannulation tool towards the target location.
[0130] Turning to FIG. 9C, protocol 900c is similar to protocol 900a in that vessel center coordinates are defined in terms of the vessel's X and Z axes (918), and similar to protocol 900b in that ultrasound imaging is performed using an out-of-plane approach. Protocol 900c differs in that a gyroscope 960 and a secondary imaging device (e.g., camera) (970) are provided to improve device orientation tracking and vessel coordinate calculation, respectively. The gyroscope collects data such as the absolute orientation of the device, which may indicate the device's yaw (962), pitch (954), and roll (966). The secondary imaging device (970) may image the target location, including the target vessel, to improve vessel tracking and provide more accurate information regarding the location of the target vessel, or a second set of information to cross-reference with data from the primary imaging source. In the case of gyroscope data, the collected data is provided as input to calculate and monitor the orientation of the device (974), which can then be used to align (920, 924) the cannulation tool to the target location and further calculate and track (928, 930) the trajectory of the cannulation tool towards the target location.
[0131] In other exemplary protocols, tracking of device orientation using a gyroscope and secondary imaging device may be performed in conjunction with in-plane imaging techniques, tracking of the Y and Z axes of the vessel coordinates, or both.
[0132] The above examples show control algorithms and designs for automating venipuncture and other cannulation methods using a handheld device. However, it should be understood by those skilled in the art that, in preferred cases, it is also possible to use the handheld device manually. Furthermore, a combination of manual input and automated control may be preferred. For example, the clinician may be left to select a preferred vein for the patient, but an automated process is used to determine the exact target location within the selected vein. In further cases, if the patient moves during the process, the clinician may choose to terminate the cannulation process without having to wait for the device to detect patient movement. Additionally, the clinician may observe image data on the display of the device to ensure that the automated process is running correctly.
[0133] The device may further comprise a combination of manual input and automated control. A particular form factor of the device may be selected to allow for discretionary use with all degrees of freedom controlled manually. Other form factors may be selected for imaging only with the cannulation unit removed, or for imaging used to guide needle insertion, or for mounting on a passive support arm that provides a fixture to the device, or for automatically orienting the device so that the needle is aligned with the vessel and then manually inserting, or for manually orienting the needle to be aligned with the vessel and then manually inserting, or for automatically orienting the needle while allowing manual adjustments and then automatically or manually inserting.
[0134] 10A to 10I show the relationship between the height (z m ) and insertion depth (Inj m 10A shows an embodiment of a device having two degrees of freedom: height, θ, and θ. In the embodiment of FIG. 10A, the height may be controlled by a motor capable of translating the needle 1080 along the direction of the Z axis. The depth of insertion may be controlled by a motor capable of translating the needle along its length in the XZ plane. In some embodiments, the pitch θ of the needle in the XZ plane may be adjusted or fixed.
[0135] The device 1000 comprises a manipulator 1010 on one side and a probe 1030, such as an ultrasound probe, on the other side. The probe 1030 may be attached to the manipulator 1010, such as by a mounting frame 1025 (see FIG. 10B). In the case of an ultrasound probe, a clip 1035, for example to hold a gel, may be attached to the bottom portion of the probe 1030.
[0136] As shown in FIG. 10B, which is an exploded view of the device 1000 of FIG. 10A, the manipulator 1010 may include a housing that houses therein a first motor 1060, such as a linear stage DC motor, for controlling the position of the device in the height direction. The height of the device may be considered as the height of the manipulator 1010 relative to the probe 1030 so that the bottom of the probe 1030 (e.g., the bottom surface of the clip 1035) can rest on or near the patient to provide proper imaging of the target location. The housing of the manipulator 1010 may also house one or more processors 1020, such as a microcontroller, that control the operation of the manipulator. The needle 1080 may be housed in a needle insertion housing 1050 located below the housing of the manipulator 1010. Further housed in the needle insertion housing 1050 may be a second motor 1040, such as a spindle drive motor, that controls the insertion depth of the needle.
[0137] 10B, the needle insertion housing 1050 may also house a guide rail 1070 therein that guides the insertion motion of the needle 1080 as the second motor 1040 drives the needle. A pair of ball bearing carriages 1072 may be coupled to the guide rail 1070 such that the needle 1080 is slidably mounted on the guide rail 1070 under the control of the second motor 1040.
[0138] Additionally, one or more sensors may be provided to provide feedback regarding the position of the device, the needle 1080, or both, and the progress of the cannulation. For example, a force sensor 1064 may be provided to detect the amount of force exerted on the needle during insertion. Additionally, a gyroscope and accelerometer may be included with the microcontroller 1020 to track the attitude and orientation of the device.
[0139] The needle may be removably attached to a clip 1078, which may be attached to the electromagnet 1074 by a washer 1076. The electromagnet may allow for easy insertion and removal of the needle clip from the needle insertion housing 1050.
[0140] 10C-10I provide additional views of portions of device 1000 to more clearly illustrate the arrangement of components described in connection with FIGS. 10A and 10B. For example, FIG. 10C illustrates device 1000 in its entirety with probe 1030 and clip 1035. FIG. 10D illustrates the device with the probe and clip removed, thereby more clearly illustrating the mounting frame 1025 to which probe 1030 may be removably attached. FIG. 10E illustrates the device with the frame and first motor removed, thereby more clearly illustrating the mounting frame used to attach needle insertion housing 1050 to the remainder of the manipulator. FIG. 10F illustrates the device with the needle housing removed, thereby more clearly illustrating the second motor 1040 used to drive the insertion depth of the needle and the guide rail 1070 along which the needle travels. FIG. 10G illustrates the device with the second motor and guide rail removed, thereby more clearly illustrating the housing of bearing 1072 used to guide the needle on the guide rail. Figure 10H shows the device with the housing removed, thereby more clearly showing the bearing 1072 and guide rail 1070, as well as the position of the electromagnet relative to the guide rail when the needle is attached, and the position of the force sensor 1064 at the rear of the needle clip. Finally, Figure 10I shows only the force sensor, electromagnet 1074, needle clip 1078 and needle 1080, as the remainder of the device has been removed.
[0141] The embodiment of Figures 10A-10I shows the structure of a device with two degrees of freedom (2-DOF). Other devices with more or fewer degrees of freedom may be designed using the exemplary device 1000 as a basic configuration. For example, Figures 11A-11G show a device 1100 with three degrees of freedom (3-DOF). In this embodiment, the three degrees of freedom include the same two degrees of freedom as the conventional device 1000, namely, needle height and insertion depth. In addition, the 3-DOF device includes an additional motor that steers the needle laterally along the direction of the Y-axis.
[0142] As shown in FIG. 11A, device 1100 is structurally designed similarly to, but not exactly the same as, the 2-DOF device 1000 described above. For example, device 1100 still has a probe side and a manipulator side, which are similarly attached to each other. Device 1100 also has a needle insertion housing, but the housing of the 3DOF system is somewhat larger than that of the 2DOF system due to the presence of additional components. Some notable differences between the two structures are described in more detail below.
[0143] Similar to FIGS. 10C-10I of the 2-DOF device 1000, FIGS. 11B-11G provide additional views of a portion of the 3-DOF device 1100 to more clearly show the orientation of the components described in connection with FIG. 11A. FIG. 11B shows the entire device 1100 with the manipulator, probe and clip. FIG. 11C shows the device with the probe, clip and mounting frame for the probe removed. The microcontroller, first motor and their housings have also been removed to more clearly show the mounting frame used to attach the needle insertion housing to the rest of the manipulator. FIG. 11D shows the device with the needle housing removed, thereby more clearly showing the lateral movement mechanism 1110 used to guide the needle along the Y-axis. The lateral movement mechanism 1110 may include a third motor 1112, such as another spindle-driven DC motor, used to drive the rotational movement of the gear 1114 along a path 1116. The track may have a length along the Y axis such that rotation of the gear 1114 translates the track 1116 to one side or the other along the Y axis. The needle clip may be coupled to the track 1116 such that translation of the track 1116 also translates the needle laterally.
[0144] Figure 11E shows the device with the third motor 1112 removed. Track 1116 is also shown coupled to the housing of the second motor so that the second motor can translate laterally with the needle to control the depth of insertion independent of the lateral position of the needle. Figures 11F and 11G are generally equivalent to Figures 10H and 10I of the 2-DOF device 1000 described above.
[0145] 12 shows an example workflow 1200 of a control device controlling needle and device movement during a cannulation event. The workflow may begin with a scanning operation (1210), whereby a probe of the device may be positioned over a target location for cannulation. Positioning the device may include collecting image data, such as image frames from a camera or video recording device.
[0146] The workflow may continue with ultrasound imaging of the target location using the device probe (1220). Ultrasound imaging may include out-of-plane imaging (1222), which may provide imaging below the patient's skin (e.g., at a depth of 0-20 mm) using ultrasound.
[0147] The workflow may continue with image analysis (1230) of the collected images. The images may be used to identify vessels for cannulation and enhance (1232) the images of the vessels, such as by determining the vessel's orientation by defining the target vessel's walls through segmentation (1234). Finally, the vessels may be selected (1236) and tracked frame by frame (1238). In the case of veins, the veins may be further analyzed to determine the blood flow through them to ensure they are good candidates for cannulation.
[0148] Once the target vessel is selected, the scan may be stopped (1240) as instructed by the user input. The user input may further serve as a decision as to whether to begin the cannulation process (1250). If it is decided not to begin cannulation (No), such as if the user is not satisfied with the identified target vessel, operation may return to image analysis (1230) so that another target vessel may be selected. Otherwise, if insertion is to begin (Yes), the workflow may continue with robotic control (1260) of the cannulation instrument, which may include calculating the kinematics of the free-held device as it moves above the target location (1262), aligning the cannulation instrument along the Y and Z axes to maintain a trajectory toward the target vessel (1264), and advancing the cannulation instrument along the insertion depth axis (1266) toward the target vessel.
[0149] Once the cannulation instrument has reached the target location, the workflow may further perform puncture detection (1270) to determine if the instrument has been or will be properly inserted into the blood vessel. Determining successful insertion may be based on several factors, including needle position as indicated by a position encoder (1272) of the motor controlling the instrument, force feedback (1274) indicated by a force sensor coupled to the instrument, ultrasound imaging (1276) to provide visual tracking of the insertion, motor current feedback to indicate if the instrument meets expected or unexpected resistance causing a spike in motor current (1278), and blood or fluid flush feedback (1280) to determine if the cannulation attempt has resulted in the collection of blood or fluid. Individually or collectively, this feedback data may be used in the robotic control stage (1260) of the workflow to correct the alignment of the instrument.
[0150] The above described devices and methods are applicable to both human patients and other animal subjects, such as veterinary applications. More generally, those skilled in the art will readily understand and appreciate that the above described devices and methods for modifying the trajectory of a handheld cannulation instrument based on movement of a target location may be applicable to any cannulation procedure in which there may be unexpected movement of the subject, unexpected movement of the holder of the device, or both.
[0151] Although the invention herein has been described with reference to particular embodiments, it is to be understood that these embodiments are merely illustrative of the principles and applications of the present invention. It is therefore to be understood that numerous modifications may be made to the exemplary embodiments and that other arrangements may be devised without departing from the spirit and scope of the present invention as defined by the accompanying drawings. The technical ideas that can be understood from the above-described embodiments will be described below. [Aspect 1] 1. A device for directing a cannula toward a subcutaneous target location in a patient, comprising: an imaging probe for continuously generating imaging data of the target location; A positioning unit, a plurality of motors configured to manipulate cannulas attached to the positioning unit; at least one cannula position sensor configured to generate cannula orientation data indicative of a position and orientation of the cannula based on corresponding motor movements; and at least one device position sensor configured to generate device attitude data indicative of an orientation and position of the device; A positioning unit comprising: 1. A processor comprising: receiving imaging data from the imaging probe; receiving cannula orientation data from the at least one cannula position sensor; receiving device attitude data from the at least one device position sensor; Identifying the target location from the imaging data; determining a trajectory for steering the cannula towards the target pose based on the imaging data, the cannula pose data and the device pose data; instructing the positioning unit to steer the cannula according to the determined trajectory; Identifying when the determined trajectory has deviated from the target position; updating the determined trajectory to a modified trajectory based on the imaging data, the cannula attitude data, and the device attitude data; Instructing the positioning unit to steer the cannula according to the modified trajectory. a processor configured to A device comprising: [Aspect 2] The device of aspect 1, wherein the imaging probe is an ultrasound probe, and the imaging data generated by the imaging probe is any one of two-dimensional imaging data, three-dimensional volumetric imaging data, and two-directional imaging data. [Aspect 3] The device of aspect 2, wherein the processor is further configured to identify and determine a posture of a target vessel underlying the patient based on the imaging data. [Aspect 4] The device is configured to acquire imaging data of the target location from a plurality of angles, and the processor: constructing three-dimensional imaging data from the received imaging data acquired from the plurality of angles; and determining a three-dimensional pose of the target location subcutaneously of the patient from the imaging data acquired from the multiple angles; The device of embodiment 3, further configured to perform at least one of the following: [Aspect 5] A device as described in any one of aspects 1 to 4, wherein the imaging data includes video data of the target position acquired at different times, and the processor is configured to determine the trajectory for maneuvering the cannula toward the target posture based on a comparison between previous and subsequent frames of the video data. [Aspect 6] Aspect 6. The device of any one of aspects 1 to 5, wherein the positioning unit is configured to manipulate the cannula in at least two degrees of freedom. [Aspect 7] A device as described in any one of aspects 1 to 6, wherein the processor is configured to identify that the determined trajectory has misaligned with the target position based on device attitude data received from the at least one device position sensor. [Aspect 8] The device of aspect 7, wherein the at least one device sensor includes at least one of a camera, an electromagnetic sensor, a gyroscope, and an accelerometer. [Aspect 9] and a force sensor configured to generate force data indicative of a measure of a force exerted on the cannula, the processor further comprising: receiving the force data from the force sensor and updating the cannula position data to a corrected position and orientation based on the received force data; instructing the cannula positioning unit to manipulate the cannula according to the modified position and orientation. 9. The device of any one of aspects 1 to 8, configured to [Aspect 10] a blood flush sensor configured to generate blood flush data indicative of blood flow into the cannula, the processor further comprising: receiving the blood flush data from the blood flush sensor; Instructing the cannula positioning unit to manipulate the cannula further based on the blood flush data. The device of embodiment 9, configured as follows: [Aspect 11] a force sensor configured to generate force data indicative of a measure of a force exerted on the cannula, the processor receiving the force data from the force sensor; obtaining a force profile from the force data; classifying the acquired force profile as one of a successful cannulation event, a vessel rotation event, an undershoot event, and an overshoot event; In the absence of a successful cannulation event, instruct the positioning unit to manipulate the cannula according to the classified event. 9. The device of any one of aspects 1 to 8, configured to [Aspect 12] A device as described in any one of aspects 1 to 11, wherein the at least one cannula position sensor comprises a plurality of motor sensors, each motor sensor associated with a corresponding one of the plurality of motors generating cannula posture data. [Aspect 13] 1. A method of guiding a cannula towards a subcutaneous target location on a patient, the method being executed by a processor coupled to an imaging probe for continuously generating imaging data of the target location, and a positioning unit comprising a plurality of motors, at least one cannula position sensor and at least one device position sensor, the plurality of motors being configured to manipulate a cannula attached to the positioning unit, the at least one cannula position sensor being configured to generate cannula orientation data indicative of a position and orientation of the cannula based on movements of corresponding motors, and the at least one device position sensor being configured to generate device orientation data indicative of an orientation and position of the device, the method comprising: receiving imaging data from the imaging probe; receiving cannula orientation data from the at least one cannula position sensor; receiving device attitude data from the at least one device position sensor; identifying the target location from the imaging data; and determining a trajectory for steering the cannula towards the target pose based on the imaging data, the cannula pose data and the device pose data; instructing the positioning unit to steer the cannula according to the determined trajectory; identifying when the determined trajectory has deviated from the target position; and updating the determined trajectory to a modified trajectory based on the imaging data, the cannula attitude data, and the device attitude data; instructing the positioning unit to steer the cannula according to the modified trajectory; A method comprising: [Aspect 14] The method further comprises: identifying and determining a pose of a target vessel underlying the patient based on the imaging data; instructing the positioning unit to steer the cannula based on the determined orientation of the target vessel; 14. The method of embodiment 13, comprising: [Aspect 15] Receiving imaging data from the probe includes receiving imaging data from a plurality of angles, the method comprising: constructing three-dimensional imaging data from the received imaging data acquired from the plurality of angles; and determining the pose of the target location subcutaneously on the patient from the imaging data acquired from the multiple angles; 14. The method of embodiment 13, comprising at least one of: [Aspect 16] Aspect 16. The method of any one of aspects 13 to 15, wherein instructing the positioning unit to manipulate the cannula includes providing instructions for manipulating each of two or more degrees of freedom of the positioning unit. [Aspect 17] A method according to any one of aspects 13 to 16, wherein identifying that the determined trajectory has deviated from the target position is based on the device attitude data received from the device position sensor. [Aspect 18] 18. The method of any one of aspects 13 to 17, wherein determining the cannula posture data is based on the imaging data using at least one of a computer vision algorithm or an electromagnetic sensor. [Aspect 19] The processor is further coupled to a force sensor that generates force data indicative of a measure of force exerted on the cannula, the method comprising: receiving the force data from the force sensor; updating the cannula positioning data to a corrected position and orientation based on the received force data; instructing the cannula positioning unit to manipulate the cannula according to the modified position and orientation; 19. The method of any one of embodiments 13 to 18, further comprising: [Aspect 20] The processor is further coupled to a force sensor that generates force data indicative of a measure of force exerted on the cannula, the method comprising: obtaining a force profile from the force data; and classifying the acquired force profile as one of a successful cannulation event, a vessel rotation event, an undershoot event, and an overshoot event; instructing the positioning unit to manipulate the cannula in response to the classified event if there is no successful cannulation event; 19. The method of any one of embodiments 13 to 18, further comprising: [Aspect 21] 1. A device for inserting a needle into a patient's blood vessel, comprising: an ultrasound probe configured to image a portion of the blood vessel disposed within a field of view of the ultrasound probe; a mechanical arm attached to the ultrasound probe, the needle being attachable to the mechanical arm, the mechanical arm comprising one or more motors configured to manipulate a position and orientation of the needle; a processor for controlling operation of the one or more motors of the mechanical arm, the processor being configured to define a frame of reference for controlling the one or more motors based on spatial parameters of the blood vessel; Equipped with the mechanical arm is configured to be capable of inserting the needle at an angle into the portion of the blood vessel located within a field of view of the ultrasound probe; a processor controlling operation of the one or more motors is automated based at least in part on the image data from the ultrasound probe; The device is configured to be handheld. [Aspect 22] 22. The device of claim 21, wherein the motor of the mechanical arm is configured to manipulate at least the insertion depth of the needle and the insertion angle of the needle. [Aspect 23] The device of aspect 22, wherein the processor is communicatively coupled to the probe and configured to determine the insertion depth based at least in part on data received from the probe, and the insertion angle is determined based on the insertion depth. [Aspect 24] 24. The device of any one of aspects 21 to 23, further comprising an accelerometer and a gyroscope configured to provide data to the processor indicative of movement and orientation of the device, respectively, wherein the processor controlling operation of the one or more motors is further automated based in part on the data from the accelerometer and gyroscope. [Aspect 25] 1. A device for directing a cannula toward a subcutaneous target location in a patient, the device comprising: at least one motor configured to steer the cannula along a length of the cannula toward a target; a force sensor configured to generate force data indicative of a measure of force exerted on the cannula; a blood flush sensor configured to generate blood flush data indicative of blood flow into the cannula; 1. A processor comprising: receiving the force data from the force sensor; receiving the blood flush data from the blood flush sensor; determining the occurrence of a successful cannulation event based on a combination of the force data and the blood flush data; instructing the at least one motor to cease steering the cannula toward the target upon the occurrence of a successful cannulation event. a processor configured to A device comprising: [Aspect 26] The apparatus of aspect 25, further comprising a cannula position sensor configured to generate cannula posture data indicative of a position of the cannula based on movement of the at least one motor, and the processor configured to determine the cannula insertion success event based on a combination of the force data, the blood flush data and the cannula posture data. [Aspect 27] an imaging probe for acquiring ultrasound imaging data, the processor further comprising: determining a likelihood of a cannulation failure event based on a combination of the force data, the blood flush data, the ultrasound imaging data, and the cannula position data; determining adjustments to the cannula to reduce the likelihood of the cannulation failure event based on a combination of the force data, the blood flush data, the ultrasound imaging data, and the cannula position data; 27. The device of embodiment 26, configured so as to [Aspect 28] The processor, obtaining a force profile from the received force data; classifying the acquired force profile as either a successful cannulation event or a failed cannulation event; Instructing the one or more motors to operate the cannula in a reverse direction upon a cannulation failure event. 27. The device of any one of embodiments 25 to 26, configured to
Claims
1. 1. A device for directing a cannula toward a target location, the target location being a central location of a target blood vessel subcutaneously in a patient, comprising: an imaging probe for continuously generating imaging data of the target location; A positioning unit, a plurality of motors configured to manipulate a cannula attached to the positioning unit toward the target location; at least one cannula position sensor configured to generate cannula orientation data indicative of a position and orientation of the cannula based on corresponding motor movements; and at least one device position sensor configured to generate device attitude data indicative of an orientation and position of the device; A positioning unit comprising:
1. A processor comprising: receiving imaging data from the imaging probe; receiving cannula orientation data from the at least one cannula position sensor; receiving device attitude data from the at least one device position sensor; During an initial setup, identifying the target location from the imaging data using an automatic segmentation algorithm that (i) segments a boundary of the target vessel, and (ii) calculates a center location of the segmented target vessel; determining a trajectory for steering the cannula towards the target location based on the imaging data, the cannula attitude data, and the device attitude data; instructing the positioning unit to steer the cannula according to the determined trajectory; After the initial setup, tracking the target location based on subsequent frames from the imaging data using an image-based tracking algorithm; Identifying, based on the imaging data, that the determined trajectory has deviated from the target position; updating the determined trajectory to a modified trajectory based on the imaging data, the cannula attitude data, and the device attitude data; Instructing the positioning unit to steer the cannula according to the modified trajectory. a processor configured to Equipped with the device further comprising a force sensor configured to generate force data indicative of a measure of force exerted on the cannula; The processor, receiving the force data from the force sensor; classifying the force data as one of a plurality of predetermined events, including a successful cannulation event; In the absence of the successful cannulation event, instruct the positioning unit to manipulate the cannula according to the classified event. The device is configured to:
2. The device of claim 1 , wherein the imaging probe is an ultrasound probe, and the imaging data generated by the imaging probe is one of two-dimensional imaging data, three-dimensional volumetric imaging data, and bidirectional imaging data.
3. The device of claim 2 , wherein the processor is further configured to identify and determine a pose of an underlying target vessel of the patient based on the imaging data.
4. The device is configured to acquire imaging data of the target location from a plurality of angles, and the processor: constructing three-dimensional imaging data from the imaging data acquired from the multiple angles; and determining a three-dimensional pose of the target location subcutaneously of the patient from the imaging data acquired from the multiple angles; The device of claim 3 , further configured to perform at least one of the following:
5. 5. The device of claim 1, wherein the imaging data includes video data of the target location acquired at different times, and the processor is configured to determine the trajectory for steering the cannula towards the target location based on a comparison between previous and subsequent frames of the video data.
6. The device of claim 1 , wherein the positioning unit is configured to manipulate the cannula in at least two degrees of freedom.
7. 7. The device of claim 1, wherein the processor is configured to identify when the determined trajectory has deviated from the target position based on device attitude data received from the at least one device position sensor.
8. The device of claim 7 , wherein the at least one device sensor includes at least one of a camera, an electromagnetic sensor, a gyroscope, and an accelerometer.
9. and a force sensor configured to generate force data indicative of a measure of a force exerted on the cannula, the processor further comprising: receiving the force data from the force sensor and updating the cannula position data to a corrected position and orientation based on the received force data; instructing the cannula positioning unit to manipulate the cannula according to the modified position and orientation. A device according to any one of claims 1 to 8, configured so as to
10. a blood flush sensor configured to generate blood flush data indicative of blood flow into the cannula, the processor further comprising: receiving the blood flush data from the blood flush sensor; Instructing the cannula positioning unit to manipulate the cannula further based on the blood flush data. The device according to claim 9 , configured to:
11. The processor, obtaining a force profile from the force data; Classifying the acquired force profile as one of a successful cannulation event, a vessel rotation event, an undershoot event, and an overshoot event. A device according to any one of claims 1 to 8, configured so as to
12. 12. The device of claim 1, wherein the at least one cannula position sensor comprises a plurality of motor sensors, each motor sensor associated with a corresponding one of the plurality of motors generating cannula posture data.
13. The device of claim 1 , wherein the automatic segmentation algorithm segments a boundary of the target vessel based in part on blood flow information.
14. 1. A method of guiding a cannula towards a target location, which is a central location of a target blood vessel subcutaneously in a patient, the method being executed by a processor coupled to an imaging probe for continuously generating imaging data of the target location, and a positioning unit comprising a plurality of motors, at least one cannula position sensor and at least one device position sensor, the plurality of motors configured to steer a cannula attached to the positioning unit towards the target location, the at least one cannula position sensor configured to generate cannula orientation data indicative of a position and orientation of the cannula based on movements of corresponding motors, and the at least one device position sensor configured to generate device orientation data indicative of an orientation and position of the device, the method comprising: receiving imaging data from the imaging probe; receiving cannula orientation data from the at least one cannula position sensor; receiving device attitude data from the at least one device position sensor; During an initial setup, identifying the target location from the imaging data using an automatic segmentation algorithm (i) that segments a boundary of the target vessel, and (ii) that calculates a center location of the segmented target vessel; determining a trajectory for steering the cannula towards the target location based on the imaging data, the cannula attitude data, and the device attitude data; instructing the positioning unit to steer the cannula according to the determined trajectory; tracking the target location based on subsequent frames from the imaging data using an image-based tracking algorithm after the initial setup; Identifying, based on the imaging data, that the determined trajectory has deviated from the target location; and updating the determined trajectory to a modified trajectory based on the imaging data, the cannula attitude data, and the device attitude data; instructing the positioning unit to steer the cannula according to the modified trajectory; Including, the processor is further coupled to a force sensor that generates force data indicative of a measure of force exerted on the cannula; The method comprises: classifying the force data as one of a plurality of predefined events, including a successful cannulation event; instructing the positioning unit to manipulate the cannula according to the classified event if the successful cannulation event is absent; The method further comprising:
15. The method further comprises: identifying and determining a pose of a target vessel underlying the patient based on the imaging data; instructing the positioning unit to steer the cannula based on the determined orientation of the target vessel; 15. The method of claim 14, comprising:
16. Receiving imaging data from the probe includes receiving imaging data acquired from a plurality of angles, the method comprising: constructing three-dimensional imaging data from the imaging data acquired from the multiple angles; and determining the pose of the target location subcutaneously on the patient from the imaging data acquired from the multiple angles; The method of claim 14, comprising at least one of:
17. 17. The method of any one of claims 14 to 16, wherein instructing the positioning unit to manipulate the cannula comprises providing instructions for manipulating each of two or more degrees of freedom of the positioning unit.
18. 18. The method of claim 14, wherein identifying when the determined trajectory has deviated from the target position is based on the device attitude data received from the device position sensor.
19. 19. The method of any one of claims 14 to 18, wherein determining the cannula posture data is based on the imaging data using at least one of a computer vision algorithm or an electromagnetic sensor.
20. The processor is further coupled to a force sensor that generates force data indicative of a measure of force exerted on the cannula, the method comprising: receiving the force data from the force sensor; updating the cannula positioning data to a corrected position and orientation based on the received force data; instructing the cannula positioning unit to manipulate the cannula according to the modified position and orientation; 20. The method of any one of claims 14 to 19, further comprising:
21. The method according to claim 20, obtaining a force profile from the force data; and classifying the acquired force profile as one of a successful cannulation event, a vessel rotation event, an undershoot event, and an overshoot event; 20. The method of any one of claims 14 to 19, further comprising:
22. 1. A device for inserting a needle into a patient's blood vessel, comprising: an ultrasound probe configured to image a portion of the blood vessel disposed within a field of view of the ultrasound probe; a mechanical arm attached to the ultrasound probe, the needle being attachable to the mechanical arm, the mechanical arm comprising one or more motors configured to manipulate a position and orientation of the needle; a processor for controlling operation of the one or more motors of the mechanical arm, the processor being configured to define a frame of reference for controlling the one or more motors based on spatial parameters of the blood vessel; Equipped with the mechanical arm is configured to be capable of inserting the needle at an angle into the portion of the blood vessel located within a field of view of the ultrasound probe; a processor controlling operation of the one or more motors is automated based at least in part on image data from the ultrasound probe; The automating of operation of the one or more motors includes: identifying the blood vessel from the image data using an automatic segmentation algorithm that segments a boundary of the blood vessel during an initial setup; Calculating center coordinates of the segmented blood vessel and a trajectory of the needle towards the center coordinates; steering the needle towards the center coordinate according to the calculated trajectory; tracking the vessel based on subsequent frames from the image data using an image-based tracking algorithm without using the automatic segmentation algorithm after the initial setup; modifying the trajectory based on the tracked blood vessels; Includes: the device is configured to be handheld; the device further comprising a force sensor configured to generate force data indicative of a measure of force exerted on the needle; The processor, receiving the force data from the force sensor; classifying the force data as one of a plurality of predetermined events, including a successful needle insertion event; If there is no successful needle insertion event, controlling operation of the one or more motors to manipulate the needle in response to the classified event. The device is configured to:
23. 23. The device of claim 22, wherein the motor of the mechanical arm is configured to manipulate at least the needle insertion depth and the needle insertion angle.
24. 24. The device of claim 23, wherein the processor is communicatively coupled to the probe and configured to determine the insertion depth based at least in part on data received from the probe, and the insertion angle is determined based on the insertion depth.
25. 25. The device of claim 22, further comprising an accelerometer and a gyroscope configured to provide data to the processor indicative of a movement and an orientation of the device, respectively, and wherein the processor controlling operation of the one or more motors is further automated based in part on the data from the accelerometer and gyroscope.
26. 1. A device for directing a cannula toward a subcutaneous target location in a patient, the device comprising: at least one motor configured to steer the cannula along its length toward a target at a predetermined speed; a force sensor configured to generate force data indicative of a measure of force exerted on the cannula by the cannula moving at the predetermined velocity; a blood flush sensor configured to generate blood flush data indicative of blood flow into the cannula; 1. A processor comprising: receiving the force data from the force sensor; receiving the blood flush data from the blood flush sensor; determining the occurrence of a successful cannulation event based on a combination of the force data and the blood flush data; instructing the at least one motor to cease steering the cannula toward the target upon the occurrence of a successful cannulation event. a processor configured to A device comprising:
27. 27. The apparatus of claim 26, further comprising a cannula position sensor configured to generate cannula posture data indicative of a position of the cannula based on movement of the at least one motor, and wherein the processor is configured to determine the successful cannulation event based on a combination of the force data, the blood flush data and the cannula posture data.
28. an imaging probe for acquiring ultrasound imaging data, the processor further comprising: determining a likelihood of a cannulation failure event based on a combination of the force data, the blood flush data, the ultrasound imaging data, and the cannula position data; determining adjustments to the cannula to reduce the likelihood of the cannulation failure event based on a combination of the force data, the blood flush data, the ultrasound imaging data, and the cannula position data; 28. The device of claim 27 configured to:
29. The processor, obtaining a force profile from the received force data; classifying the acquired force profile as either a successful cannulation event or a failed cannulation event; Instructing the one or more motors to operate the cannula in a reverse direction upon the occurrence of a cannulation failure event.
28. A device according to any one of claims 26 to 27, configured so as to
Citation Information
Patent Citations
device and method for automatically venting and inflating a catheter
DE102017201439A1
Puncture support system
JP2012035010A
Ultrasonic diagnostic apparatus
JP2012090820A
Automatic injection needle insertion device
JP2014239831A
