Ultrasound-guided alignment and insertion of percutaneous catheter instruments

Abstract

A device and method for guiding a cannula toward a target position under the skin of a patient. The device is handheld and the operation of the device can be automated. The device includes an imaging probe for imaging the target position, a positioning unit for manipulating the cannula toward the target position, and a processor. The processor receives imaging data from the imaging probe, cannula posture data from at least one cannula position sensor, and device posture data from at least one device position sensor. The target position is identified by the imaging data, and a trajectory for manipulating the cannula toward the target position is determined based on the imaging data, the cannula posture data, and the device posture data. The processor can determine that the trajectory becomes misaligned with the target position, and can update it to a corrected trajectory based on the imaging data, the cannula posture data, and the device posture data.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of the filing date of U.S. Provisional Application No. 62 / 743,283, filed on October 9, 2018, entitled “Ultrasound Guided Alignment And Insertion Of Percutaneous Canulating Instrument,” the disclosure of which is hereby incorporated by reference into this document.

[0003] STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

[0004] This invention was made with government support under Grant No. R01EB020036 from the National Institutes of Health, Grant No. EB018191 from the National Institutes of Health, Grant No. T32 GM008339 from the National Institutes of Health, Grant No. DGE-0937373 from the National Science Foundation, and Grant No. P200A150131 from the Ministry of Education. Therefore, the government may have certain rights in this invention. Background Art

[0005] The venipuncture process of inserting a cannula into a vein to draw a blood sample or deliver intravenous (IV) fluids is one of the most common clinical routines performed in medicine. It is often difficult to find a suitable venipuncture site, especially for patients with dark skin or a high body mass index. Even after the target vein is identified, the process of inserting the cannula into the blood vessel can be difficult. Failed insertion attempts often occur. Sometimes, failure may be caused by unintended movement of the vein - for example, pressure exerted by the needle tip on the vein causing the vein to deform, move, or roll. At other times, failure may be caused by unintended movement by the patient or the clinician holding the cannulation instrument. A poorly injected needle may cause complications such as increased pain, internal bleeding, or leakage of intravenous fluid into extravascular tissue.

[0006] Various imaging techniques based on near infrared (NIR) light or ultrasound (US) have been introduced to assist clinicians in visualizing veins. However, studies have shown that these imaging techniques alone do not significantly improve first-time insertion success rates compared to manual techniques. In order to improve venipuncture insertion accuracy, devices have been developed that combine NIR and US imaging techniques with automated robotic systems that guide the cannula to the center of the vessel based on image data.

[0007] Figure 1is a side profile view of an exemplary prior art automated intubation system 100. The automated intubation system 100 includes a positioning unit 110 attached to an ultrasound imaging unit 120. The positioning unit 110 may include an automated robotic arm 112 configured to control the positioning of a cannula 115, such as a needle, and may include multiple degrees of freedom for manipulating the cannula 115. The ultrasound imaging unit 120 may also include a probe 122 attached to an end of the unit 120, whereby the unit 120 is configured to acquire a US image of a portion of a patient (not shown) located opposite the probe 122. The probe 122 may be made of a flexible material designed to flex and conform to an imaging portion of a patient (e.g., a patient's arm). In some examples, the probe 122 may also include a gel compartment 125 configured to contain ultrasound gel for improving the acquisition of US images.

[0008] exist Figure 2 In the present invention, a prior art automated cannulation system 100 is used to insert a needle into a patient arm 201. The patient arm 201 may be placed on a platform 210 to ensure patient comfort and stability of the arm 201. The cannula 115 may be positioned above the patient arm 201 at an angle such that the tip of the cannula 115 is positioned to engage a cannulation site 225 directly below the imaging probe 122.

[0009] Figure 3 is a side profile view of an exemplary prior art positioning unit 110 having four degrees of freedom. Positioning unit 110 includes a robotic arm 112 having a base 302, a first extension member 304 extending from base 302, a second extension member 306 connected to and extending from first extension member 304, and a cannula instrument receptacle 308 for receiving cannula 115. Figure 3 The positioning unit 110 has four degrees of freedom: (1) the base 302 rotates around the focus 312 in the xy plane; (2) the first extension member 304 rotates around the focus 314 in the xz plane (technically, the plane formed by the z-axis and the specific axis of the first extension member); (3) the second extension member 306 rotates around the focus 316 in the xz plane (technically, the plane formed by the z-axis and the specific axis of the second extension member); (4) the cannula 115 translates forward or backward along the axis 318 of the cannula 115.

[0010] Imaging technology in example automated devices and other known devices can track the patient's motion, including the motion of a patient's target location (e.g., a vein under the patient's skin). These devices can also adjust the trajectory for precise insertion into the target location. However, in order for the device to provide such precise tracking and repositioning, the device must be attached to a tabletop or other stationary platform, such as a Figure 2The benchtop platform provides the necessary stability to ensure that the intubation instruments are not accidentally moved but only guided by the positioning unit.

[0011] The need for a venipuncture device that is fixed in place prior to use is a disadvantage. First, not every clinical environment includes a benchtop or other platform on which the device can be mounted. In addition, installing and removing the device may consume a great deal of clinician time. Ultimately, while automated venipuncture devices offer some advantages over existing handheld methods of venipuncture, such as in terms of accuracy, some clinicians may find automated devices inconvenient, and some clinical environments may not even be suitable for use with automated devices. Summary of the invention

[0012] The present disclosure relates to a handheld automated intubation device. The device has the convenience and portability of being handheld while providing improved accuracy associated with other automated devices. In order to use the automated device in a handheld manner, it is necessary to design a control scheme that can respond quickly and efficiently to real-time sensor data without having to rely on a fixed reference frame external to the device itself.

[0013] One aspect of the present disclosure relates to a device for guiding a cannula toward a target location under the skin of a patient. The device may include an imaging probe for continuously generating imaging data of the target location, a positioning unit, and a processor. The positioning unit may include a plurality of motors configured to manipulate a cannula attached to the positioning unit. The positioning unit may also include at least one cannula position sensor configured to generate cannula posture data indicating the position and orientation of the cannula based on the movement of the motor. The positioning unit may also include at least one device position sensor configured to generate device posture data indicating the orientation and position of the device.

[0014] The processor may be configured to receive imaging data from an imaging probe, receive cannula posture data from at least one cannula position sensor, receive device posture data from at least one device position sensor, identify a target position from the imaging data, and determine a trajectory for manipulating the cannula toward the target posture based on the imaging data, the cannula posture data, and the device posture data. The processor may also be configured to instruct the positioning unit to manipulate the cannula according to the determined trajectory. The processor may also be configured to identify that the determined trajectory becomes misaligned with the target position, update the determined trajectory to a corrected trajectory based on the imaging data, the cannula posture data, and the device posture data, and instruct the positioning unit to manipulate the cannula according to the corrected trajectory.

[0015] In some examples, the imaging probe may be an ultrasound probe. The imaging data generated by the imaging probe may be any one of two-dimensional imaging data, stereoscopic three-dimensional imaging data, or bi-plane imaging data. In some examples, the processor may also be configured to identify and determine the posture of the target blood vessel under the patient's skin based on the imaging data. In some examples, the device may be suitable for obtaining imaging data of the target position from multiple angles. The processor may also be configured to construct three-dimensional imaging data based on the received imaging data obtained from multiple angles, determine the three-dimensional posture of the target position under the patient's skin based on the imaging data obtained from multiple angles, or both construct three-dimensional imaging data based on the received imaging data obtained from multiple angles, and determine the three-dimensional posture of the target position under the patient's skin based on the imaging data obtained from multiple angles.

[0016] In some examples, the imaging data may include video data of the target position obtained at different times, and the processor may be configured to determine a trajectory for steering the cannula toward the target pose based on a comparison between previous and subsequent frames of the video data.

[0017] In some examples, the positioning unit can be configured to manipulate the cannula in at least two degrees of freedom.

[0018] In some examples, the processor can be configured to identify that the determined trajectory has become misaligned with the target position based on device posture data received from at least one device position sensor, such as a camera, an electromagnetic sensor, a gyroscope, an accelerometer, or any combination thereof.

[0019] In some examples, the device may include a force sensor configured to generate force data indicating a measured amount of force applied to the cannula. The processor may be configured to receive force data from the force sensor, update the cannula posture data to a corrected position and orientation based on the received force data, and instruct the cannula positioning unit to manipulate the cannula according to the corrected position and orientation. Additionally or alternatively, the processor may be configured to receive force data from the force sensor, obtain a force distribution from the force data, and classify the obtained force. Examples of force classification include successful cannulation events, vascular rolling events, undershoot events, and overshoot events. The processor may be able to classify the force data into one or more of these categories. In addition, in some examples, in the absence of a successful cannulation event, the processor may be configured to instruct the positioning unit to manipulate the cannula in response to the classified event.

[0020] In some examples, the device may further include a blood flow sensor configured to generate blood flow data indicating blood flow into the cannula. The processor may be configured to receive the blood flow data from the blood flow sensor and further instruct the cannula positioning unit to manipulate the cannula based on the blood flow data.

[0021] In some examples, each motor may be associated with a corresponding motor sensor. The motor sensor may be configured to generate cannula posture data indicative of the position and orientation of the cannula based on the movement of the corresponding motor.

[0022] Another aspect of the present disclosure relates to a method for guiding a cannula toward a target position under the skin of a patient. The method may be performed by a processor coupled to an imaging probe for continuously generating imaging data of the target position, and a positioning unit including a plurality of motors configured to manipulate a cannula attached to the positioning unit, each motor being associated with at least one cannula position sensor, at least one cannula position sensor configured to generate cannula posture data indicating the position and orientation of the cannula based on the movement of the corresponding motor. The positioning unit may also include at least one device position sensor, at least one device position sensor configured to generate device posture data indicating the orientation and position of the device. The method may include: receiving imaging data from the imaging probe; receiving cannula posture data from at least one cannula position sensor; receiving device posture data from at least one device position sensor; identifying the target position through the imaging data; determining a trajectory for manipulating the cannula toward the target posture based on the imaging data, the cannula posture data, and the device posture data; instructing the positioning unit to manipulate the cannula according to the determined trajectory; identifying that the determined trajectory becomes misaligned with the target position; updating the determined trajectory to a corrected trajectory based on the imaging data, the cannula posture data, and the device posture data; and instructing the positioning unit to manipulate the cannula according to the corrected trajectory.

[0023] In some examples, the method may also include identifying and determining a posture of a target blood vessel under the patient's skin based on the imaging data, and instructing the positioning unit to manipulate the cannula based on the determined posture of the target blood vessel.

[0024] In some examples, receiving imaging data from the probe may include receiving imaging data from multiple angles, and the method may also include constructing three-dimensional imaging data from the received imaging data obtained from the multiple angles. In some examples, the method may include determining a posture of a target location under the patient's skin based on the imaging data obtained from the multiple angles.

[0025] In some examples, instructing the positioning unit to manipulate the cannula may include providing instructions for manipulating each of two or more degrees of freedom of the positioning unit.In some examples, it may be identified that the determined trajectory has become misaligned with the target position based on device pose data received from a device position sensor.

[0026] In some examples, determining the cannula posture data can be based on the imaging data using at least one of a computer vision algorithm or an electromagnetic sensor.

[0027] In some examples, the processor may be coupled to a force sensor for generating force data indicating a measured amount of force applied to the cannula. The method may also include receiving force data from the force sensor, updating the cannula positioning data to a corrected position and orientation based on the received force data, and instructing the cannula positioning unit to manipulate the cannula according to the corrected position and orientation. Additionally or alternatively, the method may also include obtaining a force distribution through the force data, and classifying the obtained force distribution (e.g., a successful cannulation event, a vessel rolling event, an undershoot event, an overshoot event). Furthermore, in some cases, in the absence of a successful cannulation event, the method may include instructing the positioning unit to manipulate the cannula in response to the classified event.

[0028] Another aspect of the present disclosure relates to a device for inserting a needle into a patient's blood vessel. The device may include an ultrasound probe configured to image a portion of a blood vessel within a field of view of an ultrasound probe, a mechanical arm attached to the ultrasound probe, and a processor for controlling the operation of one or more motors included in the mechanical arm. The needle may be attachable to the mechanical arm. The mechanical arm may include one or more motors configured to manipulate the position and orientation of the needle. The processor may be configured to define a reference system for controlling one or more motors based on spatial parameters of the blood vessel. The mechanical arm may be adapted to enable the needle to be inserted at an angle into a portion of the blood vessel positioned within the field of view of the ultrasound probe. The processor that controls the operation of the one or more motors may be automated at least in part based on image data from the ultrasound probe, and the device may be configured to be handheld.

[0029] In some examples, the motor of the robotic arm can be configured to manipulate at least the insertion depth of the needle and the insertion angle of the needle. The processor can be communicatively coupled to the probe and can be configured to determine the insertion depth based at least in part on data received from the probe. The insertion angle can be determined based on the insertion depth.

[0030] In some examples, the device may also include an accelerometer and a gyroscope configured to provide data to the processor indicating motion and orientation of the device, respectively. The processor may control the operation of one or more motors and may be automated based in part on data from the accelerometer and gyroscope.

[0031] Another aspect of the present disclosure relates to a device for guiding a cannula toward a target location under the skin of a patient. The device may include: at least one motor, at least one motor configured to manipulate the cannula toward the target along the longitudinal direction of the cannula; a force sensor, the force sensor configured to generate force data indicating a measured amount of force applied to the cannula; a blood flow sensor, the blood flow sensor configured to generate blood flow data indicating blood flowing into the cannula; and a processor, the processor configured to receive force data from the force sensor, receive blood flow data from the blood flow sensor, determine the occurrence of a successful cannulation event based on a combination of the force data and the blood flow data, and instruct at least one motor to stop manipulating the cannula toward the target when a successful cannulation event occurs.

[0032] In some examples, the device may further include a cannula position sensor configured to generate cannula posture data indicating the position of the cannula based on the movement of the at least one motor. The processor may be configured to determine a successful cannulation event based on a combination of the force data, the blood flow data, and the cannula posture data.

[0033] In some examples, the device may also include an imaging probe for obtaining ultrasound imaging data. The processor may be configured to determine the likelihood of a failed intubation event based on a combination of the force data, the blood flow data, the ultrasound imaging data, and the intubation posture data, and to determine an adjustment to the intubation to reduce the likelihood of a failed intubation event based on the combination of the force data, the blood flow data, the ultrasound imaging data, and the intubation posture data.

[0034] In some examples, the processor can be configured to obtain a force distribution from the received force data, classify the obtained force distribution as one of a successful intubation event or an unsuccessful intubation event, and instruct one or more motors to steer the cannula in the opposite direction when an unsuccessful intubation event occurs. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is an automatic intubation device in the prior art.

[0036] Figure 2 is another view of a prior art intubation device in operation.

[0037] Figure 3 yes Figure 1 An example positioning unit of a prior art intubation device.

[0038] Figure 4 is a block diagram of an intubation device according to one aspect of the present disclosure.

[0039] Figure 5 is a diagram of a blood vessel and a coordinate reference system according to an aspect of the present disclosure.

[0040] FIG. 6A to FIG. 6D is a graphical representation of force measurements over time according to an aspect of the present disclosure.

[0041] Figure 7 is a flow chart of a procedure according to one aspect of the present disclosure.

[0042] Figure 8 is an exploded perspective view of an electromagnetic attachment mechanism according to one aspect of the present disclosure.

[0043] 9A to 9C are flow diagrams of additional procedures according to aspects of the present disclosure.

[0044] Fig. 10A is a perspective view of an intubation device according to an aspect of the present disclosure.

[0045] Fig. 10B yes Fig. 10A Exploded view of the intubation device.

[0046] FIG. 10C to FIG. 10I yes Fig. 10A A stereoscopic view of the various parts of the intubation device.

[0047] Fig.11A is a perspective view of another intubation device according to an aspect of the present disclosure.

[0048] FIG. 11B to FIG. 11G yes Fig.11A A stereoscopic view of the various parts of the intubation device.

[0049] Fig.12 is a block diagram of a workflow for operating an intubation device according to one aspect of the present disclosure. DETAILED DESCRIPTION

[0050] Figure 4 4 is a block diagram showing a control scheme for a handheld intubation device 400 according to the present disclosure. The intubation device 400 includes a main processor 410, a microcontroller 420, a display 430, and each of a plurality of motor controllers 440a, 440b for controlling the various degrees of freedom axes of the intubation instrument 450. The main processor 410 can be any processing unit known in the art with sufficient processing power to receive processing and output image data at video frame rates. The microcontroller 420 can be an integrated chip or other processing unit known in the art with sufficient processing power to control an automated robotic arm, including processing and relaying of sensor inputs and instructions for controlling motor activity.

[0051] The microcontroller 420 can be configured to receive one or a combination of analog inputs and digital inputs from various sensors, to process those inputs, and to relay the inputs to the main processor via a controller area network (CAN) connection. In some cases, the main processor 410 can include a different connection, such as a universal serial bus (USB) connection, in which case the sensor data can be relayed via the CAN to the USB connection. Conversely, instructions relayed from the main processor 410 through the microcontroller 420 can be provided via a USB to CAN connection.

[0052] The display 430 can be configured to provide a two-dimensional or three-dimensional image of the cannulation target location and the cannulation instrument. The cannulation trajectory can be superimposed on the image, such as the expected trajectory of the cannulation instrument to reach the target vessel. The image can be updated or refreshed in real time or at a video frame rate. The display can also be configured to allow the clinician to specify a location in the image presented on the display, such as a specified target location.

[0053] The display 430 may also be configured to initialize an image processing routine. For example, automatic partitioning of the boundaries of a vessel may be initiated by interaction between a user and the device 400 via the display 430. For example, the ultrasound transducer may be moved around until the target vessel is centered in the ultrasound image (the display may include a crosshair to indicate the center of the image). Once centered, the processing software in the device may be manually initiated to search for partitions starting from the center of the image. After the initial partitioning, the target may be automatically tracked thereafter.

[0054] A similar approach to starting the automatic partitioning algorithm can be used when the probe is oriented longitudinally. In this case, the vessel can be aligned so that the vessel is parallel to the longitudinal image. The partitioning algorithm can then identify the vessel from the longitudinal image and the identified vessel can be automatically tracked.

[0055] The center of the blood vessel can be detected manually or automatically based on the ultrasound image. In the case of automatic detection, the image partitioning algorithm can be applied to both the longitudinal orientation and the transverse orientation of the ultrasound probe. In the transverse orientation of the grayscale B-mode ultrasound image showing the cross-section of the blood vessel, the algorithm can detect elliptical structures in the image, and can initialize the rough partition outlines of the potential blood vessel structure based on the detected elliptical structures. These initial outlines can be improved in an iterative manner. In the longitudinal orientation of the B-mode ultrasound image parallel to the blood vessel, the algorithm can alternatively detect elongated tubular structures with an expected diameter in the range of 1 mm to 5 mm, and a similar process of outline initialization and improvement can be performed on the detected tubular structure.

[0056] Additionally, the B-mode image partitioning algorithm can be supplemented with image information extracted through the color Doppler imaging mode. In the color Doppler mode, the velocity is detected by the probe (rather than tissue backscatter). Since blood has directional motion and velocity, while the surrounding tissue is mostly static, blood flow can be directly observed through color Doppler imaging. The color Doppler image data can be combined with the B-mode intensity image data in several different ways (masking, contour initialization, weighted summation, etc.) to improve the accuracy of the partitioning algorithm.

[0057] Once the center coordinates of a vessel are identified, an image-based tracking algorithm can be used to track the vessel from one frame to the next. This allows the device to detect tissue motion in real time and adjust the instrument posture accordingly. Tracking vessels in this way also reduces the need to perform vessel segmentation from scratch in each frame, thereby speeding up computation time.

[0058] The intubation device 400 also includes a plurality of sensors for providing input to the main processor 410 and the microcontroller 420. The sensors may include an imaging probe 462 such as an ultrasound probe, a force sensor 464 for sensing the force applied to the motor as it drives the intubation instrument, a respective position encoder 466a, 466b for each motor controller 440a, 440b to monitor the manipulation of the needle by the device's motor, and a position sensor 468 for tracking changes in translation and orientation of the intubation device 400. Since the intubation device is handheld, the position sensor 468 is necessary to track whether the intubation device is moved from its initial position intentionally or unintentionally.

[0059] The main processor 410 and the microprocessor 420 are programmed with software instructions for performing the operations and procedures outlined in the present disclosure. For example, the microprocessor can be programmed to receive analog inputs from each of the force sensor, the position encoder, and the position sensor, process the received input into a format acceptable to the main processor, and then transmit the reformatted input data to the main processor for further processing. For example, the main processor can be programmed to receive and process imaging data, such as identifying a target location in the patient's body and determining a trajectory for guiding the intubation instrument to the target location. The main processor can also determine and continuously track the relative position of the intubation instrument relative to the target location based on the received input, and can update the determined trajectory based on changes in the relative position.

[0060] The processor can calculate relative positions during intubation and display these to the operator. For example, the processor can automatically infer the final position of the intubation instrument tip (e.g., the position where the tip will be when inserted) based on the current posture of the instrument when the instrument is manipulated / aligned. The offset distance between the inferred tip position and the target position can also be calculated. Similarly, the expected trajectory of the instrument (e.g., the route that the instrument tip will take during insertion) can be continuously inferred from the current posture of the instrument and can be displayed. The inferred position in the ultrasound image can be continuously updated, and the updated position is displayed. In some cases, since the update is continuously calculated, the ultrasound image and the inferred tip position can be further updated when the insertion is performed. The processor can also update the coordinates of the vascular target, for example to compensate for the movement from the tissue, the patient, and the ultrasound probe. The updated position of the vascular target will then be guided to the instrument positioning unit to adjust the posture and trajectory of the instrument tip, minimize the offset distance between the instrument tip and the vascular target, and ensure that it is always aligned with the vascular target. In another example, when the inferred tip position is accurately aligned with the blood vessel or other target positions in the 3D space, a prompt can be provided to the user. The prompt will indicate that intubation can be performed. Depending on the clinical situation, it is conceivable that intubation is performed only when the presumed tip position is aligned with the target position within a preset error tolerance. This mechanism would serve as a safety feature to prevent erroneous, inaccurate or unsafe intubation attempts.

[0061] The imaging probe 462 can be an ultrasound (US) imaging unit capable of detecting target blood vessels or tissues under the surface of the patient's skin. The US imaging unit can be capable of collecting US images in real time or at a video frame rate (e.g., 30 frames or 60 frames per second, or faster). For some imaging units, the image can be two-dimensional. For other units, the image can be biplane. In other units, the image can be three-dimensional. Example applications of these imaging types are provided in more detail below. Preferably, the frequency of the ultrasound probe is in the range of about 6 MHz to about 18 MHz. However, if it is determined that other frequencies produce higher image quality, these other frequencies can be used. For example, deeper tissue structures may require lower ultrasound frequencies, while shallower structures may require higher frequencies.

[0062] In some examples, the US imaging unit can additionally acquire any one or combination of color Doppler, power Doppler, spectral Doppler, tissue Doppler, vector Doppler, or other ultrasound-based imaging modes. These imaging techniques can be used to detect the presence and direction of blood flow within a vessel so that a suitable vessel can be selected for puncture. Additionally or alternatively, the imaging probe 462 can be paired with other auxiliary imaging units, such as visible light imaging (e.g., a camera); infrared (IR) or near infrared (NIR) imagers; multispectral imagers; structured light or depth imaging, optical tomography; laser Doppler, etc. Other imaging techniques can be used, such as photoacoustic or photoacoustic imaging techniques.

[0063] The force sensor 464 can be a material having a property that changes in response to an applied force, such as a piezoelectric material or a force sensitive resistor. The material can be embedded in an intubation device that contacts the intubation instrument. In an operational example, when the intubation instrument moves along a linear degree of freedom to advance the intubation, such as by sliding along a ball bearing slider, both normal force and friction force may act on the intubation instrument, causing the intubation to push against the force sensor 464 material. This can generate an analog signal, which is then digitized by an analog-to-digital converter such as a microprocessor 420 and relayed to the main processor 410. The processor 410 can use the received signal data to monitor the amount of force applied to the material.

[0064] Additionally or alternatively, the force sensor can be a current sensor configured to measure the current flowing through the windings of the cannulation device motor. The current can increase in proportion to the load on the motor, which means that the measured value of the current can indicate, for example, the increased force applied to the cannulation device during puncturing the target vessel wall. Again, the current measurement value can be an analog measurement value that is converted to a digital signal and relayed to the processor 410 for further processing. For example, the processor can record a series of current measurements over a period of time and then filter (e.g., by a first-order low-pass digital filter) to generate an indication of the force on the cannulation device. Taking the needle insertion force as an example, the measured force can be the sum of multiple forces, including but not limited to dynamic friction and viscous friction, elastic deformation of soft tissue, and incision through tissue.

[0065] Different types of position encoders 466a, 466b may be employed depending on the type of degree of freedom controlled by each encoder's corresponding motor. For example, if the motor controls a rotational degree of freedom of a positioning unit, the associated encoder may be a rotary encoder or a potentiometer. For further example, if the motor controls a linear degree of freedom of a positioning unit, the associated encoder may be a linear encoder for interpreting linear translation. Relative or absolute encoders may be used.

[0066] Position sensor 468 may include one or both of a gyroscope and an accelerometer to track the movement and tilt of the intubation device. In one example, the gyroscope may be responsible for tracking the direction of the device, while the accelerometer is responsible for tracking the overall movement of the device. Other combinations of sensors, such as multiple gyroscopes or multiple accelerometers, may be used to generate similar results. In some cases, an inertial measurement unit may be used to track motion. In other cases, a stereo camera may be used to improve tracking. Additionally or alternatively, an electromagnetic tracking device may be used to monitor the movement of the device. Additionally or alternatively, image data (e.g., camera image frames, ultrasound image frames) may be used to track the device as it translates or tilts. In this case, a computer vision program may be used to process the image so that changes in the image frame may indicate a shift or tilt in the device.

[0067] Position sensor 468 can be used for a variety of purposes. One example use of a motion sensor is to form a 3D image from 2D probe imaging data. If the cannulation device includes a 2D imaging probe, such as a two-dimensional ultrasound imaging unit, the device can be swept around the target location of the patient. In this way, images of the target location can be acquired from multiple angles. The images can be provided to a main processor and then effectively stitched together to generate a composite 3D image. For reasons further described below, it is advantageous to use 3D images to track the target vessel compared to 2D images.

[0068] Another example use of position sensor 468 is to provide a gravity vector. The gravity vector can be overlaid with the probe imaging data to help interpret the imaging results. In some cases, this may help identify the patient's anatomical structure.

[0069] The placement of the position sensor 468 in a specific area of ​​the cannulation device 400 does not limit the ability of the sensor to track the motion of components in other areas of the device. For example, if the position sensor is a gyroscope and accelerometer mounted to an imaging probe, it can be used to directly measure the attitude and acceleration of the probe. The directly monitored data can then be processed by the main processor to indirectly calculate the changes in the attitude and displacement of the cannulation device based on the changes in the rotational attitude and displacement of the probe in addition to the known kinematic attitude of the cannulation device relative to the probe. In a similar vein, the gyroscope and accelerometer mounted to the cannulation device can be used to indirectly calculate the attitude and displacement of the imaging probe.

[0070] In other examples, each of the probe and cannula instrument can include its own position sensor. In addition, the measurements from the position sensors can be compared to each other to check consistency, provide a higher degree of accuracy for the measurement, or both check consistency and provide a higher degree of accuracy for the measurement.

[0071] Because the intubation device 400 is handheld and includes an imaging probe, it should be understood that the image data received at the host processor is not acquired from a fixed position or angle. Therefore, an absolute world coordinate system cannot be used to characterize the objects and positions identified in the image data. Instead, a dynamic reference coordinate system must be used to associate the image with the relative position of the intubation instrument. Since the intubation instrument and the imaging probe are connected to each other, moving the intubation device will cause each of the intubation instrument and the imaging probe to move in a predictable coordinated manner.

[0072] One challenge that arises from using a handheld intubation device is the need to define the pose of the intubation instrument relative to the device and the imaging probe. Since the intubation device is handheld and easily moved or tilted, a fixed coordinate system cannot be used. Instead, the pose of the intubation instrument can be based on one or more sensors that can track the movement of the intubation instrument relative to the device.

[0073] In some cases, the intubation instrument posture data can be derived by processing data received from an electromagnetic tracker used to track the movement of the intubation instrument. Additionally or alternatively, image data (e.g., camera image frames, ultrasound image frames) can be used to track the intubation instrument posture as the intubation instrument advances toward the target position. In this case, a computer vision program can be used to process the images. Additionally or alternatively, the intubation instrument posture data can be derived from position encoders in each motor. Each joint of the kinematic chain of the robotic arm (e.g., Figure 3 The degree of freedom chain shown in ) can be defined by individual posture transformations.

[0074] The pose of the intubation instrument can be calculated based on the encoder measurements of each degree of freedom or joint and represented by a rotation and translation offset relative to the previous degree of freedom or joint. The representation can be, for example, in the form of a vector describing the rotation and a translation vector describing the offset. Alternatively, the rotation can be described using an Euler vector representation. Otherwise, the rotation and translation representations can be combined into a single homogeneous transformation matrix. Regardless of which representation is used, the relative pose of each degree of freedom or joint can be combined in series to obtain the current pose of the intubation instrument between the origin and the tip.

[0075] For example, in the case of a 4-DOF system, a quaternion vector or 4x4 matrix containing position encoder measurements for each degree of freedom may be used to represent the current intubation instrument pose. As the motor in the robotic arm manipulates the intubation instrument, updated encoder information may be provided to one or both of the main processor and the microcontroller, and the transformation chain may be updated based on the updated encoder information.

[0076] As described above, even when the motor causes the intubation instrument to move, the relative coordinate systems of the imaging unit and the robotic arm remain fixed relative to each other. In this regard, the sequence of transformations T from the device through the N joints (joint 1 to joint N) in the positioning unit can be characterized in the following manner:

[0077] (1)T 设备至器械 =T 接头N至器械 *T 接头N-1至接头N …*T 接头1至接头2 *T 设备至接头1

[0078] In this way, a reference frame can be defined based on the posture and orientation of the device, or in particular the main parts of the device. This reference frame (device) can then be used to relate other parts of the system to each other. For example, an imaging probe and an intubation instrument can be related to each other in the following way:

[0079] (2)T 探头至器械 =T 设备至器械 *T 探头至设备

[0080] Encoder-based attitude measurements may be affected by some residual errors. Residual errors are usually caused by mechanical inaccuracies, motor backlash, calibration errors, etc. Therefore, in order to reduce the residual errors of these measurements, additional sensors can be combined with the intubation device. For example, a position sensor (such as Figure 4 A position sensor) can be used to correct for residual errors. In an example arrangement, the position sensor can include a gyroscope attached to an automated robotic arm or imaging probe to estimate the absolute 3D pose of the intubation device, and one or both of an absolute position / orientation sensor attached to the vicinity of the distal joint of the intubation instrument to estimate the absolute 3D pose of the intubation instrument. Combining the outputs from these two position sensors (e.g., at a main processor) can produce an estimate of the relative instrument pose that is not affected by mechanical errors. A transformation for obtaining the 3D pose of the intubation instrument relative to the surrounding environment (such as the patient, the patient's skin, and the target location, collectively referred to as the "domain") can be obtained based on gyroscope measurements and can be characterized in the following manner:

[0081] (3)T 设备至器械 =inv(T 器械至域 )*T 设备至域

[0082] Another challenge that arises from using a handheld device is the need to represent the 3D pose of the target vessel relative to the coordinate system of the cannulation device, as opposed to representing the vessel pose based on a fixed coordinate system. This way, if the cannulation device moves, the movement can be easily tracked relative to the target vessel. Additionally, since the cannulation instrument is tracked relative to the cannulation instrument, manipulation of the cannulation instrument can also be easily tracked relative to the target vessel.

[0083] This can be accomplished by relying entirely or primarily on image data from the imaging probe. There are a variety of ways to make this representation, depending on the specific imaging data available. For clarity, reference is made to Figure 5 , which shows specific 3D coordinate axes that can be used to represent the posture of the blood vessel.

[0084] In a first example, a single two-dimensional (2D) ultrasound image can be obtained in a transverse orientation. From this image, the center of the vessel can be seen through a single cross section of the vessel, including the depth of the vessel center from the patient's skin surface. These correspond to Figure 5 X in v and Z v To confirm that the target location is suitable for puncture, the diameter of the blood vessel (d v ). If color Doppler is used, blood flow velocity can be quantified. The disadvantage of 2D images is that the vascular direction parameters cannot be determined ( Figure 5 θ v and ). Therefore, it is assumed that the blood vessels are horizontal, i.e., Figure 5 Based on this assumption, a preset insertion angle is provided for the intubation instrument. (e.g., 25 degrees) and insert the instrument at a preset angle along a linear trajectory. If the ultrasound probe is not aligned over the center of the vessel, or if the ultrasound probe is misaligned with the center of the vessel, it can be realigned by translating in the lateral (Y) direction.

[0085] In a second example, a single 2D ultrasound image can be obtained in the longitudinal orientation. In this case, the vessel depth (Z v ) and vascular orientation parameters All are visible in the ultrasound image. It can also be visualized whether the vessel is straight or curved. The vessel target location X can be selected anywhere within the field of view of the 2D ultrasound image. v The vascular target location can be determined by an automated process, such as based on computer analysis of the 2D images. Alternatively, the vascular target location can be selected manually. As with the first example, color Doppler can be used to confirm whether there is adequate blood flow through the vessel. A disadvantage of this example is that it is not possible to confirm whether the 2D imaging plane cuts exactly through the center of the vessel. In other words, Figure 5 The parameter Y in v The center of the vessel must be assumed, but this cannot be confirmed from a single image segment. This drawback can be overcome by human supervision of the interpolation process.

[0086] In a third example, multiple 2D images may be acquired in a common orientation, such as a transverse orientation. This may be accomplished by a manual "sweep" of a handheld imaging probe. The imaging probe may be used to "sweep" along the X-axis while holding the probe in the transverse orientation to determine the vessel orientation in the longitudinal direction. The sweep may be relatively short, such as only about 10 mm. Each 2D image acquired during the scan is then processed and stacked, and a 3D image may be constructed from the 2D segments. Vessel pose parameters, including θ, may then be calculated from the constructed 3D image. v and In this regard, the insertion angle of the vascular cannula may be varied based on the 3D image.

[0087] In a fourth example, multiple 2D images may be acquired in a longitudinal orientation. As in the third example, this may be accomplished using a "sweep" of the probe. The sweep may be relatively short, such as only about 5 mm or less, in order to accurately identify the target vessel (Y) by v ) to construct a 3D image.

[0088] In a fifth example, as opposed to relying on 2D images, a "biplane" ultrasound image can be obtained. This requires the use of a biplane probe capable of simultaneously acquiring transverse and longitudinal images. The simultaneously collected images can then be processed together to determine the necessary parameters of the target vessel, including vessel depth and orientation, as well as identifying the center of the vessel. A trajectory can then be determined to allow the tip of the cannulation instrument to be positioned directly over the center of the vessel and aligned with its axis during insertion and puncture. In this example, a single biplane image may be sufficient, but acquiring multiple biplane images can improve the overall accuracy of the calculations.

[0089] In a sixth example, a 3D volume ultrasound image may be obtained. In this case, the 3D image provides complete information about the center, direction and curvature of the vessel, through which the optimal trajectory can be determined.

[0090] If a 3D volumetric image is acquired, one or more 2D or bi-plane segments may be extracted from the volume in any arbitrary orientation for display. For example, an oblique segment containing the center of a vessel in a longitudinal orientation may be determined from the 3D volume and displayed as a single 2D image frame. As the probe moves, the pose of the oblique segment may be updated so that the vessel is always displayed in the resulting 2D image, as long as the vessel is somewhere within the 3D field of view of the imaging probe. Similarly, the oblique segment may be updated to always display the instrument axis, as long as that axis is somewhere within the 3D imaging volume. As an extension, multiple segments may be displayed as separate images; for example, an image segment containing a vessel and another segment containing an instrument may be displayed separately.

[0091] In addition to the above examples, auxiliary imaging techniques and devices can be combined to provide any missing information. For example, camera-based imaging can be used. This may be particularly useful when ultrasound imaging cannot provide all necessary 3D posture parameters (such as the case of longitudinal and transverse 2D ultrasound).

[0092] For example, camera-based imaging can be used to determine the orientation of the imaging probe, the cannulation device, or both relative to the patient's skin surface. One or more cameras can be mounted to the cannulation device, for example, on a side wall of the imaging probe, and pointed in the direction of the imaging probe so as to track skin surface features as the device moves over the patient's skin. With appropriate camera calibration, the camera image can be used to determine the 3D coordinates of the skin surface relative to the probe. However, since the target vessel is located below the skin, it may still be preferred to determine the posture of the target vessel relative to the patient's skin surface based on the imaging probe data.

[0093] Since the camera moves with the intubation device, the camera's coordinates must also be transferred to properly represent the movement of the patient or the device. The transformation used to obtain the 3D pose of the patient's skin based on the camera image data can be represented in the following way:

[0094] (4)T 皮肤至设备 =T 摄像头至设备 *T 皮肤至摄像头

[0095] Camera-based imaging may additionally or alternatively be used to track changes in the orientation of the intubation instrument, provided that the intubation instrument is within the field of view of the camera. The 3D pose of the intubation instrument based on camera image data may be characterized in the following manner:

[0096] (5)T 器械至设备 =T 摄像头至设备 *T 器械至摄像头

[0097] Camera-based imaging can additionally or alternatively be used to visualize blood vessels close to the surface of the patient's skin. Certain wavelengths of light, particularly near-infrared wavelengths (approximately 700 nm to 1000 nm), are strongly absorbed by hemoglobin in the blood to a depth of 4 mm below the skin surface. This results in a high-contrast image of the blood vessels from a top-down view (as seen from the skin surface). The 3D pose of a target vessel based on camera image data can be characterized as follows:

[0098] (6)T 血管至设备 =T 摄像头至设备 *T 皮肤至摄像头 *T 血管至皮肤

[0099] Camera-based imaging can also be used to confirm blood flow in the cannula at the time of puncture: Confirmation of successful puncture has previously been accomplished by visualizing blood flow in the hub of the cannula. However, this is only possible if the cannula hub is visible. Camera imaging, together with display of the camera image on a monitor, provides another way to place the cannula hub within the field of view, provided of course that the hub is within the field of view of the camera.

[0100] Camera-derived parameters can add redundancy to parameters derived using imaging probes such as ultrasound probes. In some cases, camera-derived parameters can be used to supplement missing information not obtained from imaging probe data. However, camera resolution is typically an order of magnitude lower than ultrasound resolution; therefore, ultrasound preferably remains the primary imaging modality for this technology.

[0101] Putting the above imaging data and coordinate transformations together, the 3D vessel pose relative to the cannulation instrument pose can finally be characterized as follows:

[0102] (7)T 血管至器械 =T 主设备至器械 *T 探头至主设备 *T 血管至探头

[0103] This relationship between vessel pose and instrument pose is not dependent on a particular fixed reference frame and therefore can easily adapt when the cannulation device moves above the target location or when the target location moves below the cannulation device.

[0104] In some cases, it may be beneficial to first identify a suitable vessel or characterize its 3D pose before beginning the cannulation process. For example, an initial imaging process may be used to first decide whether cannulation is required, determine the general location of puncture, or determine the health of the vessel, whether cannulation is ready, or both. In this case, it may be advantageous to perform the imaging scan without securing the cannulation instrument to the imaging device, for example to minimize the volume of the device during imaging. Thus, the imaging unit and the instrument positioning unit may include a mechanism that allows the cannulation instrument to be attached and detached from the main handheld device. However, when all units are attached to the main device, the coordinate transformation will remain the same as described above.

[0105] Another challenge that arises from using a handheld device is the need to determine the desired trajectory of the cannulation instrument without relying on a fixed coordinate system. Trajectory determination can be obtained from determining the depth of the vessel below the patient's skin surface in contact with the imaging probe ( Figure 5 Z v ). The orientation of the vessel at the cannulation point can also be determined ( Figure 5 Coordinates in ). The length of the cannulation device may be known, either as a preset value or as a manually entered value. For some activities, such as drawing blood or cannulating a vein, the cannulation device may then be set relative to the longitudinal axis of the vessel based on the determined vessel depth. For example, vessels with a depth of less than 4 mm below the skin surface may result in a default insertion angle of approximately 15 degrees, vessels with a depth between 4 mm and 8 mm may result in a default insertion angle of approximately 20 degrees, and vessels with a depth greater than 8 mm may result in a default insertion angle of approximately 30 degrees. These values ​​are based on recommendations in clinical practice but may be further adjusted based on the length of the cannulated device.

[0106] In some cases, the intubation instrument can also be moved about a vertical axis (θ v ) Rotational alignment. This may be helpful if the vessel is not completely parallel to the longitudinal x-axis. By routing the cannulation instrument, the cannulation instrument can be rotated into alignment with the vessel.

[0107] In other cases, the main processor may be configured to consider additional factors (eg, age of the patient, anatomical location of the target vessel) in order to determine the trajectory to the target vessel. These factors may be manually input.

[0108] Once the desired trajectory is established and defined within the coordinate reference system of the cannula device, cannula insertion can be started. Throughout the insertion process, the target blood vessel and its center can be tracked from one frame to the next using an image tracking algorithm based on the image data continuously collected from the imaging probe. "Continuous" data can refer to data received at a frame rate suitable for video capture. Alternatively, as described below, "continuous" can refer to acquiring data at a faster rate than the iterative instructions provided to the positioning unit of the cannula device. As described above, this "continuous" data capture allows real-time detection of the movement of the patient and the cannula device, and real-time adjustment of the posture of the cannula device accordingly. Compared with performing the original determination from scratch for each frame, tracking blood vessels based on frame-to-frame changes is preferred because it reduces the amount of calculation required by the main processor. Other forms of imaging data, such as camera imaging data, can also be used to track movement. Any one or a combination of cannula devices, cannula devices, patient skin, and target blood vessels can be tracked.

[0109] The instrument positioning unit can be repeatedly adjusted until the intubation instrument is properly aligned with the target location. The intubation instrument can be iteratively adjusted by actuators, each actuator controlling a separate degree of freedom (DOF) in the positioning unit. The target location, such as the center of a target vessel, can be tracked through each iteration of the image processing program. As the target location moves, the target location coordinates can be updated according to the above-mentioned coordinate reference system. The trajectory and posture of the intubation instrument can then be adjusted to take into account the updated target location coordinates. Such adjustments can also be included in the iterations of the coordinate updates. In this way, the instrument positioning unit can operate in a closed-loop manner based on feedback based on continuous images to align the instrument with the tissue target.

[0110] For example, if the device is being manually moved along the patient's forearm, this movement may result in a change in the depth of the target location. This may result in an adjustment to the end point of the cannulation instrument. Likewise, if the cannulation instrument is in the middle of an insertion process and horizontal rolling or translation of the target vessel is detected, the positioning unit may adjust the pose of the cannulation instrument to account for the new coordinates of the target vessel.

[0111] Depending on the DOF available in the positioning unit and the robotic arm, there are multiple ways to adjust the intubation instrument trajectory. Again, for clarity, refer to Figure 5 , which shows specific 3D coordinate axes useful for defining changes in the 3D pose of the intubation instrument.

[0112] In a first example, the positioning unit has a control insertion distance (d n ) (e.g., the cannulation instrument moves forward and backward). In this configuration, assuming that the trajectory of the cannulation instrument passes through some parts of the blood vessel, DOF1 can be iteratively adjusted until the instrument punctures the blood vessel at an intersection along the trajectory. If a specific part of the blood vessel is to be cannulated, the trajectory can be changed by reorienting the positioning unit based on the US image.

[0113] In the second example, the positioning unit has 2 degrees of freedom: (DOF1 is the same; DOF2 controls the insertion depth (Z n ) (e.g., the depth of the tip of the cannula instrument under the skin when the tip of the cannula instrument reaches the target location). In this configuration, two DOFs can be adjusted iteratively. Thus, if the trajectory of the cannula instrument tip becomes misaligned with the target, for example due to tissue or probe movement, the height of the positioning unit can be adjusted during the insertion process.

[0114] In a third example, the positioning unit has 3 degrees of freedom: (DOF1 and DOF2 are the same; DOF3 controls the off-axis position of the intubation instrument (Y n) (e.g., side-to-side movement of the cannula instrument). In this configuration, the positioning device can align the tip of the cannula instrument with the center of the blood vessel in the lateral direction (Y v , Z v ) automatic alignment. In this regard, the positioning device has the ability to compensate for off-axis (Y, Z) motion, which may be caused by probe motion, tissue motion, or vessel rolling during puncture. This example configuration may be beneficial when imaging in a lateral probe orientation so that off-axis motion can be automatically detected from the probe image and then iteratively corrected. Therefore, these lateral or off-axis adjustments can be used to correct for lateral movement of the user or patient.

[0115] In the fourth example, the positioning unit has 4 degrees of freedom: (DOF1, DOF2 and DOF3 are the same; DOF4 controls the insertion angle of the intubation instrument (e.g., the angle at which the cannula instrument is tilted downward relative to the patient's skin surface). In this configuration, the positioning unit can determine the depth of the target blood vessel (Z v ),orientation or both to position the cannulation instrument by independently controlling the insertion angle. The configuration also enables precise cannulation along a non-linear, curved trajectory. For example, the device can draw an insertion trajectory that begins puncture at a relatively steep angle (e.g., 30 degrees) and then gradually decreases the insertion angle to a relatively gentle angle (e.g., 10 degrees) before reaching the target location. A relatively steep insertion angle at the beginning may help reduce the risk of vessel rolling. A relatively gentle insertion angle at the end may help reduce the risk of puncturing the posterior portion of the vessel wall of the target vessel. Such adjustments may be particularly important when introducing the instrument into pediatric or elderly patients with small and fragile vessels, as well as high BMI patients with very deep vessels.

[0116] In the fifth example, the positioning unit has 5 degrees of freedom: (DOF1, DOF2, DOF3 and DOF4 are the same; DOF5 controls the intubation instrument around the vertical axis (θ n ). In this configuration, if the target vessel is moved so that the vessel orientation θ v is no longer parallel to the longitudinal x-axis, the cannula instrument can be rotated until the cannula instrument itself is oriented θ n and blood vessel orientation θ v Similarly, we can perform n The iterative adjustment is done to compensate for rotational motion most likely due to probe motion.

[0117] In the sixth example, the positioning unit has 6 degrees of freedom: (DOF1, DOF2, DOF3, DOF4 and DOF5 are the same; DOF6 controls the longitudinal translation of the intubation instrument (X n) (e.g., translation forward and backward along an axis orthogonal to the lateral motion and the vessel depth axis). In this configuration, the positioning device is able to center the cannulation instrument at any location along the vessel in the longitudinal direction. Once in this direction (X v ) on the target position, you can set the instrument tip (X n ) and repeatedly move it to align with the target position.

[0118] The compatibility between each DOF and the corresponding US imaging posture feature is an important aspect to ensure the usability of the device. For example, it may be advantageous to combine the above 3-DOF configuration with a lateral US probe orientation relative to the longitudinal orientation. In contrast, if Y n DOF is X n DOF instead, it may be advantageous to use a longitudinal orientation rather than a transverse orientation. Similarly, combining depth (Z n ) and orientation DOF enables non-linear insertion trajectories and is probably mainly compatible with longitudinal probes.

[0119] In some examples, the DOF of the positioning unit can include one or more remote centers of motion (RCMs) positioned around an imaging axis of the imaging unit (e.g., an imaging axis of an ultrasound transducer for an ultrasound imaging unit). In other words, the axis of the RCM can correspond to the axis of the imaging unit, such that any rotational manipulation of the cannulation instrument about the RCM will cause the inferred cannulation instrument tip position to be centered below the imaging unit (e.g., within the field of view of the imaging unit) when inserted into the target vessel. The RCM can be a supplement to the insertion depth (z-axis) DOF, such that various vessels at different depths can be targeted within a single field of view of the imaging unit without the need to move the device.

[0120] Force sensor (e.g., as combined with Figure 4 The force sensor described herein can further provide continuous feedback about the force applied to the cannulation instrument. This feedback may help confirm puncture of the target vessel wall. Ideally, puncture detected by the force sensor should be detected within T 血管至器械 ≈0, which means that the tip of the cannula instrument has just reached the coordinates of the target location.

[0121] Feedback from the force sensor can additionally or alternatively be used to control the cannula's posture and insertion trajectory prior to vessel puncture. In the time prior to vessel puncture, the force sensor data can indicate vessel motion, such as vessel rolling caused by the pressure applied by the cannula tip. Vessel rolling and other forms of vessel motion can result in displacements of less than a millimeter. However, these small displacements can be the difference between a successful puncture or a puncture of the vessel sidewall, or even the difference between completely losing the entire target vessel.

[0122] In order to correct for vascular motion, it is necessary to convert the force sensor feedback into the type of vascular motion. This can be done by providing force sensor data to a host processor or microprocessor and then comparing the force sensor data with pre-stored force data profile information to characterize the vascular motion.

[0123] Fig. 6A and Fig.6D Various force sensor graphs associated with different vascular motions are illustrated. Fig. 6A An overview of a successful puncture is shown. This puncture is characterized by a gradual increase in force on the cannulation instrument over a span of a few seconds, followed by a sudden sharp and abrupt increase in force before puncture, and then a slight decrease in force over a span of a few seconds after puncture. Considering the successful puncture profile, it can be assumed that the blood vessel did not move during the insertion and puncture process.

[0124] Figure 6B A graph showing a cannula overshooting through the back wall of a target vessel is shown. The graph is characterized by a sharp and sudden increase in force, followed by a sharp and sudden decrease in force, followed by another sharp and sudden increase in force, followed by a slow and gradual decrease in force. Given the graph of the overshoot, it can be determined that one or both of the cannula and the vessel have been displaced from their original position. The cannula instrument can then be repositioned to ensure successful puncture in the center of the vessel.

[0125] Figure 6C The graph shows a cannula resting against the anterior wall of a target vessel. The most striking feature of the graph is the sharp and sudden increase in force, followed by no decrease in force. Given the graph of the undershoot, the cannula instrument can be repositioned to ensure successful puncture in the center of the vessel.

[0126] at last, Fig.6D A graph showing a cannula sliding along the side wall of a target vessel is shown. A common cause of contact along the side wall of a vessel is rolling of the vessel caused by initial contact between the cannula and the vessel. The most notable feature of this graph is the lack of a sharp increase in force, indicating that the vessel wall has moved out of position and has not been punctured at all. Given the lack of puncture, the cannulation instrument can be repositioned to ensure successful puncture in the center of the vessel on further attempts.

[0127] In addition to force sensor feedback, other feedback sensor information can be used to characterize a successful puncture. For example, a fluid flow detector can be used to detect the presence of fluid at the hub of a cannulation instrument (e.g., blood flow from a needle inserted into a vein). A combination of force data can be combined with the detection of fluid flow to determine a successful target puncture. The combination of fluid flow data and force sensor feedback may even be sufficient to guide a person operating a handheld venipuncture device to a successful puncture, even for devices with a limited number of degrees of freedom. For example, for a device with a single degree of freedom, a combination of fluid flow data and force sensor feedback may be sufficient to indicate when the device stops advancing the cannulation instrument, or when the instrument needs to be retracted or withdrawn due to a failed attempt.

[0128] Further, for example, for devices without degrees of freedom, a combination of fluid flow data and force sensor feedback can be provided to the clinician via a display to improve the clinician's ability to successfully manually advance the instrument toward a target vessel without overshooting the vessel. Nonetheless, automated insertion of the instrument can be beneficial for a number of reasons. First, it encourages the user to reduce movement when inserting the instrument. Additionally, it allows for better control over the speed at which the instrument is inserted. Differences in insertion speed can affect the puncture force applied to the instrument, whereby higher speeds can reduce the force applied, while slower speeds can result in increased force. Controlling the insertion speed, such as ensuring a sufficiently slow insertion, can allow for more accurate force measurements to be collected.

[0129] Figure 7 is a flow chart of an example procedure 700 for performing an automated method of aligning a trajectory of an intubation device with a target location.

[0130] At box 702, an imaging probe is used to scan an area around a target blood vessel. As described above, the scan may involve collecting a single 2D image in one plane, a stack of multiple 2D images in one plane, a single 2D image in multiple planes, or other combinations of 2D images. Alternatively, in some cases, because the imaging probe is mounted to a handheld intubation device, the probe may be swept across an area around the target location, and then the imaging data may be stitched together to produce a 3D image of the target.

[0131] At block 704, a target vessel is identified in the scan area. The target vessel may be identified manually or with the aid of an imaging device such as an ultrasound imager or a camera.

[0132] At block 706, the processor of the cannulation device determines the final position of the tip of the cannulation instrument. The final position is typically the center point of the target vessel. The final position can be defined based on the vessel geometry, including the spatial [x, y, z] coordinates of the vessel, and also the polar coordinates of the vessel rotation.

[0133] At block 708, the processor determines a trajectory for guiding the intubation instrument to the final position. The trajectory is based on the current posture and position of the intubation instrument, as indicated by a position encoder associated with a motor of a positioning unit for manipulating the intubation instrument. In addition, various aspects of the trajectory can be controlled by separate motors, each of which is itself associated with a different degree of freedom (DOF) for manipulating the intubation instrument. The various DOF ​​axes and the effects of manipulating the intubation instrument along these axes have been described above.

[0134] At block 710, the processor may provide instructions to the positioning unit to steer the intubation instrument toward the final position. This may involve positioning the intubation instrument to align with the target position, steering the intubation instrument along the trajectory toward the target position, or both positioning the intubation instrument to align with the target position and steering the intubation instrument along the trajectory toward the target position.

[0135] In some examples, the instruction at block 710 can be provided iteratively, which means that the positioning unit is instructed to move the intubation device toward the target position by an incremental step, but not necessarily to the target position. Iterative positioning allows the processor to receive more imaging data, process the newly received imaging data, and correct the target position, track, or both the target position and track before the positioning unit continues to manipulate the intubation device. In this regard, the operation can continue at block 712, where the processor receives additional images or other sensor data, such as feedback data from a force sensor or position sensor. Then, at frame 714, the processor determines whether one or both of the intubation device (e.g., intubation device, imaging probe) or the patient (e.g., the patient's skin, the patient's target blood vessel) have moved in a manner inconsistent with the instruction provided to the positioning unit. If no movement is detected, the operation can continue to the next iteration of the instruction of the positioning unit at block 710, or at block 720 described below.

[0136] Otherwise, if movement is detected, then at block 716, a new target position, a new trajectory, or both a new target position and a new trajectory can be determined. In order not to determine from scratch, the detected motion can be characterized according to previously collected data, particularly according to the coordinates defining the previously collected data. Then, at block 718, the processor can instruct the positioning unit to realign the intubation instrument with the new target position, manipulate the intubation instrument along a new trajectory, or both realign the intubation instrument with the new target position and manipulate the intubation instrument along a new trajectory. In some cases, the processor can be configured to determine one or more axes on which movement occurs, and then provide instructions to the specific motor of the positioning unit's control along the determined axis. As with frame 710, instructions can be provided iteratively. Then the operation can continue at block 712.

[0137] At block 720, the processor can determine whether the cannula tip in the cannula instrument has reached the target position. If the target position has not been reached, the operation can continue at frame 710, and further iterative motion is performed toward the target position. However, if the target position has been reached, the operation can optionally continue at block 722, wherein the processor instructs the positioning unit to stop so that the cannula instrument stops moving. The processor can determine when the target is reached by tracking the posture of the target and the instrument, and further by analyzing the force sensor information. By iteratively tracking the position, orientation, or both of the instrument tip and the target center, the processor can ensure that the instrument has reached the target when the two tracking values ​​converge. In addition, the force sensor can provide an instantaneous binary confirmation of target puncture.

[0138] In some examples, the determination of successful or failed target puncture can be binary, whereby the intubation instrument stops moving upon success, and upon failure, the trajectory can be adjusted to retry the intubation attempt. Alternatively, the probability of success or failure can be determined, and the trajectory can be adjusted in a manner that increases the probability of success and / or reduces the probability of failure. In this way, the intubation instrument trajectory can be adjusted even in the middle of an insertion attempt, and before determining whether the insertion was successful or failed.

[0139] Intermediate insertion determination can be performed using a machine learning algorithm that is programmed to predict whether the target puncture will be successful. The algorithm can be trained based on recorded data from previous puncture attempts recorded from the same device or similar devices. The recorded data can include an indication of whether the attempt was successful or unsuccessful, as well as other collected data, including but not limited to image data (e.g., ultrasound information that tracks vascular movement), force sensor feedback, intubation posture data, etc. Then, during the current session, the same or similar data can be collected from device sensors and probes and used to predict the possibility of success or failure. The program can also be configured to analyze the possibility of success or failure by manipulating the intubation instrument (e.g., advancing the depth of the intubation instrument, moving the instrument to the left or right, changing the inclination of the instrument, etc.). The program can further determine whether any of the analyzed adjustments most favorably improves the possibility of success (or minimizes the possibility of failure), whereby the processor can control the manipulation of the intubation instrument accordingly.

[0140] In some cases, machine learning algorithms use multivariate learning processes — such as classification, regression, and clustering — to classify training data and then determine which class the current data belongs to and the likelihood that the classification is correct. In other cases, machine learning algorithms may use neural networks to process many raw inputs into a single probabilistic output of success / failure.

[0141] To illustrate the machine learning based routine, the following example application is provided. If during insertion but prior to vessel puncture, the vessel may begin to roll away from the insertion path. One or more sensors (including ultrasound imaging and force sensors) can be used to detect vessel rolling. However, it may be difficult to characterize the rolling of the vessel with complete certainty before the fact. Therefore, rather than identifying vessel rolling after the fact, the machine learning algorithm predicts in advance that the vessel is likely to roll away from the track and can immediately adjust the trajectory of the cannulation instrument and the target vessel to minimize the likelihood that the vessel rolling actually occurs.

[0142] The above examples generally describe the attachment of an intubation instrument. The instrument can be, for example, a needle or a catheter, depending on the particular application at a given time. In addition, the intubation device can be designed to receive several different types of intubation instruments. For example, the robotic arm of the device can be designed to receive either a needle or a catheter. The manipulators for each of the needle and the catheter can be further designed so that each manipulator is compatible with the same robotic arm, so that one instrument can be easily removed from the instrument and another instrument can be attached. Electromagnetic based attachment components can be used to attach and remove instruments. Exemplary electromagnet based attachment components are shown in Figure 8 802, using the attachment of a blood collection needle 802 for illustrative purposes. The blood collection needle 802 has a generally cylindrical shape with a first section 804 having a relatively wide diameter, the first section 804 tapering to a second section 806 having a relatively narrow diameter. The adapter 810 is configured to clamp onto the needle 802 at each of the first section 804 and the second section 806. Specifically, the adapter 810 includes an upper member 811 having a flat top surface 812 and two clamps 814 and 816 attached to the bottom surface. Each of the clamps 814 and 816 is adapted to engage with a corresponding section 804 and 806 of the needle 802. The middle portion 825 of the upper member is hollowed out, and a steel plate 835 is inserted into the middle portion 825. The steel plate is configured to engage an electromagnet included in a robotic arm (not shown). This design can be used to achieve quick detachable attachment of various cannulation instruments, so that the use of a single cannulation device can be easily converted from, for example, catheter insertion to needle insertion.

[0143] The attachment component can allow the handheld device to be easily compatible with conventional needle and / or catheter insertion equipment. For example, angle adjustment between 0 and 70 degrees, injection force of 0N to 15N, and insertion depth of 0mm to 100mm are still possible, depending on the type of vascular access being performed.

[0144] In the above example using an ultrasonic transducer, a gel attachment component attached to the head of the transducer can be used, which provides a source of acoustic coupling when the transducer is in contact with the skin. The gel attachment component can be composed of a clamp that holds the acoustic coupling gel. In addition, the acoustic coupling gel can be formulated to contain an anesthetic that is designed to be released by the ultrasonic energy to numb the insertion site prior to puncture. The gel attachment component can also be used to stabilize a blood vessel or other tissue target. In addition, the gel attachment component can flex and conform to the geometry of the patient's arm. Finally, Figure 8 The ultrasound gel attachment component and the instrument attachment component of the handheld device may be combined within a single disposable housing unit. In this way, the ultrasound and instrument attachment components may be attached to the handheld device in a single motion. For example, the two components may be fixed at a preset geometry within the disposable housing unit such that when the device is positioned and lowered over the disposable housing unit, the instrument attachment component engages with the positioning unit and the ultrasound gel attachment component engages with the ultrasound imaging unit.

[0145] The above examples generally describe procedures for an apparatus for tracking insertion of an intubation instrument using out-of-plane imaging methods. However, similar methods and procedures can be applied to planar imaging techniques, where the intubation instrument is present in the image even before reaching the target location. Fig. 9A , 9C 900a, 900b, and 900c are flow diagrams of three example procedures 900a, 900b, and 900c for performing various automated methods of aligning the trajectory of an intubation device with a target location depending on whether the imaging method is in-plane or out-of-plane and further depending on specific inputs for tracking alignment. Each example relies on two-dimensional ultrasound imaging as the primary source of image data, but Fig. 9A The two-dimensional imaging in the example procedure 900a is in-plane, while Fig. 9B and Fig. 9C The two-dimensional imaging in the example procedures 900b and 900c is out of plane, respectively. In addition, the procedure 900c relies on data received from the gyroscope and the auxiliary imaging source (e.g., camera), while the procedure 900b does not rely on data received from the gyroscope and the auxiliary imaging source.

[0146] Notice Fig. 9A, the protocol 900a begins with several manual preparation steps, including activating the device (902), clamping the desired intubation instrument to the device (904), loading the gel attachment to the device (906), and placing the device in position to image the target area (e.g., above the target area) (908). In some cases, the target area can be manually scanned to find a suitable insertion site (910). During the above-mentioned imaging process, two-dimensional ultrasound images can be captured by the device (912). As described above, these images can be continuously captured, for example in real time, to provide feedback about the movement of the target or device in the imaged target area.

[0147] A blood vessel (e.g., a vein) in an ultrasound image may be segmented to ensure that the vessel is tracked in subsequent images. The device may check whether the blood vessel in the captured image is properly segmented (914), and if not, wait for additional scans of the target area to be obtained and processed (910, 912). If the blood vessel is properly segmented, the user of the device may attempt to hold the device in place (916) rather than scanning a new target location, and operations may continue to calculate the center coordinates of the segmented blood vessel (918). The coordinate data may be stored (934) for later use.

[0148] Using the vessel coordinates, the motors used to position the instrument can be controlled so that the trajectory of the instrument is properly aligned with the target location, such as the center of the vessel. This can involve either or both of the following: operating a motor used to control the Y-axis alignment of the instrument (e.g., along axis Y n , described above as DOF3) (920); and operating a motor for controlling the Z axis or insertion depth of the instrument (e.g., along axis Z n , described above as DOF2) (924). The motor operation (920, 924) can be tracked by corresponding position encoders (922, 926).

[0149] A trajectory for moving the instrument to the center of the vessel can be calculated (928), and the calculated trajectory can be further checked for alignment with the center of the vessel (930). If the alignment is incorrect, the operation can return to controlling the motors (920, 924) to realign the instrument and recalculate the trajectory for moving the instrument to the target location (928). As described above, this tracking can be performed continuously to prevent accidental movement of the target vessel or device. However, for clarity, the trajectory alignment tracking is only shown once in the procedure.

[0150] Assuming the trajectory is properly aligned, kinematic software (936) included in the processor can begin moving the instrument along its insertion axis (938) based on the vessel center coordinates (934) and further based on image data collected from the ultrasound image indicating the position of the instrument tip (932). The instrument can be moved repeatedly, periodically checking whether the tip has reached the center of the vessel, such as based on ultrasound imaging (940). If the instrument tip has not reached the center of the vessel, the instrument can be moved further forward along the instrument's insertion axis (938). Otherwise, if it is determined that the instrument has reached the center of the vessel, or more generally the target location, then operation can continue with the designated function of the cannulation instrument (e.g., blood collection) (948).

[0151] Optionally, the determination of whether to initiate a designated operation of the cannulation instrument can be further based on force data from the force sensor (944). For example, the processor can determine whether the cannulation instrument has correctly punctured the blood vessel (942) based on the force sensor data (944). For example, the force sensor data can be classified based on several curve graphs (946), and if the curve graph of the force data matches the curve graph indicating correct puncture, the operation can continue with the designated function of the cannulation instrument (e.g., blood collection) (948). Additionally or alternatively, the processor can determine whether the cannulation instrument has correctly punctured the blood vessel (942) based on fluid flow data (944), such as feedback from a blood flow sensor in the case of blood vessel cannulation. For example, if the fluid flow data indicates that there is fluid flow in the cannulation instrument, this may be a sign of successful cannulation, and the operation can then continue with the designated function of the cannulation instrument (e.g., blood collection) (948). Otherwise, if the force sensor data is classified as a different type of profile, such as vessel rolling or overshooting the vessel wall, or if no fluid flow is detected, or any combination of feedback data, then the operation may proceed to move the instrument back along its insertion axis (938). In other cases, the success of the cannulation may be further based on the cannulation posture information, which may be image data (such as from an ultrasound imager) or motor feedback data.

[0152] Notice Fig. 9B , procedure 900b is similar to procedure 900a, except that the ultrasound imaging (912) is performed out of plane rather than in plane. As a result, the image of the target location does not feature the cannulation instrument until late in the insertion process. When the instrument tip is finally located in the image (952), the location of the instrument tip is detected (950) and can be used as an indication of whether the instrument has reached the target location so that the cannulation operation (e.g., blood sampling) can begin (948).

[0153] Additionally, in protocol 900b, the vessel center coordinates are defined in terms of the Y and Z axes of the vessel (958), as opposed to the X and Z axes described in protocol 900a. The Y and Z axis coordinates are then stored (954) as input to the kinematics software (936) for driving the cannulation instrument toward the target location (938).

[0154] Notice Fig. 9C , procedure 900c is similar to procedure 900a in that the vessel center coordinates are defined in terms of the vessel's X-axis and Z-axis (918), and is similar to procedure 900b in that the ultrasound imaging is performed using an out-of-plane approach. Procedure 900c differs in that each of a gyroscope 960 and an auxiliary imaging device (e.g., a camera) (970) is provided to enhance tracking of the device orientation and calculation of the vessel coordinates, respectively. The gyroscope collects data, such as the absolute orientation of the device, which in turn can indicate the device's yaw (962), pitch (954), and roll (966). The auxiliary imaging device (970) can image target locations, including a target vessel, to improve tracking of the vessel and provide more accurate information, or a second set of information to check against data from the primary imaging source about where the target vessel is. In the case of gyroscope data, the collected data is provided as input to calculate and monitor device orientation (974), which can then be used to align the intubation instrument with the target location (920, 924) and further calculate and track the trajectory of the intubation instrument towards the target location (928, 930).

[0155] In other example procedures, tracking of device orientation using a gyroscope and auxiliary imaging equipment may be performed in conjunction with planar imaging techniques, Y-axis and Z-axis tracking of vessel coordinates, or both.

[0156] The above examples demonstrate control algorithms and designs for automating venipuncture and other cannulation methods using a handheld device. However, it will also be appreciated by those skilled in the art that the handheld device may be used manually if desired. In addition, a combination of manual input and automated control may be preferred. For example, the clinician may be tasked with selecting a patient's preferred vein, and the automated process is used to determine the precise target location within the selected vein. For example, if the patient moves during the process, the clinician may choose to terminate the cannulation process without waiting for the device to detect the patient's movement. The clinician may further observe the image data on the device display to ensure that the automated process is being performed correctly.

[0157] The device may also include a combination of manual input and automatic control. A particular form factor of the device may be selected to allow freehand use, where all degrees of freedom are manually controlled. Other form factors may be selected for imaging only, where the cannula unit is removed; or for imaging to guide needle insertion; or for attachment to a passive support arm that provides a fixation device for the device; or for automatically positioning the device so that the needle is aligned with the vessel and then manually inserted; or for manually positioning the needle to align with the vessel and then automatically inserted; or for automatically positioning the needle while allowing manual adjustment and then automatically or manually inserted.

[0158] FIG. 10A to FIG. 10I An example of a device with two degrees of freedom is shown: height (z m ) and insertion depth (Inj m ).exist Fig. 10A In some examples, the height can be controlled by a motor that can translate the needle 1080 along the Z axis. The insertion depth can be controlled by a motor that can translate the needle in the XZ plane in the longitudinal direction of the needle. In some examples, the pitch θ of the needle in the XZ plane can be adjusted or can be fixed.

[0159] The device 1000 includes a manipulator 1010 on one side and a probe 1030, such as an ultrasound probe, on the other side. The probe 1030 can be mounted to the manipulator 1010, such as by a mounting frame 1025 (see Fig. 10B ) is mounted to the manipulator 1010. In the case of an ultrasound probe, a clamp 1035, such as for holding a gel, may be mounted to the bottom portion of the probe 1030.

[0160] like Fig. 10B As shown, it is Fig. 10A In an exploded view of the device 100 of FIG. 1 , the manipulator 1010 may include a housing in which a first motor 1060, such as a linear stage DC motor, is housed for controlling the position of the device in a height direction. The height of the device can be considered to be the height of the manipulator 1010 relative to the probe 1030, because the bottom of the probe 1030 (e.g., the bottom surface of the clamp 1035) can be placed on or near the patient to provide appropriate imaging of the target location. The housing of the manipulator 1010 can also house one or more processors 1020, such as a microcontroller, for controlling the operation of the manipulator. The needle 1080 can be housed in a needle insertion housing 1050 located below the housing of the manipulator 1010. A second motor 1040, such as a spindle drive motor, can also be housed in the needle insertion housing 1050 for controlling the insertion depth of the needle.

[0161] from Fig. 10BIt can be seen that the needle insertion housing 1050 can also accommodate a guide rail 1070 therein for guiding the insertion movement of the needle 1080 when the second motor 1040 drives the needle. A pair of ball bearing seats 1072 can be coupled to the guide rail 1070 to allow the needle 1080 to be slidably mounted to the guide rail 1070 under the control of the second motor 1040.

[0162] In addition, one or more sensors may be provided to provide feedback about the position of the device, needle 1080, or both, and the travel of the cannula. For example, a force sensor 1064 may be provided to detect the amount of force applied to the needle during insertion. In addition, a gyroscope and accelerometer may be included in the microcontroller 1020 to track the attitude and orientation of the device.

[0163] The needle can be removably attached to a clamp 1078, which in turn can be attached to an electromagnet 1074 via a washer 1076. The electromagnet can enable the needle clamp to be easily inserted and removed from the needle insertion housing 1050.

[0164] FIG. 10C to FIG. 10I Additional views of various portions of device 1000 are provided to more clearly illustrate the integration of Fig. 10A and Fig. 10B The relative positioning of the components described. For example, Fig. 10C The entire device 1000 is shown, including the probe 1030 and the clamp 1035. Fig. 10D The device is shown with the probe and clamp removed to more clearly illustrate the mounting frame 1025 to which the probe 1030 may be removably mounted. Fig.10E The device is shown with the frame and first motor removed to more clearly show the mounting frame for mounting the needle insertion housing 1050 to the rest of the manipulator. Fig.10F The device is shown with the needle housing removed, more clearly showing the second motor 1040 for driving the needle insertion depth, and the guide track 1070 in which the needle moves. Figure 10G The device is shown with the second motor and guide removed, so as to more clearly show the housing of the bearing 1072 used to guide the needle on the guide. Fig. 10H The device is shown with the housing removed, more clearly showing the bearing 1072 and guide 1070, as well as the position of the electromagnet relative to the guide when the needle is attached, and the position of the force sensor 1064 at the back of the needle holder. Fig.10I Only the force sensor, electromagnet 1074, needle holder 1078 and needle 1080 are shown as the rest of the device has been removed.

[0165] FIG. 10A to FIG. 10IThe example of 1000 shows the structure of a device with two degrees of freedom (2-DOF). Other devices with more or fewer degrees of freedom can be designed using the example device 1000 as a basic structure. For example, FIG. 11A to FIG. 11G A device 1100 having three degrees of freedom (3-DOF) is shown. In this example, the three degrees of freedom include the same two degrees of freedom as the previous device 1000, namely needle height and insertion depth. In addition, the 3-DOF device also includes an additional motor for lateral manipulation of the needle in the Y-axis direction.

[0166] like Fig.11A As shown, the device 1100 is designed in a similar manner to the 2-DOF device 1000 described above in structure, but is not identical. For example, the device 1100 still includes a probe side and a manipulator side, and the sides are mounted to each other in a similar manner. The device 1100 also has a needle insertion housing, but the housing in the 3DOF system is slightly larger than the housing in the 2DOF system due to the presence of additional components. Some significant differences between the two structures are described in more detail below.

[0167] With 2-DOF device 1000 Fig. 10C , Fig.10I Same, Fig. 11B , Fig.11G Additional views of portions of 3-DOF device 1100 are provided to more clearly illustrate the integration of Fig.11A Describes the relative positioning of components. Fig. 11B The entire apparatus 1100 including the manipulator, probe and gripper is shown. Fig. 11C The device is shown with the probe, the clamp and the mounting frame for the probe removed. The microcontroller, the first motor and its housing are also removed to more clearly show the mounting frame for mounting the needle insertion housing to the rest of the manipulator. Fig.11D The device is shown with the needle housing removed to more clearly show the lateral movement mechanism 1110 for guiding the needle along the Y axis. The lateral movement mechanism 1110 can include a third motor 1112, such as another spindle drive DC motor, for driving the rotational movement of the gear 1114 along the track 1116. The track can have a length along the Y axis so that rotation of the gear 1114 causes the track 1116 to translate to one side or the other along the Y axis. The needle clamp can be coupled to the track 1116 so that translation of the track 1116 also causes the needle to translate laterally.

[0168] Fig.11E The device is shown with the third motor 1112 removed. Track 1116 is shown also coupled to the housing of the second motor so that the second motor can translate with the needle from side to side to control the insertion depth independently of the lateral position of the needle. Fig.11F and Fig.11G To a large extent Fig. 10H and Fig.10I The above-mentioned 2-DOF device 1000 is equivalent.

[0169] Fig.12 An example workflow 1200 of a control device for controlling the operation of a needle and device during an intubation event is shown. The workflow may begin with a scanning operation 1210, whereby the probe of the device may be positioned at a target location for intubation. Positioning the device may include collecting image data, such as image frames from a camera or video recording device.

[0170] The workflow may continue with ultrasound imaging of the target location using the device probe 1220. Ultrasound imaging may include out-of-plane imaging 1222 and may provide imaging beneath the patient's skin (eg, 0 mm to 20 mm deep) using ultrasound waves.

[0171] The workflow may continue with image analysis 1230 of the collected images. The images may be used to identify vessels for cannulation and enhance the images of the vessels 1232, such as by determining the orientation of the vessels by defining the walls of the target vessels with the aid of segmentation 1234. Finally, a vessel may be selected 1236 and tracked frame by frame 1238. In the case of a vein, the vein may be further analyzed to determine the blood flow through the vessel, thereby ensuring that the vessel is a good candidate for cannulation.

[0172] Once a target vessel is selected, scanning can be stopped as indicated by user input 1240. User input can also be used to determine whether to initiate the cannulation process 1250. If a decision is made not to initiate cannulation (NO), such as if the user is not satisfied with the identified target vessel, the operation can return to image analysis 1230 so that another target vessel can be selected. Otherwise, if insertion is initiated (YES), the workflow can continue with automatic mechanical control of the cannulation instrument 1260, which can involve calculating the kinematics of the free holding device if the free holding device is moved above the target position 1262, aligning the cannulation instrument along the Y and Z axes to maintain a trajectory toward the target vessel 1264, and advancing the cannulation instrument toward the target vessel 1266 along the depth of the insertion axis.

[0173] When the cannulation instrument reaches the target location, the workflow may also perform puncture detection to determine whether the instrument has been or will be properly inserted into the vessel 1270. The determination of successful insertion may be based on several factors, including the position of the needle, such as may be indicated by a position encoder used to control the instrument's motor 1272, force feedback indicated by a force sensor coupled to the instrument 1274, ultrasound imaging 1276 to provide visual tracking of the insertion, motor current feedback 1278 indicating whether the instrument meets expected or unexpected resistance that will result in a motor current spike, and blood flow or fluid flow feedback to confirm that the cannulation attempt has resulted in the extraction of blood or fluid 1280. This feedback data, either individually or collectively, may be used in the automated mechanical control stage 1260 of the workflow to correct instrument alignment.

[0174] The above-described apparatus and methods may be applied to human patients and other animal subjects, such as in veterinary applications. More generally, those skilled in the art will readily understand and appreciate that the above-described apparatus methods for correcting the trajectory of a handheld intubation instrument based on movement of a target location may be applicable to any intubation procedure in which there may be unintended movement of the subject, the person holding the device, or both.

[0175] Although the present invention has been described herein with reference to specific embodiments, it should be understood that these embodiments are merely illustrative of the principles and applications of the present invention. It should therefore be understood that various modifications may be made to the illustrative embodiments and other arrangements may be designed without departing from the spirit and scope of the present invention as defined by the appended claims.

Claims

1. A device for guiding a cannula toward a target location under the skin of a patient, the device comprising: an imaging probe configured to continuously generate imaging data of the target location, wherein the target location is the center of a target blood vessel; A positioning unit, the positioning unit comprising: a plurality of motors configured to steer a cannula attached to the positioning unit toward the target location, at least one cannula position sensor configured to generate cannula posture data indicative of a position and orientation of the cannula based on movement of a corresponding motor; and at least one device position sensor configured to generate device posture data indicative of an orientation and position of the device; and A processor, the processor being configured to: receiving imaging data from the imaging probe; receiving cannula posture data from the at least one cannula position sensor; receiving device posture data from the at least one device position sensor; identifying the target location from the imaging data; determining a trajectory for steering the cannula toward the target location based on the imaging data, the cannula pose data, and the device pose data; instructing the positioning unit to manipulate the cannula according to the determined trajectory; identifying, during manipulation of the cannula, a misalignment of the determined trajectory with the target location based on detected movement of the target location; updating the determined trajectory to a corrected trajectory based on the imaging data, the cannula posture data, and the device posture data; and instructing the positioning unit to steer the cannula according to the corrected trajectory, wherein the apparatus further comprises a force sensor configured to generate force data indicative of a measured amount of force exerted on the cannula, and The processor is configured to receive the force data from the force sensor and classify the obtained force data into one of a plurality of predetermined events, wherein the plurality of predetermined events include a successful intubation event; and in the absence of a successful intubation event, instruct the positioning unit to manipulate the intubation in response to the classified event.

2. The device according to claim 1, wherein: The imaging probe is an ultrasound probe, and wherein the imaging data generated by the imaging probe is one of two-dimensional imaging data, stereoscopic three-dimensional imaging data, and bi-plane imaging data.

3. The device according to claim 2, wherein: The processor is also configured to identify and determine a posture of a target blood vessel beneath the patient's skin based on the imaging data.

4. The device according to claim 3, wherein: The device is adapted to obtain imaging data of the target location from multiple angles, wherein the processor is further configured to perform at least one of the following: constructing three-dimensional imaging data by receiving the imaging data obtained from multiple angles; and The three-dimensional posture of the target location under the patient's skin is determined by the imaging data obtained from multiple angles.

5. The device according to any one of claims 1 to 4, wherein: The imaging data includes video data of the target position acquired at different times, and wherein the processor is configured to determine the trajectory for steering the cannula toward the target pose based on a comparison between a previous frame and a subsequent frame of the video data.

6. The device according to any one of claims 1 to 4, wherein: The positioning unit is configured to manipulate the cannula in at least two degrees of freedom.

7. The device according to any one of claims 1 to 4, wherein: The processor is configured to identify, based on device pose data received from the at least one device position sensor, that the determined trajectory has become misaligned with the target position.

8. The device according to claim 7, wherein: The at least one device position sensor includes at least one of a camera, an electromagnetic sensor, a gyroscope, and an accelerometer.

9. The apparatus of claim 1, further comprising a force sensor configured to generate force data indicative of a measured amount of force exerted on the cannula, wherein The processor is configured to: receiving force data from the force sensor and updating the cannula posture data to a corrected position and orientation based on the received force data; and The cannula positioning unit is instructed to steer the cannula according to the corrected position and orientation.

10. The apparatus of claim 9, further comprising a blood flow sensor configured to generate blood flow data indicative of blood flowing into the cannula, wherein The processor is configured to: receiving blood flow data from the blood flow sensor; and The cannula positioning unit is instructed to steer the cannula further based on the blood flow data.

11. The device according to claim 1, wherein: The processor is configured to receive the force data from the force sensor and: acquiring force distribution through the force data; The obtained force distribution is classified as one of the successful cannulation event, the vessel rolling event, the undershoot event, and the overshoot event.

12. The device according to claim 1, wherein: The at least one cannula position sensor includes a plurality of motor sensors, each motor sensor being associated with a corresponding one of the plurality of motors to generate cannula posture data.

13. The device according to claim 1, in, The device is configured to be handheld.

14. The device according to claim 13, wherein: The plurality of motors are configured to manipulate at least an insertion depth of the cannula and an insertion angle of the cannula.

15. The device according to claim 14, wherein: The processor is communicatively coupled to the imaging probe and is configured to determine the insertion depth based at least in part on data received from the imaging probe, and wherein the insertion angle is determined based on the insertion depth.

16. The device of any one of claims 13 to 15, further comprising an accelerometer and a gyroscope, the accelerometer and the gyroscope configured to provide data indicative of motion and orientation of the device, respectively, to the processor, wherein The processor controlling operation of the plurality of motors is further automated based in part on the data from the accelerometer and the gyroscope.

Images