Robot automatic puncture method, puncture equipment, robot and storage medium

Abstract

The invention discloses a robot automatic puncture method, puncture equipment, a robot and a storage medium, and the robot automatic puncture method comprises the steps: obtaining the position of a target puncture target spot from a collected depth image, a color image and a near-infrared image; controlling the movement of a puncture mechanism according to the current position of the puncture needle and the position of the target puncture target; target detection and tracking are carried out on a puncture needle and a hand region through a color image of a puncture picture, and the puncture process is monitored in real time; and performing three-dimensional reconstruction and puncture needle navigation on subcutaneous blood vessels of the hand. The target spot recognition accuracy is improved, meanwhile, the problems that the position of the puncture needle is wrong and people are accidentally injured by the puncture needle are avoided by conducting real-time monitoring on the navigation and puncture process of the puncture needle, and therefore the safety of the automatic puncture process of the robot is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of venipuncture robots, and in particular to a robot-automated puncture method, a puncture device, a robot and a storage medium. Background Art

[0002] An automated puncture robot is an intelligent device that combines automation, robotics, and medical expertise to perform various tasks during the intravenous infusion process. It utilizes image processing and machine vision to identify the location of a blood vessel and then punctures it with a precisely controlled needle. This automated puncture robot not only improves hospital efficiency and reduces the burden on medical staff, but more importantly, it enhances the safety and accuracy of the infusion process, reducing the risk of complications caused by human error.

[0003] Currently, automated puncture solutions typically use ultrasound probes to identify veins. However, the morphological information obtained about the target vessel can change due to pressure from the ultrasound probe, resulting in significant errors in the three-dimensional information recognition. Alternatively, a monocular near-infrared camera is used to continuously capture two-dimensional near-infrared images of the target. An image neural network then scores the veins identified in these two-dimensional infrared images, with the vein with the highest score being designated as the target. During the puncture, the near-infrared camera is used to capture the current posture of the puncture needle in real time. These solutions lack effective needle navigation and puncture process monitoring, making them prone to problems such as incorrect needle placement and accidental needle injury. Summary of the Invention

[0004] In order to address the defects of the prior art, the purpose of the present invention is to provide a robotic automatic puncture method, which improves the accuracy of target recognition and the safety of the puncture process.

[0005] In order to achieve the above object, the present invention provides a robot automatic puncture method, comprising:

[0006] Obtain the location of the target puncture point from the acquired depth image, color image and near-infrared image;

[0007] Controlling the movement of the puncture mechanism according to the current position of the puncture needle and the position of the target puncture point;

[0008] Through the color image of the puncture screen, the puncture needle and hand area are detected and tracked, and the puncture process is monitored in real time;

[0009] Perform three-dimensional reconstruction of the subcutaneous blood vessels in the hand and puncture needle navigation.

[0010] Furthermore, the step of obtaining the position of the target puncture point from the collected depth image, color image and near-infrared image further includes:

[0011] A depth image and a color image of the hand are acquired through a depth camera, and a near-infrared image of the hand is acquired through a near-infrared camera;

[0012] Extracting and matching feature points of the color image and the near-infrared image, calculating a homography transformation matrix based on the matching results, and obtaining a near-infrared image after homography transformation;

[0013] Image enhancement algorithm is used to enhance the near-infrared image after homography transformation;

[0014] Perform blood vessel segmentation on the enhanced near-infrared image to obtain the blood vessel area;

[0015] extracting a blood vessel centerline from the blood vessel region;

[0016] Segmenting the blood vessel centerline into multiple segments, and using each segmentation point as a candidate target point;

[0017] Utilize an expert model trained based on a regression model or a deep learning model to screen out target puncture targets from candidate targets;

[0018] Calculate the spatial coordinates of the target puncture point.

[0019] Furthermore, the steps of extracting and matching feature points of the color image and the near-infrared image, calculating a homography transformation matrix according to the matching results, and obtaining a near-infrared image after homography transformation further include:

[0020] Extracting a first feature point from a grayscale image of the color image, extracting a second feature point from a near-infrared image, and performing feature matching on the first feature point and the second feature point to obtain a matched feature point pair;

[0021] Calculate the homography transformation matrix based on the matched feature point pairs;

[0022] Based on the homography transformation matrix, a near-infrared image after homography transformation is obtained.

[0023] Furthermore, the step of screening target puncture targets from candidate targets using the expert model obtained through regression model training further includes:

[0024] 1) Collect target data; the target data includes: the straightness, diameter, and punctureable length of the target blood vessel, which is scored by experienced puncture workers;

[0025] 2) Establish a regression model with the straightness, diameter, and punctureable length of the target vessel as input and the target score as output;

[0026] 3) Using the collected target data, a regression model is trained to derive weighted values for the straightness, diameter, and puncture length of the target blood vessel;

[0027] 4) For multiple candidate targets in the same image, perform regression analysis according to the weights calculated in step 3) to obtain a score for each candidate target;

[0028] 5) Sort candidate targets by score;

[0029] 6) The candidate target with the highest score is recommended as the target puncture target.

[0030] Furthermore, the step of controlling the movement of the puncture mechanism according to the current position of the puncture needle and the position of the target puncture point further includes:

[0031] Calculate the coordinates of the puncture needle in the global coordinate system;

[0032] Moving the puncture needle to the target puncture position according to the current coordinates of the puncture needle in the global coordinate system and the spatial coordinates of the target puncture point;

[0033] Combined with ultrasound images, the depth and diameter of the blood vessel where the target puncture point is located are obtained for puncture navigation;

[0034] adjusting the angle of the puncture needle;

[0035] Push the puncture needle horizontally to complete the puncture.

[0036] Furthermore, the step of detecting and tracking the puncture needle and the hand area through the color image of the puncture screen and monitoring the puncture process in real time also includes:

[0037] A color image of the puncture scene is acquired in real time by a color camera, wherein the puncture scene includes the puncture needle and the punctured hand area;

[0038] A method combining YOLOv10 and Kalman filtering is used to extract puncture needle features and hand area features from the color image;

[0039] Get the current position and length of the puncture needle;

[0040] Get the current position of the hand area;

[0041] According to the current position and length of the puncture needle and the current position of the hand area, an inclusion relationship between the puncture needle and the hand area is obtained;

[0042] The current puncture state is obtained according to the inclusion relationship and the current exposed length of the puncture needle.

[0043] Furthermore, the step of performing three-dimensional reconstruction of the subcutaneous blood vessels of the hand and puncture needle navigation also includes:

[0044] Acquiring continuous cross-sectional and profile images through ultrasound equipment;

[0045] The SegFormer neural network algorithm is used to segment the blood vessel area from the image;

[0046] Obtain the outline and mask of the blood vessel area, arrange them according to the image acquisition order, perform three-dimensional reconstruction of the blood vessel, and obtain the three-dimensional blood vessel structure;

[0047] A method combining YOLOv10 and Kalman filtering is used to detect and track the puncture needle and obtain its real-time position, speed, and direction.

[0048] The positional relationship between the puncture needle and the blood vessel wall is determined based on the real-time position, movement speed and movement direction of the puncture needle, and the puncture needle is navigated based on the positional relationship.

[0049] On the other hand, the present invention also provides a puncture device, comprising a memory, a processor, and a computer program stored in the memory and runnable on the processor, wherein the processor is configured to execute the computer program stored in the memory to implement the steps of the robotic automatic puncture method as described above.

[0050] On the other hand, the present invention also provides a robot comprising the puncture device as described above.

[0051] On the other hand, the present invention further provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program is loaded and executed by a processor to implement the steps of the robotic automatic puncture method as described above.

[0052] The robot-assisted puncture method provided by the present invention has the following beneficial effects compared with the prior art:

[0053] The recognition of puncture targets based on depth images and near-infrared images can improve the accuracy of dynamic target recognition; the entire robot puncture process is monitored in real time based on color images, and the three-dimensional reconstruction of subcutaneous blood vessels on the back of the hand and the puncture needle navigation are performed, which improves the safety of automatic puncture and effectively prevents the occurrence of problems such as incorrect puncture needle positioning and accidental injury by the puncture needle.

[0054] Other features and advantages of the present invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0056] Figure 1 This is a flow chart of a robot automatic puncture method according to an embodiment of the present invention;

[0057] Figure 2 A flow chart of a target recommendation algorithm according to an embodiment of the present invention;

[0058] Figure 3 This is a diagram showing the hand image enhancement effect according to an embodiment of the present invention;

[0059] Figure 4 2. A schematic diagram of a blood vessel segmentation result of a hand image according to an embodiment of the present invention;

[0060] Figure 5 Schematic diagram of blood vessel centerline extraction results according to an embodiment of the present invention;

[0061] Figure 6 is a schematic diagram of a blood vessel centerline segmentation result according to an embodiment of the present invention;

[0062] Figure 7 Schematic diagram of a puncture target according to an embodiment of the present invention;

[0063] Figure 8 This is a flowchart of puncture screen monitoring according to an embodiment of the present invention;

[0064] Figure 9 is a schematic diagram of a color image of a puncture screen according to an embodiment of the present invention;

[0065] Figure 10 A flowchart of three-dimensional reconstruction of blood vessel images and puncture navigation according to an embodiment of the present invention;

[0066] Figure 11 Schematic diagram of puncture needle tracking under ultrasound according to an embodiment of the present invention;

[0067] Figure 12 Schematic diagram of the structure of a puncture device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0068] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0069] Embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.

[0070] It should be understood that the terms "first," "second," and the like may be mentioned in the present invention only to distinguish different data or units, and are not intended to limit the order or interdependence of the functions performed by these data or units. These terms are used solely to distinguish one feature from another. For example, a first feature may be referred to as a second feature, and similarly, a second feature may be referred to as a first feature, without departing from the scope of the exemplary embodiments.

[0071] It should be noted that the modifications of "one" and "plurality" mentioned in the present invention are illustrative and non-restrictive. Those skilled in the art will understand that unless the context clearly indicates otherwise, they should be understood as "one or more". "Plurality" should be understood as two or more.

[0072] In an embodiment of the present invention, a robotic automatic puncture method is provided, comprising the following steps: obtaining the position of a target puncture point from collected depth images, color images, and near-infrared images; controlling the movement of a puncture mechanism based on the current position of the puncture needle and the position of the target puncture point; performing target detection and tracking of the puncture needle and hand area through a color image of the puncture screen, and monitoring the puncture process in real time; and performing three-dimensional reconstruction of the subcutaneous blood vessels of the hand and puncture needle navigation.

[0073] Figure 1 This is a flow chart of the robot automatic puncture method according to an embodiment of the present invention. Figure 1 The automatic puncture robot algorithm of the present invention is described in further detail.

[0074] First, in step 101, a depth image, a color image, and a near-infrared image of the hand are collected, and after image processing, the target puncture point is identified through the expert model, and the position of the target puncture point is obtained.

[0075] In the embodiment of the present invention, the target recommendation algorithm combines image processing and expert model to perform target recommendation, which can quickly screen out target puncture targets suitable for puncture. The specific steps include:

[0076] 11) Acquire a depth image and a color image of the hand using a depth camera, and acquire a near-infrared image of the hand using a near-infrared camera;

[0077] 12) extracting feature points from the grayscale image of the color image and the near-infrared image respectively and performing feature point matching, calculating a homography transformation matrix based on the matching results, and obtaining a near-infrared image after homography transformation;

[0078] 13) Using image enhancement algorithm to enhance the near-infrared image after homography transformation;

[0079] 14) performing blood vessel segmentation on the enhanced near-infrared image to obtain the blood vessel area;

[0080] 15) extracting a blood vessel centerline from the blood vessel region;

[0081] 16) Segmenting the blood vessel centerline into multiple segments, with each segmentation point being a candidate target point;

[0082] 17) Using the expert model to select target puncture targets from candidate targets;

[0083] 18) Calculating the spatial coordinates of the target puncture point.

[0084] In step 11), a depth camera is used to acquire depth and color images of the hand (e.g., the back of the hand, forearm, elbow, etc.), while a near-infrared camera is used to acquire a near-infrared image of the hand. Due to the inconsistency of the image sizes, the depth and color images need to be soft-aligned (adjusting the position and angle of the images through a smooth transition while maintaining the image content to reduce noticeable stitching artifacts and visual abruptness).

[0085] In step 12), a homography transformation is performed on the color image and the near-infrared image to obtain a homography-transformed near-infrared image. Specifically, first feature points are extracted from the grayscale image of the color image, and second feature points are extracted from the near-infrared image. These first and second feature points are then matched to obtain a pair of matched feature points. A homography transformation is then applied to the matched feature point pairs to obtain a homography-transformed near-infrared image.

[0086] A homograph is a plane-to-plane projective transformation, mapping points on one plane to another plane using a 3×3 matrix. This transformation preserves collinearity, meaning that straight lines remain straight lines after the transformation. Homographies are widely used in computer vision and image processing, particularly in image registration, stitching, and camera pose estimation.

[0087] In an embodiment of the present invention, the steps of homography transformation are as follows:

[0088] 21) Use feature point extraction algorithms (such as SIFT, ORB, SURF algorithms, etc.) to extract feature points from the image;

[0089] 22) Extract the descriptor corresponding to each feature point;

[0090] 23) Match the descriptors of the feature points using a matching algorithm (such as the KNN nearest neighbor algorithm) to find matching feature point pairs in the two images;

[0091] 24) Use the RANSAC algorithm to eliminate incorrect matching feature point pairs to improve the accuracy and robustness of matching;

[0092] 25) Solve the system of equations and calculate the homograph matrix: Solve the linear equations by at least 4 pairs of matching feature points and calculate the homograph matrix;

[0093] 26) Perform homography transformation on the image.

[0094] The near-infrared camera and the depth camera are aligned by feature matching, and the position of the puncture target is mapped to the coordinate system of the depth camera to obtain the depth information of the puncture target. This depth information has the same accuracy as the depth camera (ignoring matching error and camera calibration error).

[0095] In step 13), an image enhancement algorithm is used to enhance the near-infrared image after homography transformation to highlight the vascular features in the hand image. The image enhancement algorithm used in the embodiment of the present invention is histogram equalization. The near-infrared image after homography transformation is enhanced by a contrast-limited adaptive histogram equalization method to increase the contrast of the vascular area in the image, thereby facilitating better vascular segmentation. The specific steps are as follows:

[0096] 31) Divide the entire image into multiple small image blocks (e.g., 8×8 image blocks). This step is to process local areas of the image independently to better adapt to the local characteristics of the image;

[0097] 32) For each image block, calculate its histogram, that is, count the number of pixels at each gray level;

[0098] 33) Calculate the cumulative distribution. When calculating the cumulative distribution, limit the contrast to prevent the histogram from being particularly steep. For example, the values in the histogram above a certain threshold a are formed into a set B. Then, each element b in B is changed to an a. The sum of the values above a is obtained by dividing c by 256 (the 256 refers to the grayscale level, that is, the grayscale level is 0 to 255) to obtain d. D is added to each grayscale level.

[0099] 34) performing equalization processing on the pruned histogram, that is, calculating the equalized grayscale value according to the cumulative distribution probability of each grayscale level;

[0100] 35) Traverse the image blocks and perform bilinear interpolation between blocks to smoothly transition the contrast changes between different blocks;

[0101] 36) The smoothed image is blended with the original image to produce the final enhanced image. This step preserves some details and features of the original image while increasing its contrast and clarity. Through these steps, the contrast-constrained adaptive histogram equalization method can enhance image contrast while suppressing noise, making it particularly suitable for applications requiring high contrast, such as medical imaging.

[0102] In medical image processing, blood vessel segmentation is a crucial step in many clinical diagnoses and treatments. Because blood vessels typically appear as elongated, low-contrast structures in images, vessel enhancement is a key step in segmentation. This can be achieved using supervised image segmentation methods based on deep learning.

[0103] In step 17), the embodiment of the present invention summarizes suitable vascular puncture rules based on the experience of experienced puncture practitioners, ultimately determining the properties of the puncture target. The puncture target features selected, in descending order of importance, are high vessel straightness, large vessel diameter, and large punctureable length. Based on the vascular puncture rules, a linear regression model is established to calculate the weights of each feature of the candidate targets. A weighted summation is performed to determine the score of each potential target, and the target with the highest score is selected for recommendation.

[0104] The algorithm steps for screening puncture targets through the expert model are as follows:

[0105] 71) Collecting target data; the target data includes: the straightness and diameter of the target blood vessel, which is scored by experienced puncture workers;

[0106] 72) Establish a regression model with the model input being the straightness, diameter, and punctureable length of the target vessel, and the model output being the target score;

[0107] 73) Using the collected target data, a regression model is trained to obtain weighted values for the straightness, diameter, and puncture length of the target blood vessel;

[0108] 74) performing regression analysis on multiple candidate targets in the same image according to the weights calculated in step 73) to obtain a score for each candidate target;

[0109] 75) Sort candidate targets by score;

[0110] 76) The candidate target point with the highest score is recommended as the final target puncture target point, and its puncture direction and puncture length are obtained. The puncture direction is determined based on the angle between the blood vessel segment and the horizontal direction.

[0111] In this embodiment, the expert model is trained using a regression model. In other embodiments, the expert model can also be trained using deep learning methods. Deep learning can automatically learn feature representations from data and capture complex nonlinear relationships in the data, thereby improving the adaptability of the expert model and achieving higher efficiency when processing complex and large-scale data.

[0112] In order to further improve the target recognition accuracy, a line segment recognition method based on deep learning can be used to extract the characteristics of the blood vessel trajectory. In step 18), the image coordinates of the puncture target are converted into spatial coordinates (X C ,Y C ,Z C ), the calculation steps are as follows:

[0113] 81) Get the depth camera internal parameter f x ,f y ,u0,v0;

[0114] 82) Obtain the two-dimensional coordinates of the puncture target on the near-infrared image and index it to the depth image to obtain the depth information Z C , Z C is the depth of the puncture target in space;

[0115] 83) Calculate the spatial coordinate X of the puncture target C , the calculation formula is: X C =(u-u0)*Z C / f x ; where u is the x-coordinate of the puncture target on the near-infrared image, Z C is the depth of the puncture target in space;

[0116] 84) Calculate the spatial coordinate Y of the puncture target C , the calculation formula is: C =(v-v0)*Z C / f y ; where v is the y coordinate of the puncture target on the near-infrared image, Z C is the depth of the puncture target in space.

[0117] The present invention aligns the near-infrared camera and the depth camera through feature matching, so that the depth information of the puncture target has the same accuracy as the depth camera. It combines image processing and expert models to recommend targets, has a fast speed, and can quickly obtain puncture targets in three-dimensional space that meet the automatic puncture rules of auxiliary puncture equipment (such as automatic puncture robots). It can improve the puncture success rate and accuracy of auxiliary puncture equipment and shorten the operation time; reduce the possibility of secondary puncture, improve the puncture success rate, and also improve the work efficiency of medical staff, and also reduce the possibility of patients suffering from local infection, systemic infection, hematoma, blood vessel rupture, nerve damage, etc. due to puncture, thereby improving the quality of medical care.

[0118] Figure 2 This is a flow chart of a target recommendation algorithm according to an embodiment of the present invention. Figures 3 to 7 The following is a schematic diagram of the hand image processing result according to the example of the present invention. Figures 2 to 7 The technical effects of the present invention are described.

[0119] like Figure 2 As shown in the figure, a depth camera first obtains a depth image and a color image, while a near-infrared camera obtains a near-infrared image. The depth image is used to subsequently determine the spatial coordinates of the target, while the color image is used for feature matching with the near-infrared image. The feature matching process includes grayscale processing of the color image to obtain a grayscale image, and extracting feature points from the grayscale image using the SIFT algorithm. Similarly, feature points of the near-infrared image are extracted using the SIFT algorithm. The two sets of feature points are then matched using the KNN algorithm, and the homography matrix is calculated. The homography-transformed near-infrared image is then obtained using the homography matrix.

[0120] The near-infrared image after homography transformation is enhanced by the image enhancement algorithm to enhance the hand image and highlight the vascular features (such as Figure 3 The enhanced hand image is segmented into blood vessels, and it is determined whether there is a blood vessel area. If there is no blood vessel area, the processing of the current near-infrared image is terminated; if there is a blood vessel area (such as Figure 4 The right side of the figure is the segmented vascular region), and the vascular centerline is extracted from the vascular region using the vascular centerline extraction algorithm (e.g. Figure 5 As shown), after segmentation by the vascular centerline segmentation algorithm (as shown Figure 6 The trained expert model is used to screen out the best puncture target, puncture direction and puncture length (as shown in Figure 7 Finally, the depth information of the puncture target is obtained based on the depth image, and the coordinates of the puncture target are converted into spatial coordinates. The algorithm ends and the positioning of the puncture target is completed.

[0121] Continue to refer Figure 1In step 102, the movement of the puncture mechanism is controlled according to the current position of the puncture needle and the position of the target puncture point.

[0122] In an embodiment of the present invention, the movement of the puncture mechanism is controlled according to the current position of the puncture needle and the position of the target puncture point, which specifically includes the following steps:

[0123] ①Use the forward kinematics equation to find the coordinates of the puncture needle in the global coordinate system

[0124] ② Move the puncture needle to the target position (i.e. the position of the target puncture point).

[0125] ③ Perform puncture: According to the blood vessel depth and diameter information obtained from the ultrasound image, set the appropriate puncture distance so that the puncture needle enters the blood vessel accurately and stops at the center of the blood vessel as much as possible;

[0126] ④ Adjust the angle of the puncture needle: adjust the angle of the puncture needle to reduce the angle between the puncture needle and the blood vessel while keeping the position of the puncture needle tip unchanged;

[0127] ⑤ Push horizontally: Push the puncture needle along the blood vessel for an appropriate distance.

[0128] In step 103, a color image of the puncture scene is collected, target detection and tracking are performed on the puncture needle and the hand area, and the puncture process is monitored in real time.

[0129] In the embodiment of the present invention, a color camera is used to monitor the entire puncture process. Figure 8 As shown in FIG, a color camera is first used to obtain a color image of the puncture screen in real time. For this color image, a method combining YOLOv10 and Kalman filtering is used to detect and track the puncture needle and the back of the hand (or other puncture sites on the hand) (as shown in FIG). Figure 9 (as shown in the figure), then calculates the positional relationship between the puncture needle and the back of the hand, and uses this positional relationship to determine whether the puncture needle is in good working order. The YOLOv10 object detection model utilizes a more efficient backbone network structure, an optimized feature fusion strategy, and advanced loss function design, improving detection accuracy while maintaining high inference speed. For automated puncture robot applications, the YOLOv10 Nano version of the object detection model was selected, ensuring detection accuracy while reducing model parameters and increasing detection speed.

[0130] In terms of target tracking, this invention employs an object tracking method based on a Kalman filter and the Hungarian algorithm, introducing an appearance model to enhance the data association process. Compared with traditional tracking methods, this method can better maintain target identity consistency over long time series, especially when the target is occluded or temporarily out of view. Furthermore, by utilizing the target features extracted by the YOLOv10 convolutional neural network, it can effectively reduce mismatches, thereby further improving tracking accuracy. Through target detection and tracking, the position of the puncture needle is monitored in real time, thereby preventing problems such as incorrect needle placement and accidental needle injury.

[0131] The steps for calculating the position relationship are as follows:

[0132] Get the current position and length of the puncture needle;

[0133] Get the current position of the back of the hand area;

[0134] Based on the current position and length of the puncture needle and the current position of the hand area, the inclusion relationship between the puncture needle and the back of the hand area is determined. If the puncture needle is included in the back of the hand area and the puncture needle length is less than the standard length (the exposed length is less than the original length), it is defined as having started puncture; if the puncture needle is included in the back of the hand area and the puncture needle length is the standard length, it is defined as ready for puncture; if the puncture needle does not intersect with the back of the hand area, it is defined as not having started puncture.

[0135] In step 104, an ultrasound device is used to perform three-dimensional reconstruction of the subcutaneous blood vessels of the hand and perform puncture needle navigation.

[0136] In the embodiment of the present invention, the process of performing three-dimensional reconstruction of blood vessels and puncture needle navigation under ultrasound is as follows: Figure 10 As shown in FIG, the method includes connecting the ultrasound device, adjusting the ultrasound device parameters, and performing continuous cross-sectional image acquisition and profile image acquisition; using the SegFormer neural network algorithm to segment the blood vessel region; before performing three-dimensional reconstruction of the blood vessel image, first obtaining the contours and masks obtained by the segmentation of the blood vessel region, arranging them according to the image acquisition sequence, and using a three-dimensional reconstruction tool (such as VTK software) to construct a three-dimensional blood vessel structure; performing target detection and tracking of the puncture needle (such as Figure 11 As shown in the figure, the green area is the puncture needle and the red area is the vein), and then the real-time position, movement speed and movement direction of the puncture needle are calculated, and the puncture needle navigation is performed in combination with the three-dimensional structure information of the blood vessels measured by ultrasound images (such as the depth and diameter of the blood vessel where the target puncture point is located).

[0137] The system combines the current frame position, speed, and direction of the puncture needle to calculate the possible position of the puncture needle in the next frame. This position information is then combined with the three-dimensional structure of the blood vessel to predict the distance between the puncture needle and the vessel wall. Based on this distance relationship, operation prompts are given to achieve puncture needle navigation. For example, if the predicted puncture needle position is too close to the vessel wall, a prompt is given to stop moving the puncture needle (activate the puncture needle emergency stop mode). Otherwise, the puncture needle can continue to move.

[0138] SegFormer is a simple and efficient transformer-based model for semantic segmentation that does not rely on positional encoding and uses a hierarchical architecture for multi-scale feature representation. SegFormer integrates a lightweight MLP decoder with a transformer to create a multi-scale feature hierarchy, which improves both performance and efficiency.

[0139] In an embodiment of the present invention, the steps of selecting VTK software as a 3D reconstruction tool to perform 3D reconstruction of blood vessels are as follows:

[0140] 1) Define the rendering window and interaction mode: initialize the vtk interaction window;

[0141] 2) Define the reading interface: define the image format and select the reading tool;

[0142] 3) Pre-settings and image reading: define image size, number of images, access location, image prefix and other information;

[0143] 4) Reconstruction parameter setting: Set the (x, y, z) parameters, where x and y are the horizontal spacing of the blood vessel sections, and z is the length spacing of the blood vessel sections;

[0144] 5) Gaussian smoothing of reconstruction results: beautify the reconstructed 3D vascular structure and remove noise;

[0145] 6) Contour calculation and edge extraction: Extract the contour of the 3D blood vessel structure and calculate the length, diameter and other information of the 3D blood vessel;

[0146] 7) Pipeline operation and visualization: Display of three-dimensional vascular structure.

[0147] In an embodiment of the present invention, a YOLOv10 plus Kalman filter method is used to detect and track the target of the puncture needle under ultrasound, and the parameters such as the real-time position, movement speed and movement direction of the puncture needle are calculated.

[0148] The robotic automatic puncture method provided by the present invention realizes automatic identification of puncture targets in the patient's hand area through a target recommendation algorithm that combines advanced image imaging, image processing and expert models, and provides the position coordinates of the puncture target to the puncture needle, thereby improving the recognition accuracy; a color camera is used to monitor the entire robotic puncture process to prevent problems such as incorrect puncture needle position and accidental injury by the puncture needle, thereby improving the safety of automatic puncture; ultrasonic equipment and artificial intelligence algorithms are used to perform three-dimensional reconstruction of blood vessels in the hand area and puncture needle navigation, thereby improving the accuracy and safety of the puncture operation.

[0149] In an embodiment of the present invention, a puncture device is also provided. Figure 12 A schematic structural diagram of a puncture device according to an embodiment of the present invention is shown in FIG. Figure 12 As shown, the puncture device of the present invention includes a processor 1201, a memory 1202, and a computer program stored in the memory 1202 and executable on the processor 1201, wherein the computer program, when read and executed by the processor 1201, implements the steps in the robotic automatic puncture method described above.

[0150] An embodiment of the present invention further provides a robot comprising the aforementioned puncture device. The robot employing the aforementioned puncture device improves target recognition accuracy while effectively avoiding problems such as incorrect puncture needle placement and accidental injury, thereby enhancing the safety of the robot's automated puncture process.

[0151] In an embodiment of the present invention, a computer-readable storage medium is further provided, in which a computer program is stored. The computer program is configured to execute the steps of the robotic automatic puncture method described above when the computer program is run.

[0152] In this embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0153] Those skilled in the art will understand that the foregoing descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art will be able to modify the technical solutions described in the foregoing embodiments or substitute equivalents for some of the technical features. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A robot automatic puncture method, characterized in that: The following steps are involved: Obtain the location of the target puncture point from the acquired depth image, color image and near-infrared image; Controlling the movement of the puncture mechanism according to the current position of the puncture needle and the position of the target puncture point; Through the color image of the puncture screen, the puncture needle and hand area are detected and tracked, and the puncture process is monitored in real time; Perform three-dimensional reconstruction of the subcutaneous blood vessels in the hand and puncture needle navigation.

2. The robot automatic puncture method according to claim 1, characterized in that: The step of obtaining the position of the target puncture point from the collected depth image, color image and near-infrared image further includes: A depth image and a color image of the hand are acquired through a depth camera, and a near-infrared image of the hand is acquired through a near-infrared camera; Extracting and matching feature points of the color image and the near-infrared image, calculating a homography transformation matrix based on the matching results, and obtaining a near-infrared image after homography transformation; Image enhancement algorithm is used to enhance the near-infrared image after homography transformation; Perform blood vessel segmentation on the enhanced near-infrared image to obtain the blood vessel area; extracting a blood vessel centerline from the blood vessel region; Segmenting the blood vessel centerline into multiple segments, and using each segmentation point as a candidate target point; Utilize an expert model trained based on a regression model or a deep learning model to screen target puncture targets from candidate targets; Calculate the spatial coordinates of the target puncture point.

3. The robot automatic puncture method according to claim 2, characterized in that: The steps of extracting and matching feature points of the color image and the near-infrared image, calculating a homography transformation matrix according to the matching results, and obtaining a near-infrared image after homography transformation further include: Extracting a first feature point from a grayscale image of the color image, extracting a second feature point from a near-infrared image, and performing feature matching on the first feature point and the second feature point to obtain a matched feature point pair; Calculate the homography transformation matrix based on the matched feature point pairs; Based on the homography transformation matrix, a near-infrared image after homography transformation is obtained.

4. The robot automatic puncture method according to claim 2, characterized in that: The step of screening target puncture targets from candidate targets using the expert model obtained by training based on the regression model further includes: 1) Collect target data; the target data includes: the straightness, diameter, and punctureable length of the target blood vessel, which is scored by experienced puncture workers; 2) Establish a regression model with the straightness, diameter, and punctureable length of the target vessel as input and the target score as output; 3) Using the collected target data, a regression model is trained to derive weighted values for the straightness, diameter, and puncture length of the target blood vessel; 4) For multiple candidate targets in the same image, perform regression analysis according to the weights calculated in step 3) to obtain a score for each candidate target; 5) Sort candidate targets by score; 6) The candidate target with the highest score is recommended as the target puncture target.

5. The robot automatic puncture method according to claim 1, characterized in that: The step of controlling the movement of the puncture mechanism according to the current position of the puncture needle and the position of the target puncture point further includes: Calculate the coordinates of the puncture needle in the global coordinate system; Moving the puncture needle to the target puncture position according to the current coordinates of the puncture needle in the global coordinate system and the spatial coordinates of the target puncture point; Combined with ultrasound images, the depth and diameter of the blood vessel where the target puncture point is located are obtained for puncture navigation; adjusting the angle of the puncture needle; Push the puncture needle horizontally to complete the puncture.

6. The robot automatic puncture method according to claim 1, characterized in that: The step of detecting and tracking the puncture needle and the hand area using the color image of the puncture screen and monitoring the puncture process in real time also includes: A color image of the puncture scene is acquired in real time by a color camera, wherein the puncture scene includes the puncture needle and the punctured hand area; A method combining YOLOv10 and Kalman filtering is used to extract puncture needle features and hand area features from the color image; According to the current position and length of the puncture needle and the current position of the hand area, an inclusion relationship between the puncture needle and the hand area is obtained; The current puncture state is obtained according to the inclusion relationship and the current exposed length of the puncture needle.

7. The robot automatic puncture method according to claim 1, characterized in that: The step of performing three-dimensional reconstruction of the subcutaneous blood vessels of the hand and puncture needle navigation also includes: Acquiring continuous cross-sectional and profile images through ultrasound equipment; The SegFormer neural network algorithm is used to segment the blood vessel area from the image; Obtain the outline and mask of the blood vessel area, arrange them according to the image acquisition order, perform three-dimensional reconstruction of the blood vessel, and obtain the three-dimensional blood vessel structure; A method combining YOLOv10 and Kalman filtering is used to detect and track the puncture needle and obtain its real-time position, speed, and direction. The positional relationship between the puncture needle and the blood vessel wall is determined based on the real-time position, movement speed and movement direction of the puncture needle, and the puncture needle is navigated based on the positional relationship.

8. A puncture device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: The processor is used to execute the computer program stored in the memory to implement the steps of the robot automatic puncture method according to any one of claims 1 to 7.

9. A robot, characterized in that: Comprising the puncture device according to claim 8.

10. A computer-readable storage medium, characterized in that The storage medium stores a computer program, which is loaded and executed by a processor to implement the steps of the robot automatic puncture method according to any one of claims 1 to 7.

Images