A robotic automatic puncture method, a puncture device, a robot, and a storage medium

Abstract

A kind of robot automatic puncture method, puncture equipment, robot and storage medium, the robot automatic puncture method includes: from the depth image, color image and near infrared image collected, the position of target puncture target point is obtained;According to the current position of puncture needle, and the position of the target puncture target point, the movement of puncture mechanism is controlled;Through the color image of puncture picture, target detection and tracking are carried out to puncture needle and hand area, and puncture process is monitored in real time;The subcutaneous blood vessel of hand is three-dimensionally reconstructed and the navigation of puncture needle is guided.The present application improves the target point recognition accuracy, and through real-time monitoring of the navigation of puncture needle and puncture process, avoids the problems such as puncture needle position error and puncture needle injury, thereby improving the safety of robot automatic puncture process.

Description

Technical Field

[0001] This invention relates to the field of intravenous puncture robot technology, and in particular to a robotic automatic puncture method, puncture device, robot, and storage medium. Background Technology

[0002] Automated intravenous infusion robots are intelligent devices that combine automation, robotics, and medical expertise to perform various tasks during intravenous infusion. They utilize image processing and machine vision technology to identify blood vessel locations and then perform punctures using precisely controlled needles. Automated intravenous infusion robots not only improve hospital efficiency and reduce the workload of medical staff, but more importantly, they enhance the safety and accuracy of the infusion process, reducing the risk of complications due to human error.

[0003] Currently, automated puncture procedures typically use ultrasound probes to identify veins. However, the morphological information of the target vessel can change due to pressure from the ultrasound probe, leading to significant errors in 3D recognition. Alternatively, a monocular near-infrared camera continuously acquires 2D near-infrared images of the target. An image neural network then scores the identified vessels in the 2D images, selecting the vein with the highest score as the target. During the puncture, the near-infrared camera continuously monitors the needle's current position. These methods lack effective needle navigation and puncture process monitoring, making them prone to problems such as incorrect needle placement and accidental needle injury. Summary of the Invention

[0004] To address the shortcomings of existing technologies, the present invention aims to provide a robotic automated puncture method that improves the accuracy of target identification and the safety of the puncture process.

[0005] To achieve the above objectives, the present invention provides a robotic automated puncture method, comprising:

[0006] The location of the target puncture point is obtained from the acquired depth images, color images, and near-infrared images;

[0007] The movement of the puncture mechanism is controlled according to the current position of the puncture needle and the position of the target puncture point;

[0008] By using color images of the puncture site, the puncture needle and hand area are detected and tracked, and the puncture process is monitored in real time.

[0009] Three-dimensional reconstruction of subcutaneous blood vessels in the hand and puncture needle navigation.

[0010] Furthermore, the step of obtaining the location of the target puncture point from the acquired depth image, color image, and near-infrared image further includes:

[0011] Depth and color images of the hand are acquired using a depth camera, and near-infrared images of the hand are acquired using a near-infrared camera.

[0012] Feature points are extracted and matched from the color image and the near-infrared image. The homography transformation matrix is ​​calculated based on the matching results to obtain the near-infrared image after homography transformation.

[0013] An image enhancement algorithm is used to enhance the near-infrared image after homography transformation;

[0014] Blood vessel segmentation is performed on the enhanced near-infrared image to obtain the blood vessel region;

[0015] Extract the vascular centerline from the vascular region;

[0016] The blood vessel centerline is divided into multiple segments, and each segmentation point is used as a candidate target point.

[0017] Target puncture points are selected from candidate targets using an expert model trained based on a regression model or a deep learning model.

[0018] Calculate the spatial coordinates of the target puncture point.

[0019] Furthermore, the steps of extracting and matching feature points from the color image and the near-infrared image, calculating the homography transformation matrix based on the matching results, and obtaining the near-infrared image after homography transformation, further include:

[0020] Extract a first feature point from the grayscale image of the color image, extract a second feature point from the near-infrared image, and perform feature matching on the first feature point and the second feature point to obtain a matched feature point pair;

[0021] Calculate the homography transformation matrix based on the matched feature point pairs;

[0022] Based on the homography transformation matrix, the near-infrared image after homography transformation is obtained.

[0023] Furthermore, the step of using an expert model trained based on a regression model to screen out target puncture points from candidate target points further includes:

[0024] 1) Collect target data; the target data includes: the straightness, diameter, and puncture length of the blood vessel where the target is located, which are scored by experienced puncture technicians;

[0025] 2) Establish a regression model. The input to the model is the straightness, diameter, and puncture length of the blood vessel where the target is located. The output of the model is the target score.

[0026] 3) Using the collected target data, a regression model is trained to obtain the weight values ​​of the straightness, diameter, and puncture length of the blood vessel where the target is located.

[0027] 4) For multiple candidate target points in the same image, perform regression analysis according to the weights calculated in step 3) to obtain the score of each candidate target point;

[0028] 5) Sort the candidate targets by score;

[0029] 6) The candidate target with the highest recommended score is the target puncture target.

[0030] Furthermore, the step of controlling the movement of the puncture mechanism based on the current position of the puncture needle and the position of the target puncture point further includes:

[0031] Calculate the coordinates of the puncture needle in the global coordinate system;

[0032] Based on the current coordinates of the puncture needle in the global coordinate system and the spatial coordinates of the target puncture point, the puncture needle is moved to the target puncture position.

[0033] By combining ultrasound images to obtain the depth and diameter of the blood vessel where the target puncture point is located, puncture navigation can be performed;

[0034] Adjust the angle of the puncture needle;

[0035] Push the puncture needle horizontally to complete the puncture.

[0036] Furthermore, the step of detecting and tracking the puncture needle and hand area using a color image of the puncture site to monitor the puncture process in real time also includes:

[0037] A color image of the puncture scene is acquired in real time using a color camera. The puncture scene includes the puncture needle and the punctured hand area.

[0038] The puncture needle features and hand region features were extracted from the color image by using a combination of YOLOv10 and Kalman filtering.

[0039] Get the current position and length of the puncture needle;

[0040] Get the current position of the hand area;

[0041] Based on the current position and length of the puncture needle, and the current position of the hand region, the inclusion relationship between the puncture needle and the hand region is determined;

[0042] The current puncture status is obtained based on the inclusion relationship and the current exposed length of the puncture needle.

[0043] Furthermore, the steps of performing three-dimensional reconstruction of subcutaneous blood vessels in the hand and puncture needle navigation also include:

[0044] Continuous cross-sectional and profile images are acquired using ultrasonic equipment;

[0045] The SegFormer neural network algorithm was used to segment blood vessel regions from images;

[0046] The contours and masks of the blood vessel regions are obtained and arranged according to the image acquisition order. The blood vessels are then reconstructed in three dimensions to obtain the three-dimensional blood vessel structure.

[0047] A method combining YOLOv10 and Kalman filtering is used to detect and track the puncture needle, and to obtain the real-time position, speed and direction of movement of the puncture needle.

[0048] Based on the real-time position, speed, and direction of the puncture needle, the positional relationship between the puncture needle and the blood vessel wall is determined, and then the puncture needle is guided according to the positional relationship.

[0049] On the other hand, the present invention also provides a puncture device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor being configured to execute the computer program stored in the memory to implement the robotic automated puncture method steps as described above.

[0050] On the other hand, the present invention also provides a robot including the puncture device described above.

[0051] On the other hand, the present invention also provides a computer-readable storage medium storing a computer program, which is loaded and executed by a processor to implement the steps of the robotic automated puncture method as described above.

[0052] The robotic automated puncture method provided by this invention has the following advantages compared with the prior art:

[0053] Based on depth and near-infrared images, the accuracy of moving target identification can be improved. The robot can monitor the entire puncture process in real time using color images, perform three-dimensional reconstruction of subcutaneous blood vessels on the back of the hand and navigation of the puncture needle, improve the safety of automatic puncture and effectively prevent problems such as incorrect puncture needle position and accidental injury to the patient.

[0054] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. Attached Figure Description

[0055] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0056] Figure 1 This is a flowchart of a robotic automated puncture method according to an embodiment of the present invention;

[0057] Figure 2 This is a flowchart of the target recommendation algorithm according to an embodiment of the present invention;

[0058] Figure 3 This is an image showing the effect of hand image enhancement according to an embodiment of the present invention;

[0059] Figure 4 This is a schematic diagram of the blood vessel segmentation result of a hand image according to an embodiment of the present invention;

[0060] Figure 5 This is a schematic diagram of the results of extracting the blood vessel centerline according to an embodiment of the present invention;

[0061] Figure 6 This is a schematic diagram of the blood vessel centerline segmentation result according to an embodiment of the present invention;

[0062] Figure 7 This is a schematic diagram of the puncture target point according to an embodiment of the present invention;

[0063] Figure 8 This is a flowchart of the puncture image monitoring process according to an embodiment of the present invention;

[0064] Figure 9 This is a color image schematic diagram of a puncture scene according to an embodiment of the present invention;

[0065] Figure 10 This is a flowchart of three-dimensional reconstruction of vascular images and puncture navigation according to an embodiment of the present invention;

[0066] Figure 11 This is a schematic diagram of ultrasound-guided needle tracking according to an embodiment of the present invention;

[0067] Figure 12 This is a schematic diagram of the puncture device according to an embodiment of the present invention. Detailed Implementation

[0068] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0069] Embodiments of the present invention will now be described in more detail with reference to the accompanying drawings. While some embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the invention. It should be understood that the accompanying drawings and embodiments are for illustrative purposes only and are not intended to limit the scope of protection of the invention.

[0070] It should be understood that the concepts of "first" and "second" mentioned in this invention are used only to distinguish different data or units, and are not intended to limit the order of functions performed by these data or units or their interdependencies. These terms are used merely to distinguish one feature from another. For example, without departing from the scope of the exemplary embodiments, a first feature may be referred to as a second feature, and similarly, a second feature may be referred to as a first feature.

[0071] It should be noted that the terms "one" and "multiple" used in this invention are illustrative rather than restrictive. Those skilled in the art should understand that, unless explicitly stated otherwise in the context, they should be understood as "one or more". "Multiple" should be understood as two or more.

[0072] In an embodiment of the present invention, a robotic automatic puncture method is provided, comprising the following steps: obtaining the position of the target puncture point from acquired depth images, color images, and near-infrared images; controlling the movement of the puncture mechanism according to the current position of the puncture needle and the position of the target puncture point; performing target detection and tracking on the puncture needle and hand area through the color image of the puncture scene, and monitoring the puncture process in real time; and performing three-dimensional reconstruction of the subcutaneous blood vessels of the hand and puncture needle navigation.

[0073] Figure 1 The following is a flowchart of the robotic automated puncture method according to an embodiment of the present invention, which will be described in conjunction with... Figure 1 The automatic puncture robot algorithm of the present invention will be described in further detail.

[0074] First, in step 101, depth images, color images, and near-infrared images of the hand are acquired. After image processing, the target puncture point is identified through an expert model, and the location of the target puncture point is obtained.

[0075] In this embodiment of the invention, the target recommendation algorithm combines image processing and expert models to recommend target points, enabling rapid selection of suitable puncture targets. Specific steps include:

[0076] 11) Acquire depth and color images of the hand using a depth camera, and acquire near-infrared images of the hand using a near-infrared camera;

[0077] 12) Extract feature points from the grayscale image of the color image and the near-infrared image respectively, and perform feature point matching. Calculate the homography transformation matrix based on the matching results to obtain the near-infrared image after homography transformation.

[0078] 13) Image enhancement algorithms are used to enhance the near-infrared image after homography transformation;

[0079] 14) Perform vessel segmentation on the enhanced near-infrared image to obtain the vessel region;

[0080] 15) Extract the vascular centerline from the vascular region;

[0081] 16) Divide the blood vessel centerline into multiple segments, and use each segmentation point as a candidate target point;

[0082] 17) Use expert models to screen out target puncture points from candidate targets;

[0083] 18) Calculate the spatial coordinates of the target puncture point.

[0084] In step 11), a depth camera is used to acquire depth and color images of the hand (such as the back of the hand, forearm, elbow, etc.), and a near-infrared camera is used to acquire near-infrared images of the hand. Since the sizes of the images are inconsistent, the depth and color images need to be soft-aligned (while keeping the image content unchanged, the position and angle of the images are adjusted through a smooth transition to reduce obvious stitching marks and visual abruptness).

[0085] In step 12), homography transformation is performed on the color image and the near-infrared image to obtain the homography-transformed near-infrared image. Specifically, a first feature point is extracted from the grayscale image of the color image, and a second feature point is extracted from the near-infrared image. Feature matching is performed on the first and second feature points to obtain matched feature point pairs. Based on the matched feature point pairs, homography transformation is applied to obtain the homography-transformed near-infrared image.

[0086] Homograph transformation is a plane-to-plane projection transformation that maps points on one plane to another using a 3×3 matrix. This transformation preserves collinearity, meaning that straight lines remain straight lines after the transformation. Homograph transformation is widely used in computer vision and image processing, especially in image registration, stitching, and camera pose estimation.

[0087] In embodiments of the present invention, the homography transformation steps are as follows:

[0088] 21) Use feature point extraction algorithms (such as SIFT, ORB, SURF, etc.) to extract feature points from the image;

[0089] 22) Extract the descriptor corresponding to each feature point;

[0090] 23) Find matching feature point pairs between two images by matching feature point descriptors using a matching algorithm (such as KNN nearest neighbor algorithm);

[0091] 24) Use the RANSAC algorithm to remove incorrect matching feature point pairs, thereby improving the accuracy and robustness of matching;

[0092] 25) Solve the system of equations and calculate the Homograph (homography transformation) matrix: Solve the system of linear equations by matching at least 4 pairs of feature points and calculate the homograph transformation matrix;

[0093] 26) Perform homography transformation on the image.

[0094] By aligning the near-infrared camera and the depth camera through feature matching, the position of the puncture target is mapped to the coordinate system of the depth camera, thereby obtaining the depth information of the puncture target. This depth information has the same accuracy as the depth camera (ignoring matching error and camera calibration error).

[0095] In step 13), an image enhancement algorithm is used to enhance the near-infrared image after homography transformation to highlight the vascular features in the hand image. In this embodiment of the invention, the image enhancement algorithm used is histogram equalization. A contrast-limited adaptive histogram equalization method is used to enhance the near-infrared image after homography transformation, improving the contrast of the vascular region in the image to facilitate better vascular segmentation. The specific steps are as follows:

[0096] 31) Divide the entire image into multiple smaller image blocks (e.g., 8×8 image blocks). This step is to process local regions of the image independently to better adapt to the local characteristics of the image;

[0097] 32) For each image patch, calculate its histogram, that is, count the number of pixels at each gray level;

[0098] 33) Calculate the cumulative distribution. When calculating the cumulative distribution, limit the contrast to prevent the histogram from becoming too steep. For example, form a set B of the values ​​in the histogram that are above a certain threshold a. Then, change each element b in B to a. Sum the values ​​above a to get c. Divide c by 256 (where 256 refers to the gray level, i.e., gray level 0 to 255) to get d. Add d to each gray level.

[0099] 34) Perform equalization processing on the trimmed histogram, that is, calculate the equalized gray value based on the cumulative distribution probability of each gray level;

[0100] 35) Traverse the image blocks and perform bilinear interpolation between blocks to smoothly transition the contrast changes between different blocks;

[0101] 36) The image after smoothing the transition is blended with the original image to obtain the final enhanced image. This step aims to preserve some details and features of the original image while increasing its contrast and sharpness. Through the above steps, the contrast-limited adaptive histogram equalization method can enhance image contrast while suppressing noise, making it particularly suitable for applications requiring high contrast, such as medical imaging.

[0102] In medical image processing, blood vessel segmentation is a crucial step in many clinical diagnoses and treatments. Since blood vessels typically appear as elongated, low-contrast structures in images, vessel enhancement is a key step in segmentation. Supervised image segmentation methods based on deep learning can be employed for blood vessel segmentation.

[0103] In step 17), the embodiments of the present invention summarize the vascular puncture rules suitable for puncture based on the experience of senior puncture workers, and finally determine the properties of the puncture target. The puncture target features are selected in descending order of importance as follows: high vessel straightness, large vessel diameter, and large punctureable length. Based on the vascular puncture rules, a linear regression model is established to calculate the feature weights of the candidate target. After weighted summation, the score of each potential target is obtained, and the target with the highest score is selected for recommendation.

[0104] The algorithm steps for selecting puncture target points using an expert model are as follows:

[0105] 71) Collect target data; the target data includes: the straightness and diameter of the blood vessel where the target is located, which are scored by experienced puncture technicians;

[0106] 72) Establish a regression model. The input to the model is the straightness, diameter, and puncture length of the blood vessel where the target is located. The output of the model is the target score.

[0107] 73) Using the collected target data, a regression model was trained to obtain the weight values ​​of the straightness, diameter, and puncture length of the blood vessel where the target is located.

[0108] 74) For multiple candidate target points in the same image, perform regression analysis according to the weights calculated in step 73) to obtain the score of each candidate target point;

[0109] 75) Sort the candidate targets by score;

[0110] 76) The candidate target with the highest recommended score is the final target puncture point, and its puncture direction and puncture length are obtained. The puncture direction is determined based on the angle between the vessel segment and the horizontal direction.

[0111] In this embodiment, the expert model is trained using a regression model. In other embodiments, the expert model can also be trained using deep learning methods. Deep learning leverages its advantages, such as automatically learning feature representations from data and capturing complex nonlinear relationships in the data, to improve the adaptability of the expert model, especially when dealing with complex, large-scale data.

[0112] To further improve target identification accuracy, a deep learning-based line segment recognition method can be used to extract blood vessel trend features. In step 18), the image coordinates of the puncture target are converted into spatial coordinates (X... C ,Y C Z C The calculation steps are as follows:

[0113] 81) Obtain the intrinsic parameters f of the depth camera x ,f y ,u0,v0;

[0114] 82) Obtain the two-dimensional coordinates of the puncture target point on the near-infrared image, index them onto the depth image, and thus obtain the depth information Z. C Z C The spatial depth of the puncture target;

[0115] 83) Calculate the spatial coordinates X of the puncture target point. C The calculation formula is: X C =(u-u0)*Z C / f x Where u is the x-coordinate of the puncture target point in the near-infrared image, and Z... C The spatial depth of the puncture target;

[0116] 84) Calculate the spatial coordinates Y of the puncture target point. C The calculation formula is: Y C =(v-v0)*Z C / f y Where v is the y-coordinate of the puncture target point in the near-infrared image, and Z... C The depth of the puncture target in space.

[0117] This invention aligns a near-infrared camera with a depth camera through feature matching, ensuring that the depth information of the puncture target has the same accuracy as that of the depth camera. Combined with image processing and expert models for target recommendation, it offers high speed and can quickly acquire puncture target points in three-dimensional space that conform to the automatic puncture rules of assisted puncture devices (such as automated puncture robots). This improves the success rate and accuracy of assisted puncture devices, shortens operation time, reduces the possibility of secondary punctures, and enhances the efficiency of medical staff. It also reduces the likelihood of patients experiencing local infections, systemic infections, hematomas, vascular rupture, nerve damage, etc., due to puncture, thereby improving the quality of medical care.

[0118] Figure 2 This is a flowchart of the target recommendation algorithm according to an embodiment of the present invention. Figures 3 to 7 This is a schematic diagram of the hand image processing result according to an example of the present invention, which will be described below in conjunction with... Figures 2 to 7 The technical effects of the present invention are explained.

[0119] like Figure 2 As shown, firstly, a depth image and a color image are obtained using a depth camera, and a near-infrared image is obtained using a near-infrared camera. The depth image is used to subsequently determine the spatial coordinates of the target points, while the color image is used for feature matching with the near-infrared image. The feature matching process includes: processing the color image into grayscale to obtain a grayscale image, and extracting feature points from the grayscale image using the SIFT algorithm; similarly, extracting feature points from the near-infrared image using the SIFT algorithm; performing feature point matching using the KNN algorithm on the two sets of feature points, thereby calculating the homography matrix and obtaining the homography-transformed near-infrared image using the homography matrix.

[0120] The near-infrared image after homography transformation is enhanced by an image enhancement algorithm to highlight vascular features (such as...). Figure 3 (As shown). The enhanced hand image is segmented to identify blood vessels, and the presence of vascular regions is determined. If no vascular region exists, the processing of the current near-infrared image ends; if a vascular region exists (e.g., ...), the processing continues. Figure 4 The right side of the image shows the segmented vascular region. The vascular centerline is then extracted from the vascular region using a vascular centerline extraction algorithm (e.g.,...). Figure 5 As shown), after segmentation by the blood vessel centerline segmentation algorithm (as shown) Figure 6 As shown), the optimal puncture target point, puncture direction, and puncture length are selected using a trained expert model (e.g., Figure 7 (As shown). Finally, the depth information of the puncture target point is obtained based on the depth image, and the coordinates of the puncture target point are converted into spatial coordinates. The algorithm ends, completing the localization of the puncture target point.

[0121] Continue to refer to Figure 1In step 102, the movement of the puncture mechanism is controlled according to the current position of the puncture needle and the position of the target puncture point.

[0122] In embodiments of the present invention, the movement of the puncture mechanism is controlled according to the current position of the puncture needle and the position of the target puncture point, specifically including the following steps:

[0123] ① Calculate the coordinates of the puncture needle in the global coordinate system using the forward kinematics equations.

[0124] ② Move the puncture needle to the target position (i.e., the position of the target puncture point).

[0125] ③ Perform the puncture: Based on the vascular depth and diameter information obtained from the ultrasound image, set an appropriate puncture distance so that the puncture needle can accurately enter the blood vessel and stop at the center of the blood vessel as much as possible;

[0126] ④ Adjust the angle of the puncture needle: Adjust the angle of the puncture needle to make the angle between the puncture needle and the blood vessel smaller, while keeping the position of the puncture needle tip unchanged.

[0127] ⑤ Push: Advance the puncture needle a suitable distance along the direction of the blood vessel.

[0128] In step 103, a color image of the puncture scene is acquired, and target detection and tracking are performed on the puncture needle and hand area to monitor the puncture process in real time.

[0129] In embodiments of the present invention, a color camera is used to monitor the entire puncture process, and the monitoring process is as follows: Figure 8 As shown, firstly, a color image of the puncture site is acquired in real time using a color camera. Then, a combination of YOLOv10 and Kalman filtering is used to detect and track the puncture needle and the back of the hand (or other puncture sites on the hand). Figure 9 As shown in the diagram, the positional relationship between the puncture needle and the back of the hand is calculated, and the puncture needle's normality is determined based on this relationship. Because the YOLOv10 target detection model employs a more efficient backbone network structure, optimized feature fusion strategy, and advanced loss function design, it improves detection accuracy while maintaining high-speed inference. In the application scenario of the automated puncture robot, the YOLOv10 Nano version of the target detection model was selected, which reduces model parameters and improves detection speed while ensuring detection accuracy.

[0130] In target tracking, this invention employs an object tracking method based on Kalman filters and the Hungarian algorithm, introducing an appearance model to enhance the data association process. Compared to traditional tracking methods, this method can better maintain target identity consistency over long time series, especially when the target is occluded or temporarily out of sight. Furthermore, by utilizing target features extracted using the YOLOv10 convolutional neural network, false matches can be effectively reduced, further improving tracking accuracy. Real-time monitoring of the puncture needle position through target detection and tracking prevents problems such as incorrect needle placement and accidental needle injury.

[0131] The steps for calculating positional relationships are as follows:

[0132] Get the current position and length of the puncture needle;

[0133] Get the current position of the back of the hand area;

[0134] Based on the current position and length of the puncture needle, and the current position of the hand area, determine the inclusion relationship between the puncture needle and the back of the hand area. If the puncture needle is included in the back of the hand area and the length of the puncture needle is less than the standard length (exposed length less than the original length), it is defined as puncture has begun; if the puncture needle is included in the back of the hand area and the length of the puncture needle is the standard length, it is defined as preparation for puncture; if the puncture needle does not intersect with the back of the hand area, it is defined as puncture has not begun.

[0135] In step 104, ultrasound equipment is used to perform three-dimensional reconstruction of subcutaneous blood vessels in the hand and to guide the puncture needle.

[0136] In embodiments of the present invention, the procedure for ultrasound-guided three-dimensional reconstruction of blood vessels and puncture needle navigation is as follows: Figure 10 As shown, the process includes connecting the ultrasound equipment, adjusting the ultrasound equipment parameters, and continuously acquiring cross-sectional and profile images; using the SegFormer neural network algorithm to segment the vascular region; before performing 3D reconstruction of the vascular images, first acquiring the contours and masks obtained from the segmented vascular region, arranging them according to the image acquisition order, and using 3D reconstruction tools (such as VTK software) to construct the 3D vascular structure; and performing target detection and tracking on the puncture needle (e.g., ...). Figure 11 As shown in the diagram, the green area represents the puncture needle, and the red area represents the vein. The system then calculates the real-time position, speed, and direction of the puncture needle, and uses the three-dimensional structural information of the blood vessel obtained from the ultrasound image (such as the depth and diameter of the blood vessel where the target puncture point is located) to navigate the puncture needle.

[0137] By combining the current frame position, speed, and direction of the puncture needle, the possible position of the puncture needle in the next frame can be calculated. This position information is then combined with the three-dimensional structure information of the blood vessel to predict the distance between the puncture needle and the vessel wall. Based on this distance, operational prompts are provided, thus achieving puncture needle navigation. For example, if the predicted puncture needle position is too close to the vessel wall, a prompt is given to stop moving the puncture needle (activating the puncture needle emergency stop mode); otherwise, the puncture needle can continue moving.

[0138] SegFormer is a simple and efficient transformer-based model for semantic segmentation. It does not rely on positional encoding and uses a hierarchical architecture for multi-scale feature representation. SegFormer integrates a lightweight MLP decoder with a transformer to create a multi-scale feature hierarchy, which improves both performance and efficiency.

[0139] In an embodiment of the present invention, the steps for selecting VTK software as a three-dimensional reconstruction tool for vascular three-dimensional reconstruction are as follows:

[0140] 1) Define the rendering window and interaction mode: Initialize the VTK interactive window;

[0141] 2) Define the reading interface: Define the image format and select the reading tool;

[0142] 3) Pre-settings and image reading: Define information such as image size, number of images, storage location, and image prefix;

[0143] 4) Reconstruction parameter settings: Set the (x, y, z) parameters, where x and y are the spacing in the horizontal direction of the blood vessel cross section, and z is the spacing in the length direction of the blood vessel cross section;

[0144] 5) Gaussian smoothing of reconstruction results: Beautifies the reconstructed three-dimensional vascular structure and removes noise;

[0145] 6) Contour and edge extraction calculation: Extract the contour of the three-dimensional blood vessel structure and calculate the length, diameter and other information of the three-dimensional blood vessel;

[0146] 7) Pipeline operation and visualization: Display of three-dimensional vascular structure.

[0147] In the embodiments of the present invention, YOLOv10 with Kalman filtering is used to perform target detection and tracking of the puncture needle under ultrasound, and to calculate parameters such as the real-time position, speed and direction of movement of the puncture needle.

[0148] The robotic automated puncture method provided by this invention utilizes an advanced target recommendation algorithm combining image imaging, image processing, and expert models to automatically identify puncture target points in the patient's hand region and provide the location coordinates of the puncture target points to the puncture needle, thereby improving recognition accuracy. A color camera monitors the entire robotic puncture process to prevent problems such as incorrect needle placement and accidental needle injury, enhancing the safety of automated puncture. Furthermore, ultrasound equipment and artificial intelligence algorithms are used for three-dimensional reconstruction of blood vessels in the hand region and needle navigation, further improving the accuracy and safety of the puncture operation.

[0149] In embodiments of the present invention, a puncture device is also provided. Figure 12 This is a schematic diagram of the puncture device according to an embodiment of the present invention, such as... Figure 12 As shown, the puncture device of the present invention includes a processor 1201, a memory 1202, and a computer program stored in the memory 1202 and executable on the processor 1201, wherein the computer program, when read and executed by the processor 1201, implements the steps in the robotic automatic puncture method as described above.

[0150] In embodiments of the present invention, a robot is also provided, including the puncture device described above. The robot employing the aforementioned puncture device improves target identification accuracy while effectively avoiding problems such as incorrect needle placement and accidental needle injury, thereby enhancing the safety of the robot's automated puncture process.

[0151] In embodiments of the present invention, a computer-readable storage medium is also provided, wherein a computer program is stored therein, wherein the computer program is configured to perform the steps in the robotic automated puncture method described above when it is run.

[0152] In this embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0153] It will be understood by those skilled in the art that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A lancing device, characterized by, The puncture device is used to perform the following method steps: From the depth image, color image and near-infrared image of the collected puncture area, the position of the target puncture target point is obtained; According to the current position of the puncture needle and the position of the target puncture target point, the movement of the puncture mechanism is controlled; Through the color image of the puncture picture, the target detection and tracking of the puncture needle and the puncture area are performed, and the puncture process is monitored in real time; The subcutaneous blood vessels of the puncture area are three-dimensionally reconstructed and the puncture needle is navigated; The step of obtaining the position of the target puncture target point from the depth image, color image and near-infrared image of the collected puncture area further comprises: Based on the depth image, color image and near-infrared image, the registered near-infrared image is obtained through image registration; The registered near-infrared image is processed, and an expert model trained based on a regression model or a deep learning model is used to screen out the target puncture target point to determine the position of the target puncture target point from the subcutaneous blood vessels of the puncture area.

2. The lancing device of claim 1, wherein, The step of obtaining the position of the target puncture target point from the depth image, color image and near-infrared image further comprises: The depth image and color image of the puncture area are obtained by a depth camera, and the near-infrared image of the puncture area is obtained by a near-infrared camera; Feature point extraction and feature point matching are performed on the color image and the near-infrared image, a homography transformation matrix is calculated according to the matching result, and a homography transformed near-infrared image is obtained; An image enhancement algorithm is used to enhance the homography transformed near-infrared image; The blood vessel region is obtained by performing blood vessel segmentation on the enhanced near-infrared image; The blood vessel centerline is extracted from the blood vessel region; The blood vessel centerline is segmented into multiple segments, and each segmentation point is a candidate target point; An expert model trained based on a regression model or a deep learning model is used to screen out the target puncture target point from the candidate target points; The spatial coordinates of the target puncture target point are calculated. The step of performing feature point extraction and feature point matching on the color image and the near-infrared image, calculating a homography transformation matrix according to the matching result, and obtaining a homography transformed near-infrared image further comprises:

3. The lancing device of claim 2, wherein, First feature points are extracted from the grayscale image of the color image, and second feature points are extracted from the near-infrared image, Feature matching is performed on the first feature points and the second feature points to obtain matched feature point pairs; A homography transformation matrix is calculated according to the matched feature point pairs; Based on the homography transformation matrix, a homography transformed near-infrared image is obtained. The step of using an expert model trained based on a regression model to screen out the target puncture target point from the candidate target points further comprises:

4. The lancing device of claim 2, wherein, 1) Collect target point data; the target point data includes the straightness, diameter and puncturable length of the blood vessel where the target point is located, which is scored by experienced puncture workers; 2) Establish a regression model, the input of the model is the straightness, diameter and puncturable length of the blood vessel where the target point is located, and the output of the model is the target point score; 3) Use the collected target point data to train the regression model to obtain the weight values of the straightness, diameter and puncturable length of the blood vessel where the target point is located; ​ 4) performing regression analysis on the weights calculated in step 3) for multiple candidate target points in the same image to obtain a score for each candidate target point; 5) ranking the candidate target points according to the scores; 6) recommending the candidate target point with the highest score as the target puncture target point.

5. The lancing device of claim 1, wherein, The step of controlling the movement of the puncture mechanism according to the current position of the puncture needle and the position of the target puncture target point further comprises: calculating the coordinates of the puncture needle in the global coordinate system; moving the puncture needle to the target puncture position according to the current coordinates of the puncture needle in the global coordinate system and the spatial coordinates of the target puncture target point; obtaining the depth and diameter of the blood vessel where the target puncture target point is located in combination with the ultrasound image to perform puncture navigation; adjusting the angle of the puncture needle; pushing the puncture needle to complete the puncture.

6. The lancing device of claim 1, wherein, The step of performing target detection and tracking on the puncture needle and the puncture area through the color image of the puncture picture to monitor the puncture process in real time further comprises: acquiring the color image of the puncture picture in real time through a color camera, wherein the puncture picture includes the puncture needle and the puncture area to be punctured; extracting the features of the puncture needle and the puncture area from the color image by combining YOLOv10 and Kalman filtering; obtaining the inclusion relationship between the puncture needle and the puncture area according to the current position and length of the puncture needle and the current position of the puncture area; obtaining the current puncture state according to the inclusion relationship and the current exposed length of the puncture needle.

7. The lancing device of claim 1, wherein, The step of performing three-dimensional reconstruction of the subcutaneous blood vessels of the puncture area and puncture needle navigation further comprises: acquiring continuous cross-sectional images and cross-sectional images through an ultrasound device; segmenting the blood vessel area from the images using a SegFormer neural network algorithm; obtaining the contour and mask of the blood vessel area and arranging them according to the image acquisition sequence to perform three-dimensional reconstruction of the blood vessels and obtain the three-dimensional blood vessel structure; performing target detection and tracking on the puncture needle by combining YOLOv10 and Kalman filtering to obtain the real-time position, movement speed, and movement direction of the puncture needle; judging the positional relationship between the puncture needle and the blood vessel wall according to the real-time position, movement speed, and movement direction of the puncture needle, and then performing puncture needle navigation according to the positional relationship.

8. The lancing device of claim 1, wherein, A computer program stored in the memory and executable on the processor to implement the method steps.

9. A robot, characterized in that The puncture device of any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program which is loaded and executed by the processor to control the puncture device of any one of claims 1 to 8 to perform the method steps.

Images