A venipuncture target positioning method based on a depth camera and a near-infrared camera

Abstract

A kind of venipuncture target positioning method based on depth camera and near-infrared camera, comprising: 1) the depth image and color image of hand are acquired by depth camera, and the near-infrared image of hand is acquired by near-infrared camera;2) the transformation matrix of the color image and the near-infrared image is calculated, and the transformed near-infrared image is obtained;3) the target puncture target point is obtained using the transformed near-infrared image;4) the spatial coordinates of the target puncture target point are calculated.The depth information of puncture target point is aligned with depth camera by near-infrared camera, so that the same precision as depth camera is obtained, and the target point is recommended by combining expert model, the best puncture target point can be quickly positioned, and the spatial position information of puncture target point is obtained, the target point recognition precision is improved, and the positioning efficiency is high.

Description

Technical Field

[0001] This invention relates to the field of vein puncture target localization technology, and in particular to a method for vein puncture target localization based on a depth camera and a near-infrared camera. Background Technology

[0002] Currently, intravenous puncture is typically performed manually by nurses. During the procedure, healthcare professionals select prominent and straight veins, such as the back of the hand, forearm, or elbow, to minimize patient discomfort and complications, and perform disinfection and needle insertion. To improve puncture accuracy and reduce patient discomfort, assisted puncture devices have been widely used. These devices help healthcare professionals better identify vascular areas and recommend suitable puncture points and directions. There are many existing vascular recognition algorithms for assisted puncture devices. One common approach is to use ultrasound probes to identify veins, which can obtain morphological information of the vessels. However, the pressure from the ultrasound probe may alter the three-dimensional information of the vessels, leading to recognition errors. Another approach is based on neural networks and image processing technology for puncture target point identification. This method typically calculates on two-dimensional images of the vessels, ignoring three-dimensional information, resulting in spatial errors in the calculated puncture target point. Therefore, puncture accuracy still needs improvement. Summary of the Invention

[0003] To address the shortcomings of existing technologies, the present invention aims to provide a method for locating vein puncture targets based on depth cameras and near-infrared cameras, which improves the accuracy of target identification and quickly locates the optimal puncture target.

[0004] To achieve the above objectives, the present invention provides a method for locating vein puncture target points based on a depth camera and a near-infrared camera, comprising:

[0005] 1) Acquire depth and color images of the hand using a depth camera, and acquire near-infrared images of the hand using a near-infrared camera;

[0006] 2) Calculate the transformation matrix between the color image and the near-infrared image, and obtain the transformed near-infrared image;

[0007] 3) Obtain the target puncture point using the transformed near-infrared image;

[0008] 4) Calculate the spatial coordinates of the target puncture point.

[0009] Furthermore, step 2) also includes:

[0010] 21) Extract feature points from the grayscale and near-infrared images of the color image, respectively;

[0011] Extract the descriptor corresponding to each feature point;

[0012] 22) By matching the descriptors of feature points using a matching algorithm, we can find matching feature point pairs in two images;

[0013] 23) Calculate the homography transformation matrix based on the matching results;

[0014] 24) Using the homography transformation matrix, obtain the near-infrared image after homography transformation.

[0015] Furthermore, step 3) also includes:

[0016] 31) Image enhancement algorithms are used to enhance the near-infrared image after homography transformation;

[0017] 32) Perform vessel segmentation on the enhanced near-infrared image to obtain the vessel region;

[0018] 33) Extract the vascular centerline from the vascular region;

[0019] 34) Divide the central line of the blood vessel into multiple segments, and use each segmentation point as a candidate target point;

[0020] 35) Use expert models to screen out target puncture points from candidate targets.

[0021] Furthermore, step 31) also includes:

[0022] The image is divided into multiple image blocks;

[0023] For each image patch, its histogram is calculated to obtain the number of pixels at each gray level;

[0024] Calculate the cumulative distribution; limit contrast when calculating the cumulative distribution;

[0025] The trimmed histogram is equalized, and the equalized gray value is calculated based on the cumulative distribution probability of each gray level.

[0026] Traverse the image blocks and perform bilinear interpolation between blocks to smoothly transition the contrast changes between different blocks;

[0027] The processed image is then blended with the original image to obtain the final enhanced image.

[0028] Furthermore, step 35) also includes:

[0029] 51) Collect target data; the target data includes: the straightness, diameter, and puncture length of the blood vessel where the target is located, which are scored by experienced puncture technicians;

[0030] 52) Establish a regression model. The input to the model is the straightness, diameter, and puncture length of the blood vessel where the target is located. The output of the model is the target score.

[0031] 53) Using the collected target data, a regression model was trained to obtain the weight values ​​of the straightness, diameter, and puncture length of the blood vessel where the target is located.

[0032] 54) For multiple candidate target points in the same image, perform regression analysis according to the weights calculated in step 53) to obtain the score of each candidate target point;

[0033] 55) Sort candidate targets by score;

[0034] 56) The candidate target with the highest recommended score is the target puncture target.

[0035] Furthermore, step 4) also includes:

[0036] 61) Obtain the depth value of the corresponding depth image based on the pixel coordinates of the target puncture point;

[0037] 62) Using the depth value and the intrinsic parameters of the depth camera, the three-dimensional spatial coordinates of the target puncture point are obtained.

[0038] To achieve the above objectives, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor is configured to execute the computer program stored in the memory to implement the method for locating vein puncture targets based on a depth camera and a near-infrared camera as described above.

[0039] To achieve the above objectives, the present invention also provides a computer-readable storage medium storing a computer program, which is loaded and executed by a processor to implement the method for locating vein puncture targets based on a depth camera and a near-infrared camera as described above.

[0040] The method for locating vein puncture targets based on depth cameras and near-infrared cameras provided by this invention has the following advantages compared with existing technologies:

[0041] By aligning the near-infrared camera with the depth camera, the depth information of the puncture target point can be made to have the same accuracy as that of the depth camera. Combined with the expert model, the target point can be recommended based on the aligned near-infrared image. This allows for the rapid location of the optimal puncture target point and the acquisition of its spatial location information, thereby improving the accuracy of target point identification and increasing the positioning efficiency.

[0042] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. Attached Figure Description

[0043] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0044] Figure 1 This is a flowchart of a method for locating a vein puncture target based on a depth camera and a near-infrared camera according to an embodiment of the present invention.

[0045] Figure 2 This is an exemplary algorithm flowchart according to an embodiment of the present invention;

[0046] Figure 3 This is an image of hand enhancement according to an embodiment of the present invention;

[0047] Figure 4 This is a schematic diagram of the blood vessel segmentation result of a hand image according to an embodiment of the present invention;

[0048] Figure 5 This is a schematic diagram of the blood vessel centerline extraction results according to an embodiment of the present invention;

[0049] Figure 6 This is a schematic diagram of the blood vessel centerline segmentation result according to an embodiment of the present invention;

[0050] Figure 7 This is a schematic diagram of the puncture target point according to an embodiment of the present invention;

[0051] Figure 8 This is a schematic diagram of an electronic device structure according to an embodiment of the present invention. Detailed Implementation

[0052] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0053] Embodiments of the present invention will now be described in more detail with reference to the accompanying drawings. While some embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the invention. It should be understood that the accompanying drawings and embodiments are for illustrative purposes only and are not intended to limit the scope of protection of the invention.

[0054] It should be understood that the concepts of "first" and "second" mentioned in this invention are used only to distinguish different data or units, and are not intended to limit the order of functions performed by these data or units or their interdependencies. These terms are used merely to distinguish one feature from another. For example, without departing from the scope of the exemplary embodiments, a first feature may be referred to as a second feature, and similarly, a second feature may be referred to as a first feature.

[0055] It should be noted that the terms "one" and "multiple" used in this invention are illustrative rather than restrictive. Those skilled in the art should understand that, unless explicitly stated otherwise in the context, they should be understood as "one or more". "Multiple" should be understood as two or more.

[0056] Example 1

[0057] In an embodiment of the present invention, a method for locating vein puncture target points based on a depth camera and a near-infrared camera is provided, comprising the following steps: 1) acquiring depth and color images of the hand using a depth camera, and acquiring near-infrared images of the hand using a near-infrared camera; 2) extracting feature points from the grayscale image of the color image and the near-infrared image respectively, and performing feature point matching, calculating the homography transformation matrix based on the matching results, and obtaining the near-infrared image after homography transformation; 3) enhancing the near-infrared image after homography transformation using an image enhancement algorithm; 4) segmenting blood vessels in the enhanced near-infrared image to obtain blood vessel regions; 5) extracting the blood vessel centerline from the blood vessel regions; 6) dividing the blood vessel centerline into multiple segments, with each segmentation point as a candidate target point; 7) using an expert model to select the target puncture point from the candidate target points; 8) calculating the spatial coordinates of the target puncture point.

[0058] Figure 1 The flowchart below shows a method for locating vein puncture target points based on a depth camera and a near-infrared camera according to an embodiment of the present invention. Figure 1 The method of the present invention will be described in further detail below.

[0059] First, in step 101, hand images are acquired using a depth camera and a near-infrared camera respectively: the depth camera acquires depth and color images of the hand (such as the back of the hand, forearm, elbow, etc.), and the near-infrared camera acquires near-infrared images of the hand.

[0060] In embodiments of the present invention, since the sizes of the images are inconsistent, it is necessary to perform soft alignment between the depth image and the color image (while keeping the image content unchanged, the position and angle of the image are adjusted in a smooth transition manner, thereby reducing obvious splicing marks and visual abruptness).

[0061] In step 102, homography transformation is performed on the color image and the near-infrared image to obtain the homography-transformed near-infrared image. In this step, a first feature point is extracted from the color image, and a second feature point is extracted from the near-infrared image. Feature matching is performed on the first and second feature points to obtain matched feature point pairs. Based on the matched feature point pairs, homography transformation is applied to obtain the homography-transformed near-infrared image.

[0062] Homograph transformation is a plane-to-plane projection transformation that maps points on one plane to another using a 3×3 matrix. This transformation preserves collinearity, meaning that straight lines remain straight lines after the transformation. Homograph transformation is widely used in computer vision and image processing, especially in image registration, stitching, and camera pose estimation.

[0063] In embodiments of the present invention, the specific steps of homography transformation are as follows:

[0064] 21) Use feature point extraction algorithms (such as SIFT, ORB, SURF, etc.) to extract feature points from the image;

[0065] 22) Extract the descriptor corresponding to each feature point;

[0066] 23) Find matching feature point pairs between two images by matching feature point descriptors using a matching algorithm (such as KNN nearest neighbor algorithm);

[0067] 24) Use the RANSAC algorithm to remove incorrect matching feature point pairs, thereby improving the accuracy and robustness of matching;

[0068] 25) Solve the system of equations and calculate the Homograph (homography transformation) matrix: Solve the system of linear equations by matching at least 4 pairs of feature points and calculate the homograph transformation matrix;

[0069] 26) Perform homography transformation on the image.

[0070] Through the above steps, the near-infrared camera and the depth camera are aligned by feature matching, and the position of the puncture target is mapped to the coordinate system of the depth camera, thereby obtaining the depth information of the puncture target. This depth information has the same accuracy as the depth camera (ignoring matching error and camera calibration error).

[0071] In step 103, the near-infrared image after homography transformation is enhanced. An image enhancement algorithm is used to highlight the vascular features in the hand image.

[0072] In embodiments of the present invention, a contrast-limited adaptive histogram equalization method is used to enhance the near-infrared image after homography transformation, thereby improving the contrast of the vascular region in the image and facilitating better vascular segmentation. Specific steps include:

[0073] 31) Divide the entire image into multiple smaller image blocks (e.g., 8×8 image blocks). This step is to process local regions of the image independently to better adapt to the local characteristics of the image;

[0074] 32) For each image patch, calculate its histogram, that is, count the number of pixels at each gray level;

[0075] 33) Calculate the cumulative distribution. When calculating the cumulative distribution, limit the contrast to prevent the histogram from becoming too steep. For example, form a set B of the values ​​in the histogram that are above a certain threshold a. Then, change each element b in B to a. Sum the values ​​above a to get c. Divide c by 256 (where 256 refers to the gray level, i.e., gray level 0 to 255) to get d. Add d to each gray level.

[0076] 34) Perform equalization processing on the trimmed histogram, that is, calculate the equalized gray value based on the cumulative distribution probability of each gray level;

[0077] 35) Traverse the image blocks and perform bilinear interpolation between blocks to smoothly transition the contrast changes between different blocks;

[0078] 36) The image after smoothing the transition is blended with the original image to obtain the final enhanced image. This step aims to preserve some details and features of the original image while increasing its contrast and sharpness. Through the above steps, the contrast-limited adaptive histogram equalization method can enhance image contrast while suppressing noise, making it particularly suitable for applications requiring high contrast, such as medical imaging.

[0079] In step 104, blood vessel segmentation is performed on the enhanced hand image to obtain the blood vessel region.

[0080] In medical image processing, vascular segmentation is a crucial step in many clinical diagnoses and treatments. Since blood vessels typically appear as elongated, low-contrast structures in images, vascular segmentation is a key step in puncture target selection. Supervised image segmentation based on deep learning methods can be used to segment blood vessels, extracting vascular regions from hand images.

[0081] In step 105, the vascular centerline is extracted from the vascular region. In embodiments of the present invention, vascular centerline extraction algorithms such as those based on weak texture enhancement, curvature, or the topological relationship of vascular branches can be used to extract the vascular centerline from the vascular region.

[0082] In step 106, the centerline of the blood vessel is segmented to obtain multiple blood vessel segments.

[0083] In step 107, an expert model is used to screen puncture target points from each blood vessel segment. Based on the experience of senior puncture workers, suitable vascular puncture rules are summarized, and the properties of puncture target points are determined. The characteristics of puncture target points are selected in descending order of importance as follows: vascular fullness, high vascular straightness, large vascular diameter, and large punctureable length. According to the vascular puncture rules, a linear regression model is established to calculate the feature weights of candidate target points. After weighted summation, a score is obtained for each potential target point, and the target point with the highest score is recommended.

[0084] The algorithm steps for selecting puncture target points using an expert model are as follows:

[0085] 71) Collect candidate target data, including the straightness, diameter, and puncture length of the blood vessel where the candidate target is located, and have it scored by experienced puncture technicians.

[0086] 72) Establish a regression model. The input to the model is the straightness, diameter, and puncture length of the blood vessel where each candidate target point is located. The output of the model is the target point score.

[0087] 73) Using the collected candidate target data, the regression model is trained to obtain the weight values ​​of the straightness, diameter, and puncture length of the blood vessel where the candidate target is located.

[0088] 74) For multiple candidate target points in an image, perform regression analysis according to the weights calculated in step 73, and calculate the score of each candidate target point.

[0089] 75) Sort the candidate targets according to their scores;

[0090] 76) The candidate target with the highest recommended score is the final target puncture point, and its puncture direction and puncture length are obtained. The puncture direction is determined based on the angle between the vessel segment and the horizontal direction.

[0091] To further improve the accuracy of target identification, a deep learning-based line segment recognition method can be used to extract the trend features of blood vessels.

[0092] In step 108, the coordinates of the puncture target point in space are calculated.

[0093] In embodiments of the present invention, the depth value of the corresponding depth image is obtained based on the pixel coordinates of the target puncture point; the three-dimensional spatial coordinates of the target puncture point are obtained using the depth value and the intrinsic parameters of the depth camera. This step converts the image coordinates of the puncture point into spatial coordinates (X... C ,Y C ZC The specific calculation steps are as follows:

[0094] 81) Obtain the intrinsic parameters f of the depth camera x ,f y ,u0,v0; where, f x ,f y These are the calibration parameters for the focal length; u0 and v0 are the coordinates of a reference point on the image, usually with the center of the image as the reference point.

[0095] 82) Obtain the two-dimensional coordinates of the puncture target point on the near-infrared image, index them onto the depth image, and thus obtain the depth information Z. C Z C The spatial depth of the puncture target;

[0096] 83) Calculate the spatial coordinates X of the puncture target point. C The calculation formula is: X C =(u-u0)*Z C / f x Where u is the x-coordinate of the puncture target point in the near-infrared image, and Z... C The spatial depth of the puncture target;

[0097] 84) Calculate the spatial coordinates Y of the puncture target point. C The calculation formula is: Y C =(v-v0)*Z C / f y Where v is the y-coordinate of the puncture target point in the near-infrared image, and Z... C The depth of the puncture target in space.

[0098] This invention provides a method for locating venipuncture target points based on depth and near-infrared cameras. It processes images of the patient's hand region and aligns the near-infrared camera with the depth camera through feature matching, ensuring the depth information of the puncture target point has the same accuracy as the depth camera. Combining image processing and expert models for target point recommendation, this method offers high speed and can quickly acquire puncture target points in three-dimensional space that conform to the automatic puncture rules of assisted puncture devices (such as automated puncture robots). This improves the success rate and accuracy of assisted puncture devices, shortens operation time, reduces the possibility of secondary punctures, and enhances the efficiency of medical staff. It also reduces the likelihood of patients experiencing local infection, systemic infection, hematoma, vascular rupture, nerve damage, etc., due to puncture, thereby improving the quality of medical care.

[0099] Figure 2 This is an exemplary algorithm flowchart according to an embodiment of the present invention. Figures 3 to 7 This is a schematic diagram of the hand image processing result according to an example of the present invention, which will be described below in conjunction with... Figures 2 to 7 The technical effects of the present invention are explained.

[0100] like Figure 2 As shown, firstly, a depth image and a color image are obtained using a depth camera, and a near-infrared image is obtained using a near-infrared camera. The depth image is used to subsequently determine the spatial coordinates of the target points, while the color image is used for feature matching with the near-infrared image. The feature matching process includes: processing the color image into grayscale to obtain a grayscale image, and extracting feature points from the grayscale image using the SIFT algorithm; similarly, extracting feature points from the near-infrared image using the SIFT algorithm; performing feature point matching using the KNN algorithm on the two sets of feature points, thereby calculating the homography matrix and obtaining the homography-transformed near-infrared image using the homography matrix.

[0101] The near-infrared image after homography transformation is enhanced by an image enhancement algorithm to highlight vascular features (such as...). Figure 3 (As shown). The enhanced hand image is segmented to identify blood vessels, and the presence of vascular regions is determined. If no vascular region exists, the processing of the current near-infrared image ends; if a vascular region exists (e.g., ...), the processing continues. Figure 4 The right side of the image shows the segmented vascular region. The vascular centerline is then extracted from the vascular region using a vascular centerline extraction algorithm (e.g.,...). Figure 5 As shown), after segmentation by the blood vessel centerline segmentation algorithm (as shown) Figure 6 As shown), the optimal puncture target point, puncture direction, and puncture length are selected using a trained expert model (e.g., Figure 7 (As shown). Finally, the depth information of the puncture target point is obtained based on the depth image, and the coordinates of the puncture target point are converted into spatial coordinates. The algorithm ends, completing the localization of the puncture target point.

[0102] In embodiments of the present invention, an electronic device is also provided. Figure 8 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention, such as... Figure 8 As shown, the electronic device of the present invention includes a processor 801, a memory 802, and a computer program stored in the memory 802 and executable on the processor 801. When the computer program is read and executed by the processor 801, it implements the steps in the method for locating vein puncture targets based on a depth camera and a near-infrared camera as described above.

[0103] In embodiments of the present invention, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer program, wherein the computer program is configured to execute the steps in the method for locating vein puncture targets based on a depth camera and a near-infrared camera as described above when it is run.

[0104] In this embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0105] It will be understood by those skilled in the art that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for locating a venipuncture target based on a depth camera and a near-infrared camera, characterized in that, The method comprises the following steps: 1) obtaining a depth image and a color image of a hand through a depth camera, and obtaining a near-infrared image of the hand through a near-infrared camera; 2) calculating a transformation matrix of the color image and the near-infrared image, and obtaining a transformed near-infrared image; 3) obtaining a target puncture target point by using the transformed near-infrared image; 4) calculating a spatial coordinate of the target puncture target point; the step 2) further comprises: 21) respectively extracting feature points from a grayscale image of the color image and the near-infrared image; extracting a descriptor corresponding to each feature point; 22) matching the descriptors of the feature points by using a matching algorithm to find a matching feature point pair in the two images; 23) calculating a homography transformation matrix according to the matching result; 24) obtaining a homography transformed near-infrared image by using the homography transformation matrix.

2. The method of claim 1, wherein, The step 3) further comprises: 31) enhancing the homography transformed near-infrared image by using an image enhancement algorithm; 32) segmenting a blood vessel region from the enhanced near-infrared image; 33) extracting a blood vessel centerline from the blood vessel region; 34) dividing the blood vessel centerline into multiple segments to obtain candidate target points; 35) screening the target puncture target point from the candidate target points by using an expert model.

3. The method of claim 2, wherein the method further comprises: The step 31) further comprises: dividing the image into multiple image blocks; for each image block, calculating a histogram to obtain the number of pixels of each grayscale level; calculating a cumulative distribution; limiting the contrast when calculating the cumulative distribution; performing equalization processing on the pruned histogram; calculating an equalized grayscale value according to the cumulative distribution probability of each grayscale level; traversing the image blocks to perform bilinear interpolation between the blocks to smooth the contrast change between different blocks; and mixing the processed image with the original image to obtain a final enhanced image. 4.The method of claim 2, wherein, The step 35) further comprises: 51) collecting target point data; the target point data comprises straightness, diameter and puncturable length of a blood vessel where the target point is located, and the straightness, diameter and puncturable length are scored by a senior puncture worker; 52) establishing a regression model, wherein the input of the model is the straightness, diameter and puncturable length of the blood vessel where the target point is located, and the output of the model is a target point score; 53) training the regression model by using the collected target point data to obtain weight values of the straightness, diameter and puncturable length of the blood vessel where the target point is located; 54) performing regression analysis on multiple candidate target points in the same image according to the weight values calculated in the step 53) to obtain a score of each candidate target point; 55) sorting the candidate target points according to the scores; and 56) recommending the candidate target point with the highest score as the target puncture target point.

5. The method of claim 1, wherein, The step 4) further comprises: 61) obtaining a depth value of a corresponding depth image according to a pixel coordinate of the target puncture target point; and 62) obtaining a three-dimensional spatial coordinate of the target puncture target point by using the depth value and an internal parameter of the depth camera.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor is configured to execute a computer program stored in the memory to implement the method for locating a venipuncture target point based on a depth camera and a near-infrared camera according to any one of claims 1-5.

7. A computer-readable storage medium, characterized in that, The storage medium has a computer program stored therein, and the computer program is loaded and executed by the processor to implement the steps of the vein puncture target positioning method based on the depth camera and the near-infrared camera according to any one of claims 1-5.

Images