3D Imaging Method and System for Superficial Venous Vessels on the Body Surface Based on 3D Mapping

Abstract

The present invention provides a 3D imaging method and system for surface vein blood vessels based on three-dimensional mapping, belonging to the technical field of 3D imaging of surface vein blood vessels. The method includes: emitting infrared light to the surface of the part to be examined; obtaining an infrared feature map of the vein blood vessels; performing preprocessing filtering on the infrared feature map of the vein blood vessels, and using a deep learning image feature extraction and segmentation algorithm to perform feature extraction and filtering to obtain a 2D feature map of the surface vein blood vessels; obtaining the extension feature and thickness feature of each section of blood vessel, and performing three-dimensional reconstruction to obtain a 3D model of the surface vein blood vessels; performing real-time video capture on the part to be examined to obtain a visible light image of the part to be examined; identifying the features of the part to be examined to obtain a 3D model of the part to be examined; performing three-dimensional mapping on the 3D model of the surface vein blood vessels and the 3D model of the part to be examined to become one model, and obtaining a 3D imaging map of the surface vein blood vessels of the part to be examined. The present invention has the characteristics of real-time imaging, 3D imaging, and convenient operation.

Description

Technical Field

[0001] The present invention relates to the technical field of 3D imaging of body surface veins, and in particular to a method and system for 3D imaging of body surface veins based on three-dimensional mapping. Background Art

[0002] At present, the applications of surface venous imaging in the medical field include vascular reconstruction surgery design, varicose vein diagnosis and treatment, venipuncture treatment, venipuncture treatment, and venous thromboembolism (VTE) prediction.

[0003] Conventional venous imaging applications are generally fixed in locations and are large in size. Market research reveals the following market gaps and trends:

[0004] 1. Wartime Disaster Area Rescue: In large-scale disasters or wartime situations, medical rescue typically utilizes a tiered treatment process, including on-site rescue, early treatment, and specialized care. In these emergency situations, the treatment environment is often limited by poor equipment and a lack of highly skilled medical personnel. Therefore, portable, easy-to-use, highly visible, and non-invasive surface venous imaging tools are needed.

[0005] 2. Miniaturization of medical devices: The trend toward miniaturization of medical devices is growing to address the high cost, complex operation, and limited usage scenarios of large-scale equipment such as MRI and CT. For example, during superficial vein reconstruction surgery, surgeons cannot view the current status of the blood vessels in real time due to space, time, and personnel constraints. Therefore, a portable, easy-to-use, and highly visual superficial vein imaging tool is needed.

[0006] 3. Online healthcare trends: With the widespread adoption of the internet, online healthcare services have garnered widespread attention, placing particular emphasis on the visualization and qualitative analysis of medical data. If medical devices can connect to the cloud and upload patient medical data in real time, multiple doctors can simultaneously review the same data, enabling collaborative online diagnosis and treatment. Requirements for this surface venous imaging system include accurate qualitative assessments, highly visualized results, and support for cloud-based data upload.

[0007] Therefore, according to the requirements of the above scenarios, a surface venous 3D imaging system is needed that can perform real-time 3D imaging, is compact, easy to operate, harmless, non-invasive, sterile, can operate independently, has a certain endurance, and supports uploading highly visual scan data to the cloud.

[0008] Currently, commonly used surface venous imaging products include laser vascular imagers, near-infrared vascular imagers, CT angiography, color Doppler ultrasound, optical coherence tomography (OCT), etc. The following is an analysis:

[0009] Laser angiograph: Through Class II laser irradiation technology, the laser angiograph can present real-time red vascular images on the skin surface. Although it is convenient to operate and has a high degree of visualization, the display range is small, and it is impossible to obtain a complete vascular network. At the same time, the Class II laser used in the device has potential harm, cannot achieve 3D imaging, and also lacks support for modern medical data management and telemedicine needs.

[0010] Near-infrared vascular DLP projector: Through near-infrared light and DLP technology, it projects vascular images onto the skin surface in real time, with clear results and simple operation. However, like the laser angiograph, it can only provide 2D images, cannot achieve 3D imaging or obtain complete vascular network information. In addition, the device only has one display method and does not support data export and cloud upload, which limits the in-depth analysis and sharing of data.

[0011] CT angiography: It is a technology that generates images of the vascular system through computed tomography (CT) and intravenous injection of contrast agents. It can obtain complete 3D vascular network information at one time, with excellent visual effects. However, this technology is complex to operate, requires professional personnel for operation and training, and because it requires injection of contrast agents, it has a certain degree of invasiveness. In addition, the inability of CT angiography to achieve real-time imaging is also a major limitation.

[0012] Color Doppler ultrasound examination: It shows the vascular structure by sending ultrasonic pulses and receiving reflected signals. Although it can achieve real-time imaging and obtain complete vascular network information, it cannot perform 3D imaging, and the degree of visualization is average. Operating this device requires professional training, and because it requires the application of coupling agent, it has a certain degree of invasiveness. In addition, the device is large and not easy to carry, and it depends on external power supply, which limits its application in special environments.

[0013] Optical coherence tomography OCT: It is a technology that uses the principle of optical interference to measure the reflection properties of light to obtain microscopic structural images of tissues. It can achieve real-time 3D imaging, but the operation requires professional medical staff for training. On the other hand, OCT cannot obtain complete vascular network information during a single scan, and its degree of visualization is low. In addition, high-end OCT devices are often large, heavy, not easy to move, and usually need to be connected to a power supply, which limits their application in power-free environments. Summary of the Invention

[0014] The present invention provides a method and system for 3D imaging of superficial venous blood vessels based on three-dimensional mapping, which has the characteristics of real-time imaging, 3D imaging, and easy operation.

[0015] This specification provides a method for 3D imaging of superficial venous blood vessels based on three-dimensional mapping, including:

[0016] Emitting infrared light to the surface of the examined part;

[0017] Obtain an infrared characteristic map of venous blood vessels obtained from the infrared light reflected by the surface of the inspected part;

[0018] Preprocess and filter the infrared characteristic map of the venous blood vessels, and use a deep learning image feature extraction and segmentation algorithm for feature extraction and filtering to obtain a 2D characteristic map of the surface venous blood vessels;

[0019] Use a vascular mapping analysis algorithm to process the 2D characteristic map of the surface venous blood vessels, obtain the extension characteristics and thickness characteristics of each segment of blood vessels, and perform three-dimensional reconstruction to obtain a 3D model of the surface venous blood vessels;

[0020] Perform real-time video capture on the inspected part to obtain a visible light image of the inspected part;

[0021] Based on the visible light image of the inspected part, use the YOLOv5 object recognition algorithm to identify the characteristics of the inspected part and obtain a 3D model of the inspected part;

[0022] Perform three-dimensional mapping on the 3D model of the surface venous blood vessels and the 3D model of the inspected part to become one model, and obtain a 3D imaging map of the surface venous blood vessels of the inspected part.

[0023] In some embodiments of this specification, the preprocessing filter uses median filtering to reduce pixel noise by calculating the median of pixel values in the surrounding area of the pixel and can preserve the edge information of the image.

[0024] In some embodiments of this specification, the deep learning image feature extraction and segmentation algorithm uses a U-Net network with a symmetric encoder-decoder structure.

[0025] In some embodiments of this specification, the specific process of preprocessing and filtering the infrared characteristic map of the venous blood vessels and using a deep learning image feature extraction and segmentation algorithm for feature extraction and filtering to obtain a 2D characteristic map of the surface venous blood vessels is as follows:

[0026] Input the infrared characteristic map of the venous blood vessels, use a 3x3 convolutional kernel to step along the Z-shaped path of each pixel on the infrared characteristic map of the venous blood vessels, with a step size of 1 and padding of 0, calculate the gray values of the 8 pixel points around each pixel and sort them, and take the median as the pixel gray value at this position of the output image;

[0027] Then the filtered image enters the encoder of the U-Net. First, perform a convolution operation to extract the features of the image. The convolution structure uniformly uses a 3x3 convolutional kernel, k = 3, padding is p = 0, and the step size is s = 1. According to the formula for calculating the output features of the convolutional layer as shown below, n out is the output of the convolutional layer, n inis the input of the convolutional layer, p is the padding, k is the convolutional kernel size, and s is the stride;

[0028]

[0029] Get: n out = n in - 2;

[0030] After convolution, ReLU is used as the activation function to perform non - linear mapping on the convolutional output. The ReLU function is:

[0031] f(x)= max(x, 0);

[0032] After each layer undergoes 2 convolutional operations, it enters the next layer of the encoder through a max - pooling operation. The kernel size of each pooling layer is k = 2, the padding is p = 0, and the stride is s = 2. Get:

[0033] n out = n in / 2;

[0034] After 5 layers of convolution and 4 max - pooling operations, a feature map of 32x22x1024 is obtained, and this feature map is fed into the U - Net decoder;

[0035] The original size of the feature map is restored through the U - Net decoder. This process consists of convolution, upsampling, and skip - connection structures. First, the feature map is upsampled by 2x2, and the formula is:

[0036] n out = s(n in - 1)- 2p + k;

[0037] A feature map of 64x44x512 is obtained. A skip - connection operation is performed on this feature map. The feature map of the corresponding mirror layer is cropped and stitched together with the feature map from the upper layer to form a feature with more channels, that is, stitched with the upsampled feature map from the fifth layer of the U - Net encoder to generate a feature map of size 64x44x1024. Convolution and re - activation operations are performed on this feature map, and this operation is repeated 4 times. A feature map of 452x292x1 is obtained in the first layer, that is, a 2D map of the body surface vein blood vessel features is obtained.

[0038] In some embodiments of this specification, the specific process of using the blood vessel mapping analysis algorithm to process the 2D feature map of the body surface vein blood vessels, obtain the extension features and thickness features of each blood vessel segment, and perform three - dimensional reconstruction to obtain the 3D model of the body surface vein blood vessels is as follows:

[0039] Use Sobel filtering to perform edge detection and extraction on the 2D feature map of the body surface vein blood vessels to obtain a binary feature map with obvious edges;

[0040] Skeletonize the binary feature map and use the Zhang-Suen algorithm to extract the central axis, obtaining the central axis map of the venous blood vessels;

[0041] Overlap the binary feature map and the central axis map to obtain the central axis mapping map;

[0042] Based on the central axis mapping map, sample points are sequentially taken on the map according to the step length, and the distance from the sampling point to the edge of the venous blood vessel is obtained as the radius data of the venous blood vessel at this point;

[0043] Reconstruct a 3D model based on the central axis and radius data in the central axis map. First, each pixel point on the central axis is used as a feature point for feature matching. Based on the result of feature matching, image registration is achieved. Based on the result of registration, the position of each feature point in three-dimensional space is estimated to obtain the three-dimensional coordinates of each feature point. For each pair of adjacent three-dimensional points, a cylinder is used to simulate the blood vessel to connect these two points. Finally, by connecting all the cylinders, a continuous three-dimensional blood vessel is simulated to obtain the 3D model of the superficial venous blood vessels.

[0044] In some embodiments of this specification, the process of using Sobel filtering to perform edge detection and extraction on the 2D feature map of superficial venous blood vessels to obtain a binary feature map with obvious edges is as follows:

[0045] Perform a convolution operation on the image through two 3x3 Sobel operators as follows:

[0046]

[0047] Regard the feature map as a two-dimensional function. The Sobel operator is the gradient of the image change in the vertical and horizontal directions. That is, the Sobel operator is a two-dimensional object, and the elements of the two-dimensional object are the first-order derivatives of the functions in the horizontal and vertical directions respectively:

[0048]

[0049] The Sobel operator makes pixel value differences in the horizontal and vertical directions to obtain an approximate value of the image gradient. Combine the two feature maps obtained by performing convolution on the original image using two Sobel operators to obtain the edge features, and calculate the norm of Sobel. The calculation formula is as follows:

[0050]

[0051] Calculate the gradient approximate values in the horizontal and vertical directions to obtain the gradient magnitude and direction of each pixel point, and perform binarization on it to obtain a binary feature map with obvious edges.

[0052] In some embodiments of this specification, the process of skeletonizing the binary feature map and using the Zhang-Suen algorithm to extract the central axis is as follows:

[0053] Based on the binary feature map, the contour of the venous blood vessel is refined to a pixel width, and the Zhang-Suen algorithm is used to extract the central axis. When the algorithm iterates, the non-zero pixels on the image are traversed. When judging whether to delete or retain the pixel to be operated on at each center, it is judged according to the values of its surrounding 8 pixels. The operator of the Zhang-Suen algorithm is as follows:

[0054]

[0055] Each iteration goes through two-step operations to meet all the requirements of each stage to clear the pixel to be operated on. The operation rule of the first step is:

[0056] 2 ≤ B(P1) ≤ 6

[0057] A(P1) = 1

[0058] P2 × P4 × P6 = 0

[0059] P4 × P6 × P8 = 0;

[0060] Where B(P1) represents the number of non-zero neighbors among 8 neighbors, and A(P1) refers to how many times the 0-1 change occurs from P2 to P8; [[ID=2,8]]

[0061] The operation rule of the second step is:

[0062] 2 ≤ B(P1) ≤ 6

[0063] A(P1) = 1

[0064] [[ID=D38]]P2 × P4 × P8 = 0

[0065] P2 × P6 × P8 = 0;

[0066] Until no new pixels are deleted, the algorithm ends, and the central axis that can describe the venous blood vessel trend characteristics of the examined part is obtained, that is, the central axis map of the venous blood vessel is obtained.

[0067] In some embodiments of this specification, each pixel point of the central axis is used as a feature point for pairing and matching, and feature matching is achieved by calculating the distance between feature points to find the corresponding feature points in different images.

[0068] In some embodiments of this specification, based on the registration result, the position of each feature point in the three-dimensional space is estimated to obtain the three-dimensional coordinates of each feature point. For each pair of adjacent three-dimensional points, a cylinder is used to simulate a blood vessel to connect these two points. Finally, by connecting all the cylinders, a continuous three-dimensional blood vessel is simulated. The process of obtaining the 3D model of the surface vein blood vessels is as follows:

[0069] Calculate the disparity between each pair of matching points to deduce their depth information in the three-dimensional space. The specific process of disparity calculation is as follows:

[0070] For each pair of matching feature points (x i,j , y i,j ) and (x k,j , yk ,j ), calculate the disparity Δz i,k ;

[0071]

[0072] where d is the baseline distance between the two image planes, and f is the focal length of the camera;

[0073] For each feature point (x i,j , y i,j ), calculate its three-dimensional coordinates x i,j , y i,j , z i,j ) according to the disparity and the registration result; for each pair of adjacent three-dimensional points (x i , y i , z i ) and (x i+1 , y i+1 , z i+1 ), use a cylinder to simulate a blood vessel to connect these two points. The radius of the cylinder is determined by the blood vessel radius of the corresponding unit point, denoted as r i ; the center of the bottom surface of the cylinder is C1 = (x i , y i , z i ), the center of the top surface is C2 = (x i+1 , y i+1 , z i+1 ), and the radius is r i . Then any point P = (x, y, z) on the surface of the cylinder is determined by the following equation:

[0074]

[0075] Finally, by connecting all the cylinders, a continuous three-dimensional blood vessel is simulated.

[0076] The embodiments of this specification also provide a 3D imaging system for surface vein blood vessels based on three-dimensional mapping, which is used to implement the 3D imaging method for surface vein blood vessels based on three-dimensional mapping described in any one of the above. The 3D imaging system for surface vein blood vessels based on three-dimensional mapping includes:

[0077] A calculation and imaging display module;

[0078] A scanning module, connected to the calculation and imaging display module;

[0079] Among them, the scanning module includes a fill light module and a near-infrared camera module.

[0080] The embodiments of this specification can at least achieve the following beneficial effects:

[0081] Real-time imaging: Using a mobile phone as a computing device, through pipeline optimization technology, real-time imaging is achieved, enabling medical staff to obtain accurate vein imaging results in the shortest time.

[0082] 3D imaging (high visualization): By analyzing the 2D blood vessel feature map, a highly visual 3D model is reconstructed, which can clearly show the distribution and direction of veins at the examined site.

[0083] Lightweight and fast: The device is small and easy to carry and operate. When in use, only need to plug the scanning module into the mobile phone charging port and open the supporting APP to start using.

[0084] Simple operation: The graphical interface of the mobile phone APP is simple and intuitive, with an operation guidance function, enabling users to get started quickly.

[0085] Handheld wireless scanning: The scanning head + mobile phone can achieve handheld wireless scanning.

[0086] No redundant operations: Only one complete scan is required to obtain the complete vein blood vessel network information, improving work efficiency.

[0087] Multiple display methods: The system of the present invention can not only use the mobile phone as a screen display, but also be cast to other media.

[0088] Harmless, non-invasive, and sterile: Using a narrow-band infrared light source that is harmless to the human body, the operation process is non-invasive and supports a sterile environment.

[0089] Can operate independently: The system of the present invention only requires a mobile phone to operate independently.

[0090] Battery life: Using the mobile phone's reverse power supply as the energy source, it can operate for a long time in situations without energy supply such as in disaster areas or during wartime.

[0091] Result Qualification: The system finally outputs the relative positions between the venous blood vessels and between them and the examined part, providing a more accurate basis for diagnosis. Brief Description of the Drawings

[0092] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0093] Figure 1 It is a schematic diagram of the steps of the 3D imaging method of the surface venous blood vessels based on three-dimensional mapping involved in the present invention.

[0094] Figure 2 It is a schematic diagram of the 3D imaging system of the surface venous blood vessels based on three-dimensional mapping involved in the present invention.

[0095] Figure 3 It is a schematic diagram of the scanning module involved in the present invention.

[0096] Figure 4 It is a schematic diagram of the structure of the near-infrared camera module involved in the present invention.

[0097] Figure 5 It is an application schematic diagram of the 3D imaging system of the surface venous blood vessels based on three-dimensional mapping involved in the present invention.

[0098] Figure 6 It is a schematic diagram of the process of the 3D imaging method of the surface venous blood vessels based on three-dimensional mapping involved in the present invention.

[0099] Figure 7 It is a schematic diagram of the U-Net encoder and U-Net decoder involved in the present invention.

[0100] Figure 8 It is a schematic diagram of the three-dimensional modeling involved in the present invention.

[0101] Figure 9 It is a schematic diagram of the YOLOv5 object recognition involved in the present invention.

[0102] Figure 10 It is a schematic diagram of the network structure of YOLOv5 involved in the present invention.

[0103] Reference Signs:

[0104] 1. Mobile phone; 11. Mobile phone charging port;

[0105] 2. Scanning module; 21. Housing - upper; 22. Housing - lower; 23. Near-infrared camera module; 24. Fill light module; 25. USB-C male connector; 26. Voice coil focus motor; 27. Narrow-band lens; 28. PVC diffuser; 29. Narrow-band near-infrared LED light. DETAILED DESCRIPTION

[0106] Hereinafter, only certain exemplary embodiments are briefly described. As will be appreciated by those skilled in the art, the described embodiments may be modified in various ways without departing from the spirit or scope of the embodiments of the present invention. Therefore, the drawings and description are to be regarded as illustrative in nature and not restrictive.

[0107] The disclosure below provides many different embodiments or examples for implementing different structures of the embodiments of the present invention. In order to simplify the disclosure of the embodiments of the present invention, the components and configurations of specific examples are described below. Of course, these are merely examples and are not intended to limit the embodiments of the present invention. In addition, the embodiments of the present invention may repeat reference numerals and / or reference letters in different examples. Such repetition is for the purpose of simplicity and clarity and does not in itself indicate the relationship between the various embodiments and / or configurations discussed.

[0108] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0109] like Figure 1 As shown, this specification provides a 3D imaging method for surface venous vessels based on 3D mapping, including:

[0110] Emit infrared light to the surface of the body part being examined;

[0111] Acquiring an infrared characteristic image of the venous blood vessels obtained by infrared light reflected from the surface of the inspected part;

[0112] Preprocessing and filtering the infrared feature map of the veins, and using a deep learning image feature extraction and segmentation algorithm to extract and filter features to obtain a 2D feature map of the surface veins;

[0113] Using a blood vessel mapping analysis algorithm to process the 2D feature map of the surface veins, obtain the extension characteristics and thickness characteristics of each blood vessel segment, perform three-dimensional reconstruction, and obtain a 3D model of the surface veins;

[0114] Capturing the inspected part through real-time video to obtain a visible light image of the inspected part;

[0115] Use the YOLOv5 target recognition algorithm to identify the features of the inspected part and obtain a 3D model of the inspected part;

[0116] The 3D model of the body surface vein blood vessels and the 3D model of the examined part are three-dimensionally mapped into one model to obtain a 3D imaging map of the body surface vein blood vessels of the examined part.

[0117] In some embodiments of this specification, the preprocessing filter uses median filtering to reduce the noise of pixels by calculating the median of the pixel values in the surrounding area of the pixel and can preserve the edge information of the image.

[0118] In some embodiments of this specification, the deep learning image feature extraction and segmentation algorithm uses a U-Net network with a symmetric encoder-decoder structure.

[0119] In some embodiments of this specification, the specific process of preprocessing and filtering the infrared feature map of the vein blood vessels and using the deep learning image feature extraction and segmentation algorithm for feature extraction and filtering to obtain a 2D feature map of the body surface vein blood vessels is as follows:

[0120] Input the infrared feature map of the vein blood vessels, use a convolutional kernel of size 3x3 to step along the Z-shaped path for each pixel on the infrared feature map of the vein blood vessels, with a step size of 1 and padding of 0, calculate the gray values of the 8 pixel points around each pixel and sort them, and take the median as the pixel gray value at this position of the output image;

[0121] Then the filtered image enters the encoder of the U-Net. First, a convolution operation is performed to extract the features of the image. The convolution structure uniformly uses a convolutional kernel of 3x3, k = 3, padding p = 0, and step size s = 1. According to the formula for calculating the output features of the convolutional layer as shown below, n out is the output of the convolutional layer, n in is the input of the convolutional layer, p is the padding, k is the size of the convolutional kernel, and s is the step size;

[0122]

[0123] We get: n out =n in -2;

[0124] After convolution, the ReLU function is used as the activation function to perform a non-linear mapping on the convolution output. The ReLU function is:

[0125] f(x) = max(x, 0);

[0126] After two convolution operations for each layer, a max pooling operation is performed to enter the next layer of the encoder. The kernel size of each pooling layer is k = 2, the padding is p = 0, and the step size is s = 2. We get:

[0127] n out =n in / 2;

[0128] After performing 5 layers of convolution and 4 times of max pooling, a feature map of 32x22x1024 is obtained, and this feature map is fed into the U-Net decoder;

[0129] The original size of the feature map is restored through the U-Net decoder. This process consists of convolution, upsampling, and skip connection structures. First, the feature map is upsampled by a factor of 2x2, and the formula is:

[0130] n out = s(n in - 1) - 2p + k;

[0131] A feature map of 64x44x512 is obtained. A skip connection operation is performed on this feature map. The feature map of the corresponding mirror layer is cropped and concatenated with the feature map from the upper layer to form a feature with more channels, that is, concatenated with the upsampled feature map from the fifth layer of the U-Net encoder to generate a feature map of size 64x44x1024. Convolution and activation operations are performed on this feature map, and this operation is repeated 4 times. A feature map of 452x292x1 is obtained in the first layer, that is, a 2D map of the surface vein blood vessel features is obtained.

[0132] In some embodiments of this specification, the specific process of using the blood vessel mapping analysis algorithm to process the 2D feature map of the surface vein blood vessels, obtaining the extension features and thickness features of each blood vessel segment, and performing three-dimensional reconstruction to obtain a 3D model of the surface vein blood vessels is as follows:

[0133] Use Sobel filtering to perform edge detection and extraction on the 2D feature map of the surface vein blood vessels to obtain a binary feature map with obvious edges; [[ID=2,3]]]

[0134] Perform skeletonization on the binary feature map and use the Zhang-Suen algorithm to extract the central axis to obtain a central axis map of the vein blood vessels;

[0135] Overlap the binary feature map and the central axis map to obtain a central axis mapping map;

[0136] Based on the central axis mapping map, sampling points are sequentially taken on the map according to the step length, and the distance from the sampling point to the edge of the vein blood vessel is obtained as the radius data of the vein blood vessel at this point;

[0137] Reconstruct a 3D model based on the central axis and radius data in the central axis diagram. First, each pixel point on the central axis is used as a feature point for feature matching. Based on the results of feature matching, image registration is achieved. Based on the registration results, the position of each feature point in three-dimensional space is estimated to obtain the three-dimensional coordinates of each feature point. For each pair of adjacent three-dimensional points, a cylinder is used to simulate a blood vessel to connect these two points. Finally, by connecting all the cylinders, a continuous three-dimensional blood vessel is simulated to obtain a 3D model of the superficial venous blood vessels.

[0138] In some embodiments of this specification, the process of using Sobel filtering to perform edge detection and extraction on the 2D feature map of superficial venous blood vessels to obtain a binary feature map with obvious edges is as follows:

[0139] Perform a convolution operation on the image through two 3x3 Sobel operators as follows:

[0140]

[0141] Regard the feature map as a two-dimensional function. The Sobel operator is the gradient of the change of the image in the vertical and horizontal directions. That is, the Sobel operator is a two-dimensional object, and the elements of the two-dimensional object are the first-order derivatives of the functions in the horizontal and vertical directions respectively:

[0142]

[0143] The Sobel operator performs pixel value differences in the horizontal and vertical directions to obtain an approximate value of the image gradient. Combine the two feature maps obtained by performing convolution on the original image using two Sobel operators to obtain the combined results in the vertical and horizontal directions, and calculate the norm of Sobel. The calculation formula is as follows:

[0144]

[0145] Calculate the approximate values of the gradients in the horizontal and vertical directions to obtain the gradient magnitude and direction of each pixel point, and perform binarization on it to obtain a binary feature map with obvious edges.

[0146] In some embodiments of this specification, the process of performing skeletonization on the binary feature map and using the Zhang-Suen algorithm to extract the central axis to obtain the central axis diagram of the venous blood vessels is as follows:

[0147] Based on the binary feature map, refine the contour of the venous blood vessels to a pixel width, and use the Zhang-Suen algorithm to extract the central axis. When the algorithm iterates, traverse the non-zero pixels on the image. When judging whether to delete or retain the pixel to be operated on at each center, judge according to the values of its surrounding 8 pixels. The operator of the Zhang-Suen algorithm is as follows:

[0148]

[0149] Each iteration goes through two steps of operation to meet all the requirements of each stage to clear the pixels to be operated. The operation rules of the first step are:

[0150] 2≤B(P1)≤6

[0151] A(P1)=1

[0152] P2×P4×P6=0

[0153] P4×P6×P8=0;

[0154] Where B(P1) represents the number of non-zero neighbors among the 8 neighbors, and A(P1) refers to how many times the value of 0-1 changes from P2 to P8;

[0155] The second step operation rules are:

[0156] 2≤B(P1)≤6

[0157] A(P1)=1

[0158] P2×P4×P8=0

[0159] P2×P6×P8=0;

[0160] The algorithm ends when no new pixels are deleted, and a central axis that can describe the direction characteristics of the surface veins of the inspected part is obtained, that is, a central axis map of the veins is obtained.

[0161] In some embodiments of the present specification, each pixel point on the central axis is paired and matched as a feature point, and feature matching is achieved by calculating the distance between the feature points to find corresponding feature points in different images.

[0162] In some embodiments of this specification, based on the registration results, the position of each feature point in three-dimensional space is estimated to obtain the three-dimensional coordinates of each feature point. For each pair of adjacent three-dimensional points, a cylinder is used to simulate a blood vessel to connect the two points. Finally, by connecting all the cylinders, a continuous three-dimensional blood vessel is simulated to obtain a 3D model of the surface veins as follows:

[0163] Calculate the disparity between each pair of matching points to derive their depth information in three-dimensional space. The specific process of disparity calculation is as follows:

[0164] For each pair of matching feature points (x i,j ,y i,j ) and (x k,j ,yk ,j ), calculate the disparity Δz between themi,k ;

[0165]

[0166] Among them, d is the baseline distance between two image planes, and f is the focal length of the camera;

[0167] For each feature point (x i,j , y i,j ), its three-dimensional coordinates x i,j , y i,j , z i,j ) are calculated according to the parallax and registration results; for each pair of adjacent three-dimensional points (x i , y i , z i ) and (x i+1 , y i+1 , z i+1 ), a cylinder is used to simulate a blood vessel to connect these two points, and the radius of the cylinder is determined by the blood vessel radius of the corresponding unit point, denoted as r i ; the center of the bottom surface of the cylinder is C1 = (x i , y i , z i ), the center of the top surface is C2 = (x i+1 , y i+1 , z i+1 ), and the radius is r i , then any point P = (x, y, z) on the surface of the cylinder is determined by the following equation:

[0168]

[0169] Finally, by connecting all the cylinders, a continuous three-dimensional blood vessel is simulated.

[0170] The embodiment of this specification also provides a 3D imaging system for surface vein blood vessels based on three-dimensional mapping, which is used to implement the 3D imaging method for surface vein blood vessels based on three-dimensional mapping described in any one of the above. The 3D imaging system for surface vein blood vessels based on three-dimensional mapping includes:

[0171] A calculation and imaging display module;

[0172] A scanning module 2, connected to the calculation and imaging display module;

[0173] Among them, the scanning module 2 includes a supplementary light module 24 and a near-infrared camera module 23.

[0174] The technical concept of the present invention is as follows:

[0175] The implementation principle of this invention is mainly based on the fact that hemoglobin has two absorption peaks at wavelengths of 850 nm and 760 nm. Therefore, near-infrared light with a wavelength of 850 nm is easily absorbed by veins. The system of this invention uses an infrared LED to project and obtain the surface reflection feature map of the inspected part, and through preprocessing filtering, deep learning feature extraction and segmentation, and analysis and reconstruction algorithms to obtain a 3D model of the surface vein blood vessels. At the same time, for intuitive and highly visual display, this invention maps the 3D model of the vein blood vessels to the shape model of the inspected part obtained from mobile phone 1, and finally realizes the 3D imaging of the surface vein blood vessels. The hardware of the system of this invention mainly consists of two parts (as Figure 2 shown):

[0176] I. The scanning module 2 composed of a housing, a fill light module 24, and a near-infrared camera module 23 (as Figure 3 shown):

[0177] ① The housing is composed of upper and lower parts fixed by M1.5 stainless steel screws and is made of antibacterial polyether ether ketone (PEEK), which has excellent mechanical and chemical resistance. By adding antibacterial agents, it has antibacterial properties.

[0178] ② The fill light module 24 consists of 4 850-nm narrow-band near-infrared LED lights 29 and a PVC light homogenizing plate 28, and can emit 850-nm near-infrared light to the inspected part (as Figure 4 shown).

[0179] ③ The near-infrared camera module 23 consists of a CMOS imaging chip, a voice coil focusing motor 26 system, an 850-nm narrow-band lens 27, a protocol conversion chip, and a USB-C male head 25 (as Figure 4 shown), and can obtain the near-infrared light reflected from the inspected part, and convert the protocol of the CMOS imaging chip into the USB-UVC protocol that can be directly read by mobile phone 1 through the protocol conversion chip and transmit it to mobile phone 1 through the USB-C interface. At the same time, it can control the driving chip to drive the voice coil focusing motor  26 to obtain clear images. (as Figure 5 shown)

[0180] II. The computing and imaging display module composed of mobile phone 1: The operation APP of the system of this invention has good compatibility and high flexibility for the computing and imaging display module. The computing and imaging display module supports mobile phones 1 of different brands and different systems in terms of hardware, and non-USB-C interface mobile phones 1 can be connected through an adapter; in terms of software, because the output is the standard USB-UVC protocol, it supports mobile phones 1 of all systems well.

[0181] The processing flow of the system of this invention (as Figure 6As shown in the figure: After connecting the scanning module 2 of the system of the present invention to the computing and imaging display module (mobile phone 1), open the mobile phone 1 APP, and you can directly start scanning. After clicking the "Start Scanning" button, place the inspected part under the scanning module 2 and rotate the inspected part. At this time, hemoglobin in the superficial veins of the inspected part will absorb the infrared rays from the fill light module 24 of the scanning module 2, and the superficial part will reflect the infrared rays that are not absorbed and enter other parts. The near-infrared camera module 23 of the scanning module 2 will capture this phenomenon and obtain the original infrared feature map of the superficial veins containing noise. These original infrared maps of the superficial veins containing noise are transmitted to the computing and imaging display module (mobile phone 1) through the USB-C interface via the USB-UVC protocol for preprocessing. The original infrared feature map of the superficial veins containing noise will be transformed into a segmented 2D feature map of the superficial veins through preprocessing filtering and deep learning image feature extraction and segmentation algorithms. Then, use the vascular mapping analysis algorithm to obtain the extension features and thickness features of each segment of the blood vessel for 3D reconstruction, and finally obtain the 3D model of the superficial veins of the inspected part. At the same time, the camera of the computing and imaging display module (mobile phone 1) captures the image of the inspected part in real time, and the YOLOv5 deep learning object detection model deployed on the computing and imaging display module (mobile phone 1) adaptively identifies the feature points of the inspected part. Then, associate the 3D model of the superficial veins of the inspected part with the joint points of the inspected part to achieve the precise combination of the 3D model of the superficial veins of the inspected part and the model of the inspected part. Use the augmented reality (AR) function developed based on Unity3D (U3D) to superimpose the blood vessel model on the real-time image of the mobile phone 1 camera. The entire imaging process is real-time 3D imaging, and the imaging map of the superficial veins of the entire inspected part can be obtained with just one scan. The final 3D imaging result can not only be viewed on the software of this system, but also supports outputting files in different formats for convenient transfer to other software for comprehensive analysis.

[0182] In a specific embodiment, the hardware architecture of the system is as Figure 2 shown: The computing and imaging display module composed of the mobile phone 1 and the scanning module 2 are connected and communicate through USB-C.

[0183] Computing and imaging display module (mobile phone 1): The data output format of the scanning module 2 of this system is the standard USB-UVC protocol, which supports different systems well. Therefore, there is no specific requirement for the model of the mobile phone 1 required by the computing and imaging display module. Mobile phones 1 of different brands and different systems are supported. Mobile phones 1 with non-USB-C interfaces can be connected through an adapter (that is, different mobile phone charging ports 11 can be connected with different adapters), such as Android phones and Windows phones with USB-C interfaces, Android phones and Windows phones with microUSB interfaces, IOS phones with Lighting interfaces, etc.

[0184] Scanning module 2: The size of scanning module 2 is 33mm×61mm×19mm. The scanning module 2 (as shown in Figure 3 ) consists of a housing, a supplementary light module 24, and a near-infrared camera module 23. The supplementary light module 24 and the near-infrared camera module 23 are fixed on a six-layer FR4 material printed circuit board (PCB) with a size of 30mm×45mm and are fixed to the bottom housing by 4 antibacterial stainless steel M1.5 screws. (as shown in Figure 4 )

[0185] (1) The housing consists of two parts: housing - upper 21 and housing - lower 22. They are connected in a male-female plug-in form and fixed with antibacterial stainless steel M1.5 screws. The housing body is made of antibacterial polyether ether ketone (PEEK), which has excellent mechanical and chemical resistance. By adding antibacterial agents, it has antibacterial properties.

[0186] (2) The supplementary light module 24 consists of 4 narrow-band near-infrared LED lights 29 with a wavelength of 850nm and a rated power of 1W@350ma (encapsulated as 5050) and a PVC light homogenizing plate 28 (as shown in Figure 4 ): The brightness of the 4 LED beads is controlled by the data conversion chip CYUSB306X of the near-infrared camera module 23; the light homogenizing plate is composed of organic acrylic glass burned and pasted with a soft light film, which can convert the point light source into a uniform surface light source.

[0187] (3) The near-infrared camera module 23 consists of a CMOS imaging chip (IMX462LLR), a voice coil focusing motor 26 system (ML1813 + FP5510), an 850nm narrow-band lens 27 (M12), a protocol conversion chip (CYUSB3065), and a USB-C male head 25 (as shown in Figure 5 ):

[0188] 1. The CMOS imaging chip IMX462LLR is a black-and-white CMOS imaging chip produced by Sony. The wavelength band is visible light + 850nm narrow band, which can obtain the venous infrared feature map reflected by the inspected part. The exposure method is global exposure, and it can clearly, without distortion and without jelly effect, obtain the feature map of each frame under dynamic recognition. The resolution of IMX462LLR is 1920×1080, and the frame rate can reach 120 frames per second under the condition of outputting in MIPI-4Lane protocol.

[0189] 2. The voice coil focus motor 26 system consists of a VCM voice coil motor ML1813 and a VCM voice coil motor driver FP5510. The VCM voice coil motor ML1813 is a device that controls the distance between the lens and the CMOS. The lens is fixed to the internally movable coil through an M12 thread. By changing the driving current, different magnetic field magnitudes are generated in the coil, and different displacements are generated under the condition of the internal permanent magnetic field, thereby controlling the distance between the lens fixed on the coil and the CMOS. The movement range of ML1813 is from -1 mm to 2 mm, and the rated maximum driving current is 120 mA. FP5510 is a VCM voice coil motor driver with a rated maximum driving current of 120 mA, which is within the range of the VCM voice coil motor ML1813. It can be programmed through the I2C bus to make its output port output an accurate current magnitude, thereby precisely controlling the focusing operation. The protocol conversion chip (CYUSB3065) and FP5510 are connected through the I2C bus to control the position of the VCM voice coil motor ML1813.

[0190] 3. The 850 nm narrowband lens 27 (M12) consists of an M12 lens installed with an 850 nm narrowband filter. The focal length of the M12 lens is 3.6 MM, the maximum pixel is 1080p, the target surface size is 1 / 2.7", and the size of the 850 nm filter is 14 mm × 14 mm × 1 mm. It only allows light with a wavelength of 850 nm ± 5 nm to pass through, filtering out the interference of visible light. The 850 nm narrowband lens 27 (M12) is installed on the VCM voice coil motor ML1813 of the voice coil focus motor 26 system.

[0191] 4. The protocol conversion chip (CYUSB3065) is a bridge controller of MIPI CSI-2 to USB3 Gen1 produced by Infineon Technologies. Its main purpose is to convert the MIPI-CSI 4Lane protocol signal output by the CMOS imaging chip IMX462LLR into a standard USB-UVC protocol. And it has an ARM core with a frequency of 200 MHz, so it can be programmed to control the voice coil focus motor 26 system and adjust the brightness of the fill light module 24.

[0192] 5. The USB-C male connector 25 complies with the USB TYPE-C interface specification. The maximum data transfer protocol it transmits is USB3 Gen2@10 Gbps. In this invention, the transmission protocol used is USB3 Gen1@5 Gbps, and it is downward compatible with USB2@480 Mbps. The USB-C male connector 25 is connected to the computing and imaging display module (mobile phone 1) through the mobile phone charging port 11 and transmits the original image data. At the same time, the computing and imaging display module (mobile phone 1) supplies power to the scanning module 2.

[0193] The software processing flow of the system of this invention is asFigure 6 As shown, after the scanning starts, the scanning module 2 of the system of the present invention transmits the infrared feature map of the vein blood vessels on the examined part, which is reflected by the near-infrared camera module 23 from the fill light module 24 of the scanning module 2, to the mobile phone 1 through the USB-C interface. The processing and calculation part is completed by the mobile phone 1. However, these feature maps are the original infrared maps of the body surface vein blood vessels containing noise. To ensure the correctness of subsequent image feature extraction and segmentation, it needs to be preprocessed while retaining the detailed information. First, preprocessing filtering is performed on the original infrared map of the body surface vein blood vessels containing noise, and then a deep learning image feature extraction and segmentation algorithm is used for feature extraction and filtering. Finally, the 2D feature map of the body surface vein blood vessels is obtained. Median filtering is used for preprocessing filtering. It reduces the pixel noise by calculating the median of the pixel values in the surrounding area of the pixel and can preserve the edge information of the image. The deep learning image feature extraction and segmentation algorithm uses the U-Net network, which is a fully convolutional neural network architecture for image feature extraction and segmentation. The characteristic of this network is that it has a symmetric encoder-decoder structure and can effectively capture the local features and context information in the image (as Figure 6 、 Figure 7 shown).

[0194] The specific process is as follows:

[0195] The input original infrared map of the body surface vein blood vessels containing noise is a grayscale image with a size of 640x480. A convolutional kernel with a size of 3x3 steps in a Z-shaped manner along each pixel on the original infrared map of the body surface vein blood vessels containing noise, with a step size of 1 and a padding of 0. This convolutional kernel will sort the gray values of the 8 pixel points around this pixel and the gray value of this pixel and take the median as the pixel gray value at this position of the output image. Median filtering will remove salt-and-pepper noise and Gaussian noise. Since median filtering is an operation based on sorting, it can, to a certain extent, preserve the edge information in the image and will not produce too much blurring effect.

[0196] Then the filtered image enters the encoder (contraction path) of the U-Net, with a size of 640x480. First, a convolutional operation is performed to extract the features of the image. The convolutional structure of the present invention uniformly uses a convolutional kernel with k = 3 of 3x3, a padding of p = 0, and a step size of s = 1. According to the formula for calculating the output features of the convolutional layer (n out is the output of the convolutional layer, n in is the input of the convolutional layer, p is the padding, k is the size of the convolutional kernel, and s is the step size):

[0197]

[0198] We get:

[0199] n out = n in-2;

[0200] From Figure 7 it can be seen that each layer of the encoder convolutional layer is composed of two 3x3 convolutional layers. Passing through each convolutional layer reduces the size of the feature map by 2 pixels. After convolution, ReLU is used as the activation function to perform a non-linear mapping on the convolutional output. The ReLU function is:

[0201] f(x) = max(x, 0);

[0202] After each layer undergoes two convolutional operations, it needs to go through a max-pooling operation to enter the next layer of the encoder. The kernel size of each pooling layer is k = 2, the padding is p = 0, and the stride is s = 2. So we get:

[0203] n out = n in / 2;

[0204] From Figure 7 it can be seen that after each max-pooling, the size of the 2x2 feature map is reduced to 1 / 2 of the original. After 5 layers of convolution and 4 max-pooling operations, a feature map with a size of 32x22 and a channel number of 1024 is obtained. The fifth layer does not have a 2x2 max-pooling operation, but directly sends the obtained feature map into the U-Net decoder (expansion path).

[0205] Next, the feature map is restored to its original size through the U-Net decoder. This process consists of convolution, upsampling, and skip connection structures. First, the U-Net decoder upsamples the feature map with a size of 32x22x1024 from the fifth layer of the U-Net encoder by 2x2. The formula is:

[0206] n out = s(n in - 1) - 2p + k;

[0207] It can be seen from Figure 7 the U-Net decoder part that the output size expands to 2 times the original, which is 64x44, and the channel number is 1 / 2 of the original, which is 512. Then the U-Net performs a skip connection operation, cropping the feature map of the corresponding mirror layer and splicing it with the feature map from the upper layer to form a feature with more channels. From Figure 5In the fourth layer of the U-Net decoder, we can see that the feature map transferred from the fifth layer of the U-Net encoder is of size 72x52x512. After cropping, it is reduced to 64x44x512. This is then concatenated with the upsampled feature map from the fifth layer of the U-Net encoder to generate a feature map of size 64x44x1024. Finally, similar to the convolution operation in the U-Net encoder, the feature map is convolved and reactivated. This process is repeated four times, resulting in a 452x292x1 feature map in the first layer. This results in a 2D map of surface venous features.

[0208] After obtaining the 2D image of the surface vein characteristics, the present invention uses the vascular mapping analysis algorithm to obtain the extension characteristics and thickness characteristics of each segment of the blood vessel, performs three-dimensional reconstruction, and finally obtains a 3D model. This is also the core technology of the present invention. Figure 8 As shown, the specific process is as follows:

[0209] To obtain the extended features, the central axis of the vein needs to be obtained. First, edge detection is performed to detect the boundary of the object from the 2D feature map. The present invention uses Sobel filtering for edge detection and extraction. The Sobel filter performs a convolution operation on the image through two 3x3 operators (Sobel operators), as follows:

[0210]

[0211] If the feature map is considered as a two-dimensional function, then the Sobel operator is the gradient of the image in the vertical and horizontal directions. It is a two-dimensional object, and the elements of the object are the first-order derivatives of the function in the horizontal and vertical directions respectively:

[0212]

[0213] Where I is a two-dimensional function about x and y, grad(I) is the gradient of function I, and are the partial derivatives of function I with respect to x and y.

[0214] The Sobel operator performs pixel value differences in the horizontal and vertical directions to obtain an approximate value of the image gradient. By using two operators to perform convolution on the original image to obtain edge features, the two feature maps obtained are combined to obtain the comprehensive results in the vertical and horizontal directions, and the Sobel norm is calculated. The calculation formula is as follows:

[0215]

[0216] By calculating the approximate gradient values in the horizontal and vertical directions, we can get the gradient size and direction of each pixel. A larger gradient value usually means that the image color changes quickly, and these points may be the edges in the image. Then binarize it and finally get the following Figure 8 The binary feature map shown on the upper left has clear edges.

[0217] After obtaining the image obtained through edge detection, the present invention performs skeleton processing on the image with edge features to obtain the central axis of the vein, such as Figure 8 Upper right. The purpose of skeletonization is to refine the outline of the vein to one pixel width, while retaining the topological structure and geometric features of the original vein as much as possible, so as to describe the shape and direction of the vein. The present invention extracts the central axis through the Zhang-Suen algorithm. When the algorithm iterates in a loop, it traverses the pixels on the image that are not 0. When determining whether to delete or retain each center pixel to be operated (P1), the algorithm makes a judgment based on the values of the 8 pixels (P2-P8) around it. The operator of the Zhang-Suen algorithm is as follows:

[0218]

[0219] Each iteration of the Zhang-Suen algorithm requires two steps of operation to meet all the requirements of each stage before the pixel to be operated (P1) can be cleared to zero. The operation rules of the first step are:

[0220] 2≤B(P1)≤6

[0221] A(P1)=1

[0222] P2×P4×P6=0

[0223] P4×P6×P8=0;

[0224] Where B(P1) represents the number of non-zero neighbors among the 8 neighbors, and A(P1) refers to how many times the value changes from 0 to 1 from P2 to P8.

[0225] The second step operation rules are:

[0226] 2≤B(P1)≤6

[0227] A(P1)=1

[0228] P2×P4×P8=0

[0229] P2×P6×P8=0;

[0230] Steps 3 and 4 of Phase 1 are to remove the points on the east and south boundary lines and the corner points at the northwest corner. Steps 3 and 4 of Phase 2 are to remove the points on the west and north boundary lines and the corner points at the southeast corner. For the binary image obtained by the previous Sobel filtering, the algorithm needs to continuously iterate Steps 1 and 2 until no new pixels are deleted in a certain phase, and then the algorithm ends. Finally, the central axis that can describe the characteristics of the surface vein vessels of the inspected part is obtained.

[0231] After the first two steps, an image containing the boundary information of the surface vein vessels of the inspected part and an image containing the characteristic information of the vessel direction are obtained. Then, the two images are overlapped according to the feature points, and the central axis mapping diagram as shown in Figure 6 the lower right is obtained.

[0232] As shown in Figure 6 the lower left, then sampling points are sequentially taken on the reticular central axis feature map according to the specified step length, and the distance from the sampling point to the edge of the vein vessel is obtained as the radius data of the vein vessel at this point and stored.

[0233] Finally, the extracted central axis and radius data are used to reconstruct the 3D model using the principle of three-dimensional coordinate mapping. Three-dimensional coordinate mapping is to map the feature points in the two-dimensional image to the corresponding positions in the three-dimensional space, and then use the cylinder prefab to simulate and form a continuous three-dimensional structure. The following are the detailed steps and principles:

[0234] Each pixel point on the central axis is paired and matched as a feature point. Feature matching is achieved by calculating the distance between feature points (such as Euclidean distance). This step aims to find the corresponding feature points in different images and provide a basis for estimating the relative motion and depth change between images.

[0235] Based on the results of feature matching, image registration is achieved. Image registration refers to aligning multiple images to the same coordinate system for easy comparison and analysis. Suppose there are N images at different angles, and the corresponding planes of these images in the three-dimensional space are Π1, Π2, …, Π N .

[0236] Estimation of depth and three-dimensional position

[0237] Based on the registration results, estimate the position of each feature point in the three-dimensional space. This step usually involves calculating the parallax between each pair of matching points to deduce their depth information in the three-dimensional space.

[0238] The specific process of parallax calculation is as follows:

[0239] For each pair of matching feature points (x i,j , y i,j ) and (x k,j , yk,j ), calculate the parallax Δz between them i,k :

[0240]

[0241] where d is the baseline distance between the two image planes and f is the focal length of the camera.

[0242] Three-dimensional coordinate estimation:

[0243] For each feature point (x i,j , y i,j ), calculate its three-dimensional coordinates x i,j , y i,j , z i,j ) according to the parallax and the image registration result.

[0244] Generation and connection of blood vessels

[0245] For each pair of adjacent three-dimensional points (x i , y i , z i ) and (x i+1 , y i+1 , z i+1 ), use a cylinder to simulate a blood vessel to connect these two points. The radius of the cylinder is determined by the blood vessel radius of the corresponding unit point, denoted as r i . Assume that the center of the bottom surface of the cylinder is C1 = (x i , y i , z i ), the center of the top surface is C2 = (x i+1 , y i+1 , z i+1 ), and the radius is r i . Then any point P = (x, y, z) on the surface of the cylinder can be determined by the following equation:

[0246]

[0247] Finally, simulate a continuous three-dimensional blood vessel by connecting all the cylinders.

[0248] Meanwhile, the built-in camera of mobile phone 1 completes the detection of joint points of the inspected part. Its processing flow is as follows:

[0249] Use the built-in camera of mobile phone 1 to capture real-time video. The image frames captured by the camera are transmitted in real time to the YOLOv5 model on the mobile phone 1 side for real-time detection to identify the joint positions of the inspected part. Taking the hand as an example, the key points are shown in Figure 9 . The model will output information including joint coordinates and confidence levels, and these data will be used for subsequent association binding of 3D model positions and AR display.

[0250] The YOLOv5 algorithm is a deep learning-based object detection algorithm. Its core idea is to regard object detection as a regression problem and directly predict the position and category of the object through a convolutional neural network. The network structure of YOLOv5 is as follows Figure 10 shown. It mainly includes four modules: Input, Backbone network, Neck structure, and Head structure (Prediction), which are responsible for preprocessing the input image, feature extraction, feature fusion, and outputting detection information respectively. Among them, the Backbone module uses the BottleNeckCSP structure to extract rich information features from the input image.

[0251] Finally, the results of the first two parts are combined, and the surface vein blood vessel model of the examined part after 3D reconstruction is associated and bound with the detected key points, and is accurately displayed using augmented reality (AR) on the screen of the mobile phone 1, and the position and pose are updated in real time as the pose of the examined part changes. For the convenience of users to analyze and process it, the video or the vein blood vessel model can be exported as common model formats such as OBJ, DICOM, STL, VTK, PLY, etc. for analysis in other analysis software.

[0252] The above-described embodiments are used to illustrate the present invention, not to limit the present invention. Therefore, changes in the exemplified numerical values or replacement of equivalent elements still belong to the scope of the present invention.

[0253] From the above detailed description, those of ordinary skill in the art can clearly understand that the present invention can indeed achieve the foregoing objectives, and it actually complies with the provisions of the patent law.

[0254] Although the preferred embodiments of the present invention have been described, those skilled in the art can make additional changes and modifications to these embodiments once they know the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present invention. The above is only the preferred embodiments of the present invention and is not used to limit the present invention. It should be noted that any modifications, equivalent replacements, and improvements made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

[0255] It should be noted that the above description of the process is only for illustration and explanation, and does not limit the scope of application of this specification. For those skilled in the art, various corrections and changes can be made to the process under the guidance of this specification. However, these corrections and changes are still within the scope of this specification.

[0256] The basic concepts have been described above. Obviously, for those of ordinary skill in the art after reading this application, the above invention disclosure is only an example and does not constitute a limitation to this application. Although not explicitly stated here, those of ordinary skill in the art may make various modifications, improvements, and corrections to this application. Such modifications, improvements, and corrections are suggested in this application, so such modifications, improvements, and corrections still fall within the spirit and scope of the exemplary embodiments of this application.

[0257] At the same time, this application uses specific terms to describe the embodiments of this application. For example, "one embodiment", "an embodiment", and / or "some embodiments" mean a certain feature, structure, or characteristic related to at least one embodiment of this application. Therefore, it should be emphasized and noted that "an embodiment" or "one embodiment" or "an alternative embodiment" mentioned two or more times at different positions in this specification is not necessarily referring to the same embodiment. In addition, certain features, structures, or characteristics in one or more embodiments of this application can be appropriately combined.

[0258] In addition, those of ordinary skill in the art can understand that various aspects of this application can be illustrated and described by several patentable types or situations, including any new and useful process, machine, product, or combination of substances, or any new and useful improvement thereof. Therefore, various aspects of this application can be implemented entirely by hardware, entirely by software (including firmware, resident software, microcode, etc.), or by a combination of hardware and software. The above hardware or software can all be referred to as "units", "modules", or "systems". In addition, various aspects of this application can take the form of a computer program product embodied in one or more computer-readable media, in which computer-readable program code is included.

[0259] The computer program code required for the operations of various parts of this application can be written in any one or more programming languages, including object-oriented programming languages such as Java, Scala, Smalltalk, Eiffel, JADE, Emerald, C++, C#, VB.NET, Python, etc., conventional procedural programming languages such as C programming language, VisualBasic, Fortran2103, Perl, COBOL2102, PHP, ABAP, dynamic programming languages such as Python, Ruby and Groovy, or other programming languages, etc. This program code can run entirely on the user's computer, or run on the user's computer as an independent software package, or run partially on the user's computer and partially on a remote computer, or run entirely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer through any network form, such as a local area network (LAN) or a wide area network (WAN), or connected to an external computer (for example, through the Internet), or in a cloud computing environment, or used as a service such as software as a service (SaaS).

[0260] In addition, unless clearly stated in the claims, the order of the processing elements and sequences, the use of numbers and letters, or the use of other names in this application are not used to limit the order of the processes and methods of this application. Although some currently considered useful embodiments of the invention are discussed through various examples in the above disclosure, it should be understood that such details are only for illustrative purposes, and the appended claims are not limited to the disclosed embodiments. On the contrary, the claims are intended to cover all modifications and equivalent combinations that conform to the essence and scope of the embodiments of this application. For example, although the implementation of the above various components can be embodied in a hardware device, it can also be implemented as a pure software solution, for example, installed on an existing server or mobile device.

[0261] Similarly, it should be noted that, in order to simplify the description of this application disclosure and thus help the understanding of one or more embodiments of the invention, in the previous description of the embodiments of this application, sometimes multiple features are merged into one embodiment, drawing or description thereof. However, this method of this application should not be construed as reflecting the intention that the claimed subject matter requires more features than those clearly recited in each claim. On the contrary, the subject of the invention should have fewer features than the above single embodiment.

Claims

1. A 3D imaging method for surface veins based on three-dimensional mapping, characterized in that: include: Emit infrared light to the surface of the body part being examined; Acquiring an infrared characteristic image of the venous blood vessels obtained by infrared light reflected from the surface of the inspected part; Preprocessing and filtering the infrared feature map of the veins, and using a deep learning image feature extraction and segmentation algorithm to extract and filter features to obtain a 2D feature map of the surface veins; Using a blood vessel mapping analysis algorithm to process the 2D feature map of the surface veins, obtain the extension characteristics and thickness characteristics of each blood vessel segment, perform three-dimensional reconstruction, and obtain a 3D model of the surface veins; Capturing the inspected part through real-time video to obtain a visible light image of the inspected part; Use the YOLOv5 target recognition algorithm to identify the features of the inspected part and obtain a 3D model of the inspected part; The 3D model of the surface veins of the body and the 3D model of the inspected part are three-dimensionally mapped into one model to obtain a 3D imaging image of the surface veins of the inspected part; The specific process of using the blood vessel mapping analysis algorithm to process the 2D feature map of the surface veins, obtain the extension characteristics and thickness characteristics of each blood vessel segment, perform three-dimensional reconstruction, and obtain the 3D model of the surface veins is as follows: Sobel filtering is used to detect and extract edges from the 2D feature map of surface veins to obtain a binary feature map with clear edges. Skeletonizing the binary feature map and extracting the central axis using the Zhang-Suen algorithm to obtain a central axis map of the venous blood vessels; Overlapping the binary feature map and the central axis map to obtain a central axis mapping map; Based on the central axis mapping diagram, sampling points are sequentially taken on the diagram according to the step length, and the distance between the sampling point and the edge of the vein is obtained as the radius data of the vein at that point; A 3D model is reconstructed based on the central axis and radius data in the central axis diagram. First, feature matching is performed on each pixel point of the central axis as a feature point. Based on the result of feature matching, alignment between images is achieved. Based on the result of alignment, the position of each feature point in three-dimensional space is estimated to obtain the three-dimensional coordinates of each feature point. For each pair of adjacent three-dimensional points, a cylinder is used to simulate a blood vessel to connect the two points. Finally, by connecting all the cylinders, a continuous three-dimensional blood vessel is simulated to obtain a 3D model of the surface veins.

2. The method for 3D imaging of surface veins based on 3D mapping according to claim 1, characterized in that: The pre-processing filter adopts median filtering to reduce pixel noise by calculating the median of pixel values in the area surrounding the pixel and preserve edge information of the image.

3. The method for 3D imaging of surface veins based on 3D mapping according to claim 1, characterized in that: The deep learning image feature extraction and segmentation algorithm uses a U-Net network with a symmetrical encoder-decoder structure.

4. The method for 3D imaging of surface veins based on 3D mapping according to claim 1, characterized in that: The specific process of pre-processing and filtering the infrared feature map of the venous blood vessels, and using a deep learning image feature extraction and segmentation algorithm to extract and filter the features to obtain a 2D feature map of the surface venous blood vessels is as follows: Input the venous infrared feature map, use a 3x3 convolution kernel to step along each pixel in a zigzag pattern on the venous infrared feature map, with a step size of 1 and padding of 0. Calculate the grayscale value of the eight pixels around each pixel and sort them with the pixel's grayscale value, and take the median as the grayscale value of the corresponding position of the current processing pixel in the output image; The filtered image then enters the U-Net encoder, where a convolution operation is first performed to extract the features of the image. The convolution structure uniformly uses a 3x3 convolution kernel, k=3, p=0 padding, and a step size of s=1. The formula for calculating the output features of the convolution layer is as follows: n out is the output of the convolution layer, n in is the convolution layer input, p is the padding, k is the convolution kernel size, and s is the step size; Get: n out =n in -2; After convolution, ReLU is used as the activation function to perform nonlinear mapping on the convolution output. The ReLU function is: f(x)=max(x,0); After two convolution operations in each layer, it enters the next layer of the encoder through a maximum pooling operation. The kernel size of each pooling layer is k=2, the padding is p=0, and the step size is s=2, resulting in: n out =n in / 2; After 5 layers of convolution and 4 times of maximum pooling, a 32x22x1024 feature map is obtained, which is fed into the U-Net decoder. The original size of the feature map is restored through the U-Net decoder. The process consists of convolution, upsampling and skip structure. First, the feature map is upsampled to 2x2. The formula is: n out =s(n in -1)-2p+k; A 64x44x512 feature map is obtained, and a skip connection operation is performed on the feature map. The feature map of the corresponding mirror layer is cropped and spliced with the feature map from the upper layer to form a feature with more channels. That is, it is spliced with the upsampled feature map from the fifth layer of the U-Net encoder to generate a feature map of size 64x44x1024. The feature map is convolved and reactivated. This operation is repeated 4 times to obtain a 452x292x1 feature map in the first layer, that is, a 2D map of the surface venous blood vessel features is obtained.

5. The method for 3D imaging of surface veins based on 3D mapping according to claim 1, characterized in that: The process of using Sobel filtering to perform edge detection and extraction on the 2D feature map of surface veins to obtain a binary feature map with obvious edges is as follows: The image is convolved using two 3x3 Sobel operators as follows: Consider the feature map as a two-dimensional function. The Sobel operator is the gradient of the image in the vertical and horizontal directions. That is, the Sobel operator is a two-dimensional object. The elements of the two-dimensional object are the first-order derivatives of the function in the horizontal and vertical directions respectively: The Sobel operator performs pixel value differences in the horizontal and vertical directions to obtain an approximate value of the image gradient. The two feature maps obtained by convolving the original image with two Sobel operators to obtain edge features are combined to obtain the comprehensive results in the vertical and horizontal directions. The Sobel norm is calculated as follows: |grad(I)| represents the image gradient magnitude, and Represent the gradient components of the image in the x-direction and y-direction respectively; I represents the grayscale value of the image pixel; Calculate the approximate gradient values in the horizontal and vertical directions to obtain the gradient size and direction of each pixel, and binarize it to obtain a binary feature map with obvious edges.

6. The method for 3D imaging of surface veins based on 3D mapping according to claim 1, characterized in that: The process of skeletonizing the binary feature map and extracting the central axis using the Zhang-Suen algorithm to obtain the central axis map of the venous blood vessels is as follows: Based on the binary feature map, the outline of the vein is refined to a pixel width, and the Zhang-Suen algorithm is used to extract the central axis. When the algorithm iterates, it traverses the non-zero pixels on the image. When deciding whether to delete or retain each central pixel to be operated, it makes a judgment based on the values of the eight pixels around it. The operator of the Zhang-Suen algorithm is as follows: 11 is the current pixel, and 12-P9 is its 8 neighboring pixels; Each iteration goes through two steps of operation to meet all the requirements of each stage to clear the pixels to be operated. The operation rules of the first step are: 2≤B(P1)≤6 A(P1)=1 P2×P4×P6=0 P4×P6×P8=0; Where B(P1) represents the number of non-zero neighbors among the 8 neighbors, and A(P1) refers to how many times the value of 0-1 changes from P2 to P8; The second step operation rules are: 2≤B(P1)≤6 A(P1)=1 P2×P4×P8=0 P2×P6×P8=0; The algorithm ends when no new pixels are deleted, and a central axis that can describe the direction characteristics of the surface veins of the inspected part is obtained, that is, a central axis map of the veins is obtained.

7. The method for 3D imaging of surface veins based on 3D mapping according to claim 1, characterized in that: Each pixel point on the central axis is paired and matched as a feature point, and feature matching is achieved by calculating the distance between feature points to find the corresponding feature points in different images.

8. The method for 3D imaging of surface veins based on 3D mapping according to claim 1, characterized in that: Based on the registration results, the position of each feature point in three-dimensional space is estimated to obtain the three-dimensional coordinates of each feature point. For each pair of adjacent three-dimensional points, a cylinder is used to simulate a blood vessel to connect the two points. Finally, by connecting all the cylinders, a continuous three-dimensional blood vessel is simulated to obtain a 3D model of the surface veins. The process is as follows: Calculate the disparity between each pair of matching points to derive their depth information in three-dimensional space. The specific process of disparity calculation is as follows: For each pair of matching feature points (x i,j ,y i,j ) and (x k,j ,y k,j ), calculate the disparity Δz between them i,k ; Where d is the baseline distance between the two image planes, and f is the focal length of the camera; For each feature point (x i,j ,y i,j ), calculate its three-dimensional coordinates (x i,j ,y i,j ,z i,j ); for each pair of adjacent three-dimensional points (x i ,y i ,z i ) and (x i+1 ,y i+1 ,z i+1 ), a cylinder is used to simulate blood vessels to connect these two points. The radius of the cylinder is determined by the radius of the blood vessel at the corresponding unit point, expressed as r i ; The center of the bottom of the cylinder is C1=(x i ,y i ,z i ), the center of the top surface is C2=(x i+1 ,y i+1 ,z i+1 ), with a radius of r i , then any point P = (x, y, z) on the surface of the cylinder is determined by the following equation: Finally, by connecting all the cylinders, a continuous three-dimensional blood vessel is simulated.

9. A 3D imaging system for surface veins based on three-dimensional mapping, characterized in that: A 3D imaging system for surface veins based on three-dimensional mapping, used to implement the method for 3D imaging of surface veins based on three-dimensional mapping according to any one of claims 1 to 8, comprises: Computing and imaging display module; A scanning module connected to the computing and imaging display module; Wherein, the scanning module includes a fill light module and a near-infrared camera module.

Images