A 3D imaging method and system for surface veins

Through infrared light and deep learning technology, real-time 3D imaging of surface venous blood vessels is achieved, solving the problem that the existing technology cannot achieve complex 3D imaging and operation, and providing highly visualized vascular network information, suitable for a variety of medical applications.

CN118710819BActive Publication Date: 2024-11-19UNIV OF ELECTRONIC SCI & TECH OF CHINA CHENGDU COLLEGE +2
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411195432.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-29
Publication Date
2024-11-19
Estimated Expiration
2044-08-29

AI Technical Summary

Technical Problem

The existing surface venous angiography technology has problems such as inability to achieve 3D imaging, complex operation, high invasiveness, and harmful to the human body, and it is impossible to obtain complete vascular network information in real time.

Method used

The infrared light emission and deep learning image feature extraction and segmentation algorithm are used, combined with the vascular mapping analysis algorithm for preprocessing and three-dimensional reconstruction, and the 3D model of the surface venous blood vessels is obtained, and it is mapped and rendered with the 3D model of the tested part to achieve Real-time 3D imaging.

Benefits of technology

Real-time, harmless and invasive 3D imaging is achieved, simplified operation, and a single scan can obtain complete vascular network information. The results are highly visualized, suitable for different groups of people and suitable for telemedicine.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118710819B_ABST
    Figure CN118710819B_ABST
Patent Text Reader

Abstract

The present invention provides a method and system for 3D imaging of surface veins, belonging to the technical field of 3D imaging of surface veins, and the method includes: emitting infrared light to the surface of the inspected part; obtaining infrared characteristic map of veins; preprocessing and filtering the infrared characteristic map of veins, and extracting and filtering the features using a deep learning image feature extraction and segmentation algorithm to obtain a 2D characteristic map of surface veins; obtaining the extension features and thickness features of each segment of the blood vessel, performing three-dimensional reconstruction, and obtaining a 3D model of surface veins; obtaining a 3D model of the inspected part; three-dimensionally mapping the 3D model of surface veins and the 3D model of the inspected part into a combined model, and performing coloring, transparent material filling, rendering and visualization processing to obtain a 3D imaging map of the surface veins of the inspected part. The method has the characteristics of real-time imaging, 3D imaging, easy operation, and high visualization of the results.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of 3D imaging of body surface veins, and in particular to a method and system for 3D imaging of body surface veins. Background Art

[0002] At present, the applications of surface venous imaging in the medical field include vascular reconstruction surgery, diagnosis and treatment of skin diseases (varicose veins, etc.), venous puncture treatment, intravenous infusion, blood collection, etc. At present, the commonly used surface venous imaging products include laser vascular imagers, near-infrared vascular imagers, CT angiography, color Doppler ultrasound examination, optical coherence tomography OCT, etc., but these products all have certain defects:

[0003] Laser vascular imager: uses laser processing technology to irradiate images onto the skin surface, and the images appear red. The visualization is high but the display area is small. It uses Class II lasers, which emit radiation and are harmful to the human body when viewed directly. It cannot form 3D images, and a single scan cannot obtain complete vascular network information.

[0004] Near-infrared vascular imager: Detects veins through near-infrared light and projects images onto the skin surface using DLP technology. High degree of visualization but small display area. 3D imaging is not possible, and a single scan cannot obtain complete vascular network information.

[0005] CT angiography: uses computed tomography (CT) and intravenous contrast agents to generate images of the vascular system. A single scan can obtain complete 3D vascular network information, with high visualization and good results. However, it is difficult to operate and requires professional training; it requires injection of contrast agents, which is somewhat invasive; and it is not real-time imaging.

[0006] Color Doppler ultrasound examination: Ultrasonic pulses are sent to the body through an ultrasound probe, and the probe then receives the reflected ultrasonic signals. These signals are converted into images, showing the structure of the blood vessels; a single scan can obtain complete vascular network information, and 3D imaging is not possible; the degree of visualization is average; the operation requires professional training; a coupling agent needs to be applied, which is somewhat invasive.

[0007] Optical coherence tomography (OCT): uses the principle of optical interference to measure the reflection of light. A beam of light is split into two parts, one for interaction with the tissue being imaged and the other as a reference light. When the two beams of light are recombined, they produce an interference pattern, and the reflective properties of the tissue can be inferred based on the changes in the interference pattern. The operation requires professional training; a single scan cannot obtain complete vascular network information; the degree of visualization is low. Summary of the invention

[0008] The present invention provides a 3D imaging method and system for surface venous blood vessels, which has the characteristics of real-time imaging, 3D imaging, easy operation and high visualization of results.

[0009] The embodiment of this specification discloses a method for 3D imaging of surface venous vessels, including:

[0010] Emit infrared light to the surface of the body part being examined;

[0011] Acquire the infrared characteristic image of the venous blood vessels obtained by the infrared light reflected from the surface of the examined part;

[0012] Preprocessing and filtering the infrared feature map of the venous blood vessels, and using a deep learning image feature extraction and segmentation algorithm to extract and filter features, to obtain a 2D feature map of the surface venous blood vessels;

[0013] Using a blood vessel mapping analysis algorithm to process the 2D feature map of the surface veins, obtain the extension characteristics and thickness characteristics of each segment of the blood vessel, perform three-dimensional reconstruction, and obtain a 3D model of the surface veins;

[0014] Obtaining a 3D model of the examined area;

[0015] The 3D model of the surface veins and the 3D model of the inspected part are three-dimensionally mapped into a combined model, and the venous part of the combined model is colored and mapped, and the part of the structure of the inspected part is filled with transparent material. Finally, rendering and visualization are performed to obtain a 3D imaging image of the surface veins of the inspected part.

[0016] In some embodiments of the present specification, the pre-processing filter adopts median filtering to reduce the noise of the pixel by calculating the median of the pixel values ​​in the area around the pixel, and to preserve the edge information of the image.

[0017] In some embodiments of the present specification, the deep learning image feature extraction and segmentation algorithm uses a U-Net network with a symmetric encoder-decoder structure.

[0018] In some embodiments of this specification, the specific process of preprocessing and filtering the infrared feature map of the venous blood vessels, and extracting and filtering the features using a deep learning image feature extraction and segmentation algorithm to obtain a 2D feature map of the surface venous blood vessels is as follows:

[0019] Input the venous infrared feature map, use a 3x3 convolution kernel to step along each pixel in a zigzag pattern on the venous infrared feature map, with a step length of 1 and fill with 0, calculate the grayscale value of the 8 pixels around each pixel and the pixel, sort them, and take the median as the grayscale value of the pixel at that position in the output image;

[0020] Then the filtered image enters the U-Net encoder, where a convolution operation is first performed to extract the features of the image. The convolution structure uniformly uses a 3x3 convolution kernel, k=3, p=0 padding, and a step size of s=1. The formula for calculating the output features of the convolution layer is as follows:

[0021] ;

[0022] get: ; is the output of the convolutional layer, is the convolution layer input, p is the padding, k is the convolution kernel size, and s is the step size;

[0023] After convolution, ReLU is used as the activation function to perform nonlinear mapping on the convolution output. The ReLU function is:

[0024] ;

[0025] After two convolution operations in each layer, a maximum pooling operation is performed to enter the next layer of the encoder. The kernel size of each pooling layer is k=2, the padding is p=0, and the step size is s=2, resulting in:

[0026] ;

[0027] After 5 layers of convolution and 4 times of maximum pooling, a 32x22x1024 feature map is obtained, which is fed into the U-Net decoder;

[0028] The original size of the feature map is restored through the U-Net decoder. The process consists of convolution, upsampling and skip structure. First, the feature map is upsampled by 2x2. The formula is:

[0029] ;

[0030] A feature map of 64x44x512 is obtained, and a skip connection operation is performed on the feature map. The feature map of the corresponding mirror layer is cropped and spliced ​​with the feature map from the upper layer to form a feature with more channels, that is, it is spliced ​​with the upsampled feature map from the fifth layer of the U-Net encoder to generate a feature map of size 64x44x1024. A convolution and reactivation operation is performed on the feature map. This operation is repeated 4 times to obtain a feature map of 452x292x1 in the first layer, that is, a 2D map of the surface venous blood vessel features is obtained.

[0031] In some embodiments of the present specification, the specific process of using the blood vessel mapping analysis algorithm to process the 2D feature map of the surface veins, obtain the extension characteristics and thickness characteristics of each segment of the blood vessel, perform three-dimensional reconstruction, and obtain the 3D model of the surface veins is as follows:

[0032] Sobel filtering is used to detect and extract the edges of the 2D feature map of the surface veins to obtain a binary feature map with obvious edges.

[0033] The binary feature map is skeletonized, and the central axis is extracted using the Zhang-Suen algorithm to obtain a central axis map of the venous blood vessel;

[0034] Overlapping the binary feature map and the central axis map to obtain a central axis mapping map;

[0035] Based on the central axis mapping diagram, sampling points are sequentially taken on the diagram according to the step length, and the distance between the sampling point and the edge of the venous blood vessel is obtained as the radius data of the venous blood vessel at the point;

[0036] The 3D model is reconstructed based on the central axis and radius data in the central axis diagram. The 2D central axis is first rotated and translated according to its direction information to determine the position of each vein in the 3D space. Then, according to the 2D radius data, the central axis is stretched into a tubular structure, and the tubular structures corresponding to different veins are connected to obtain a 3D model of surface veins.

[0037] In some embodiments of this specification, the process of using Sobel filtering to detect and extract edges from a 2D feature map of surface venous vessels to obtain a binary feature map with obvious edges is as follows:

[0038] The image is convolved using two 3x3 Sobel operators as follows:

[0039] ;

[0040] The feature map is regarded as a two-dimensional function. The Sobel operator is the gradient of the image in the vertical and horizontal directions. That is, the Sobel operator is a two-dimensional object. The elements of the two-dimensional object are the first-order derivatives of the function in the horizontal and vertical directions:

[0041] ;

[0042] in, Represents the gradient vector of the image, including the derivatives in the horizontal and vertical directions; Represents the first-order derivative of the image in the horizontal direction (x direction), that is, the gradient of the image in the horizontal direction (x direction), that is, the rate of change of the grayscale value; Represents the first-order derivative of the image in the vertical direction (y direction), that is, the gradient of the image in the vertical direction (y direction); Represents the transpose of a vector, converting a row vector into a column vector. In image processing, the gradient is usually expressed in the form of a column vector for ease of calculation and representation;

[0043] The Sobel operator makes a difference in pixel values ​​in the horizontal and vertical directions to obtain an approximate value of the image gradient. The two feature maps obtained by convolving the original image with two Sobel operators to obtain edge features are combined to obtain the comprehensive results in the vertical and horizontal directions. The Sobel norm is calculated as follows:

[0044] ;

[0045] in Represents the norm of the image gradient;

[0046] Calculate the approximate gradient in the horizontal and vertical directions to obtain the gradient size and direction of each pixel, binarize it, and obtain a binary feature map with obvious edges.

[0047] In some embodiments of this specification, the process of skeletonizing the binary feature map and extracting the central axis using the Zhang-Suen algorithm to obtain the central axis map of the venous blood vessel is as follows:

[0048] Based on the binary feature map, the outline of the vein is refined to a pixel width, and the Zhang-Suen algorithm is used to extract the central axis. When the algorithm is iterated, the non-zero pixels on the image are traversed. When the judgment of deleting or retaining each central pixel to be operated is made, the judgment is made based on the values ​​of the 8 pixels around it. The operator of the Zhang-Suen algorithm is as follows:

[0049] ;

[0050] Each iteration goes through two steps of operation to meet all the requirements of each stage to clear the pixels to be operated. The operation rules of stage one are:

[0051] ;

[0052] Where B(P1) represents the number of non-zero neighbors among the 8 neighbors, and A(P1) refers to how many times 0-1 changes from P2 to P8;

[0053] The operation rules of stage 2 are:

[0054] ;

[0055] The algorithm ends when no new pixels are deleted, and a central axis that can describe the characteristics of the direction of the superficial venous vessels in the examined area is obtained, that is, a central axis map of the venous vessels is obtained.

[0056] In some embodiments of this specification, the process of obtaining the 3D model of the examined part is as follows:

[0057] Self-calibrate the stereo depth camera to ensure that it can accurately capture objects in three-dimensional space;

[0058] Use a binocular depth camera to scan the inspected area to obtain a depth map;

[0059] Use feature point detection algorithm to identify and locate the various feature points of the hand;

[0060] A deep convolutional neural network is used to convert the depth map into a 3D model.

[0061] The feature points are bound to the skeleton of the generated 3D model so that when the examined part moves or changes its posture, the displayed model of the examined part can match the transformation in real time to obtain a 3D model of the examined part.

[0062] In some embodiments of this specification, the process of obtaining a 3D image of the surface venous blood vessels of the examined part is as follows:

[0063] Preprocess the 3D model of the superficial venous vessels and the 3D model of the examined part by scaling, rotating and translating them to ensure that their sizes and positions match and they can be aligned in the same coordinate system;

[0064] After preprocessing, the U-Net deep learning network is used for feature extraction to bind the 3D model of the surface venous vessels and the 3D model of the examined part;

[0065] Fine-tune the 3D model of the surface veins and the 3D model of the examined part to merge them into a model that can reflect the structure of the examined part and the distribution of the veins;

[0066] The venous blood vessels of the model are colored and mapped, and the structure of the inspected part is filled with transparent material, and then rendered and visualized to obtain a 3D image of the surface venous blood vessels of the inspected part.

[0067] The embodiment of this specification also discloses a 3D imaging system for surface venous vessels, including:

[0068] Host, with built-in processing module;

[0069] A display screen connected to the host;

[0070] The bottom background plate is provided with a bracket;

[0071] An imaging module, disposed on the bracket and connected to the host;

[0072] Wherein, the imaging module includes a near-infrared camera, a binocular depth camera and an infrared fill light;

[0073] The infrared fill light is used to emit infrared light to the surface of the inspected part; the near-infrared camera is used to obtain the infrared characteristic map of the venous blood vessels; the binocular depth camera is used to obtain the depth map; and the processing module is used to execute any of the above-mentioned 3D imaging methods for surface venous blood vessels.

[0074] The embodiments of this specification can at least achieve the following beneficial effects:

[0075] 1. Real-time imaging: Thanks to the streaming and processing functions of the system of the present invention, the final imaging can be viewed in real time, which is convenient for accurate diagnosis and treatment of skin diseases (varicose veins, etc.) and accurate implementation of venipuncture, infusion, and blood drawing. At the same time, it can also be saved and viewed again, which is convenient for accurate design of vascular reconstruction surgery.

[0076] 2. 3D imaging, harmless, non-invasive: Different from the traditional imaging method, the system of the present invention adopts the data acquisition source of near-infrared camera + depth camera, which can quickly and accurately obtain the 3D network information of the surface veins. Near-infrared light and visible light are harmless to the human body, and in principle, all opportunities that may cause direct harm to the human body are eliminated. It is suitable for children, the elderly, pregnant women and other people of different ages.

[0077] 3. Easy to operate, a single scan can obtain complete vascular network information: just connect the detection module to the host, open the imaging software, wait for the self-calibration to complete, click "Start Scan", and rotate the inspected part in the scanning area to scan, or remove the imaging module from the bracket and hold the imaging module to scan the inspected part. A single scan can obtain complete vascular network information, without multiple scanning operations. The operation is convenient, the human-computer interaction effect is good, and it can be operated by one person without professional training, which is suitable for telemedicine scenarios.

[0078] 4. High degree of visualization of results: The final results can be displayed in a variety of ways. Not only can the comprehensive model composed of the 3D model of the appearance of the examined part and the 3D model of the distribution direction of the venous vessels be viewed with high interactivity and visualization on this system software, but it also supports the output of files in different formats for easy transfer to other software for comprehensive analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0080] Figure 1 It is a schematic diagram of a method for 3D imaging of superficial venous vessels involved in some embodiments of the present invention.

[0081] Figure 2 It is a schematic diagram of a 3D imaging system of superficial venous vessels involved in some embodiments of the present invention.

[0082] Figure 3 is a schematic diagram of an imaging module involved in some embodiments of the present invention.

[0083] Figure 4 It is a schematic diagram of the operation of the imaging module involved in some embodiments of the present invention.

[0084] Figure 5a The present invention is a flowchart of a method for 3D imaging of superficial venous vessels according to some embodiments of the present invention.

[0085] Figure 5b The figure is a schematic diagram of the physical process of the method for 3D imaging of superficial venous vessels involved in some embodiments of the present invention.

[0086] Figure 6 Schematic diagram of a U-Net encoder and a U-Net decoder involved in some embodiments of the present invention.

[0087] Figure 7 It is a schematic diagram of three-dimensional modeling involved in some embodiments of the present invention.

[0088] Reference numerals:

[0089] 1. Host; 2. Display screen; 3. Imaging module; 4. Bracket; 5. Bottom background plate;

[0090] 31. Binocular depth camera; 32. Infrared fill light; 33. Near-infrared camera; 34. USB3 interface; 35. Gigabit network port; 36. USB-C interface. DETAILED DESCRIPTION

[0091] In the following, only some exemplary embodiments are briefly described. As those skilled in the art will appreciate, the described embodiments may be modified in various ways without departing from the spirit or scope of the embodiments of the present invention. Therefore, the drawings and descriptions are considered to be exemplary and non-restrictive in nature.

[0092] The disclosure below provides many different embodiments or examples to implement different structures of the embodiments of the present invention. In order to simplify the disclosure of the embodiments of the present invention, the parts and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the embodiments of the present invention. In addition, the embodiments of the present invention can repeat reference numbers and / or reference letters in different examples, and this repetition is for the purpose of simplification and clarity, which itself does not indicate the relationship between the various embodiments and / or settings discussed.

[0093] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0094] like Figure 1 As shown, the embodiment of this specification discloses a method for 3D imaging of surface venous vessels, including:

[0095] Emit infrared light to the surface of the body part being examined;

[0096] Acquire the infrared characteristic image of the venous blood vessels obtained by the infrared light reflected from the surface of the examined part;

[0097] Preprocessing and filtering the infrared feature map of the venous blood vessels, and using a deep learning image feature extraction and segmentation algorithm to extract and filter features, to obtain a 2D feature map of the surface venous blood vessels;

[0098] Using a blood vessel mapping analysis algorithm to process the 2D feature map of the surface veins, obtain the extension characteristics and thickness characteristics of each segment of the blood vessel, perform three-dimensional reconstruction, and obtain a 3D model of the surface veins;

[0099] Obtaining a 3D model of the examined area;

[0100] The 3D model of the surface veins and the 3D model of the inspected part are three-dimensionally mapped into a combined model, and the venous part of the combined model is colored and mapped, and the part of the structure of the inspected part is filled with transparent material. Finally, rendering and visualization are performed to obtain a 3D imaging image of the surface veins of the inspected part.

[0101] In some embodiments of the present specification, the pre-processing filter adopts median filtering to reduce the noise of the pixel by calculating the median of the pixel values ​​in the area around the pixel, and to preserve the edge information of the image.

[0102] In some embodiments of the present specification, the deep learning image feature extraction and segmentation algorithm uses a U-Net network with a symmetric encoder-decoder structure.

[0103] In some embodiments of this specification, the specific process of preprocessing and filtering the infrared feature map of the venous blood vessels, and extracting and filtering the features using a deep learning image feature extraction and segmentation algorithm to obtain a 2D feature map of the surface venous blood vessels is as follows:

[0104] Input the venous infrared feature map, use a 3x3 convolution kernel to step along each pixel in a zigzag pattern on the venous infrared feature map, with a step length of 1 and fill with 0, calculate the grayscale value of the 8 pixels around each pixel and the pixel, sort them, and take the median as the grayscale value of the pixel at that position in the output image;

[0105] Then the filtered image enters the U-Net encoder, where a convolution operation is first performed to extract the features of the image. The convolution structure uniformly uses a 3x3 convolution kernel, k=3, p=0 padding, and a step size of s=1. The formula for calculating the output features of the convolution layer is as follows:

[0106] ;

[0107] get: ;

[0108] is the output of the convolutional layer, is the convolution layer input, p is the padding, k is the convolution kernel size, and s is the step size;

[0109] After convolution, ReLU is used as the activation function to perform nonlinear mapping on the convolution output. The ReLU function is:

[0110] ;

[0111] After two convolution operations in each layer, a maximum pooling operation is performed to enter the next layer of the encoder. The kernel size of each pooling layer is k=2, the padding is p=0, and the step size is s=2, resulting in:

[0112] ;

[0113] After 5 layers of convolution and 4 times of maximum pooling, a 32x22x1024 feature map is obtained, which is fed into the U-Net decoder;

[0114] The original size of the feature map is restored through the U-Net decoder. The process consists of convolution, upsampling and skip structure. First, the feature map is upsampled by 2x2. The formula is:

[0115] ;

[0116] A feature map of 64x44x512 is obtained, and a skip connection operation is performed on the feature map. The feature map of the corresponding mirror layer is cropped and spliced ​​with the feature map from the upper layer to form a feature with more channels, that is, it is spliced ​​with the upsampled feature map from the fifth layer of the U-Net encoder to generate a feature map of size 64x44x1024. A convolution and reactivation operation is performed on the feature map. This operation is repeated 4 times to obtain a feature map of 452x292x1 in the first layer, that is, a 2D map of the surface venous blood vessel features is obtained.

[0117] In some embodiments of the present specification, the specific process of using the blood vessel mapping analysis algorithm to process the 2D feature map of the surface veins, obtain the extension characteristics and thickness characteristics of each segment of the blood vessel, perform three-dimensional reconstruction, and obtain the 3D model of the surface veins is as follows:

[0118] Sobel filtering is used to detect and extract the edges of the 2D feature map of the surface veins to obtain a binary feature map with obvious edges.

[0119] The binary feature map is skeletonized, and the central axis is extracted using the Zhang-Suen algorithm to obtain a central axis map of the venous blood vessel;

[0120] Overlapping the binary feature map and the central axis map to obtain a central axis mapping map;

[0121] Based on the central axis mapping diagram, sampling points are sequentially taken on the diagram according to the step length, and the distance between the sampling point and the edge of the venous blood vessel is obtained as the radius data of the venous blood vessel at the point;

[0122] The 3D model is reconstructed based on the central axis and radius data in the central axis diagram. The 2D central axis is first rotated and translated according to its direction information to determine the position of each vein in the 3D space. Then, according to the 2D radius data, the central axis is stretched into a tubular structure, and the tubular structures corresponding to different veins are connected to obtain a 3D model of surface veins.

[0123] In some embodiments of this specification, the process of using Sobel filtering to detect and extract edges from a 2D feature map of surface venous vessels to obtain a binary feature map with obvious edges is as follows:

[0124] The image is convolved using two 3x3 Sobel operators as follows:

[0125] ;

[0126] The feature map is regarded as a two-dimensional function. The Sobel operator is the gradient of the image in the vertical and horizontal directions. That is, the Sobel operator is a two-dimensional object. The elements of the two-dimensional object are the first-order derivatives of the function in the horizontal and vertical directions:

[0127] ;

[0128] Calculate the approximate gradient in the horizontal and vertical directions to obtain the gradient size and direction of each pixel, binarize it, and obtain a binary feature map with obvious edges.

[0129] In some embodiments of this specification, the process of skeletonizing the binary feature map and extracting the central axis using the Zhang-Suen algorithm to obtain the central axis map of the venous blood vessel is as follows:

[0130] Based on the binary feature map, the outline of the vein is refined to a pixel width, and the Zhang-Suen algorithm is used to extract the central axis. When the algorithm is iterated, the non-zero pixels on the image are traversed. When the judgment of deleting or retaining each central pixel to be operated is made, the judgment is made based on the values ​​of the 8 pixels around it. The operator of the Zhang-Suen algorithm is as follows:

[0131] ;

[0132] Each iteration goes through two steps of operation to meet all the requirements of each stage to clear the pixels to be operated. The operation rules of stage one are:

[0133] ;

[0134] Where B(P1) represents the number of non-zero neighbors among the 8 neighbors, and A(P1) refers to how many times 0-1 changes from P2 to P8;

[0135] The operation rules of stage 2 are:

[0136] ;

[0137] The algorithm ends when no new pixels are deleted, and a central axis that can describe the characteristics of the direction of the superficial venous vessels in the examined area is obtained, that is, a central axis map of the venous vessels is obtained.

[0138] In some embodiments of this specification, the process of obtaining the 3D model of the examined part is as follows:

[0139] Performing self-calibration on the binocular depth camera 31 to ensure that objects in the three-dimensional space can be accurately captured;

[0140] Scanning the inspected part using a binocular depth camera 31 to obtain a depth map;

[0141] Use feature point detection algorithm to identify and locate the various feature points of the hand;

[0142] A deep convolutional neural network is used to convert the depth map into a 3D model.

[0143] The feature points are bound to the skeleton of the generated 3D model so that when the examined part moves or changes its posture, the displayed model of the examined part can match the transformation in real time to obtain a 3D model of the examined part.

[0144] In some embodiments of this specification, the process of obtaining a 3D image of the surface venous blood vessels of the examined part is as follows:

[0145] Preprocess the 3D model of the superficial venous vessels and the 3D model of the examined part by scaling, rotating and translating them to ensure that their sizes and positions match and they can be aligned in the same coordinate system;

[0146] After preprocessing, the U-Net deep learning network is used for feature extraction to bind the 3D model of the surface venous vessels and the 3D model of the examined part;

[0147] Fine-tune the 3D model of the surface veins and the 3D model of the examined part to merge them into a model that can reflect the structure of the examined part and the distribution of the veins;

[0148] The venous blood vessels of the model are colored and mapped, and the structure of the inspected part is filled with transparent material, and then rendered and visualized to obtain a 3D image of the surface venous blood vessels of the inspected part.

[0149] The embodiment of this specification also discloses a 3D imaging system for surface veins, including:

[0150] Host 1, with built-in processing module;

[0151] Display screen 2, connected to the host 1;

[0152] The bottom background plate 5 is provided with a bracket 4;

[0153] An imaging module 3 is disposed on the bracket 4 and connected to the host 1;

[0154] The imaging module 3 includes a near infrared camera 33, a binocular depth camera 31 and an infrared fill light 32;

[0155] The infrared fill light 32 is used to emit infrared light to the surface of the inspected part; the near-infrared camera 33 is used to obtain the infrared characteristic map of the venous blood vessels; the binocular depth camera 31 is used to obtain the depth map; and the processing module is used to execute any of the above-mentioned 3D imaging methods for surface venous blood vessels.

[0156] The technical concept of the present invention is as follows:

[0157] The principle of the invention is mainly based on the fact that hemoglobin has two absorption peaks at wavelengths of 850nm and 760nm. Therefore, near-infrared light with a wavelength of 850nm is easily absorbed by veins. The system of the present invention uses infrared LEDs to project and obtain the surface reflection feature map of the inspected part, and obtains the 3D model of the surface veins through pre-processing filtering, deep learning feature extraction and segmentation, and analysis and reconstruction algorithms. At the same time, for the sake of intuitiveness and high visualization, the present invention maps the 3D model of the veins to the shape model of the inspected part obtained from the binocular depth camera 31, and finally realizes 3D imaging of the surface veins. The hardware of the system of the present invention mainly consists of two parts (such as Figure 2 and Figure 3 as shown):

[0158] 1. Detection module consisting of imaging module 3, bracket 4, and bottom background plate 5:

[0159] ① The bracket 4 is made of aluminum alloy and is connected to the bottom background plate 5 and the imaging module 3 by bolts. The imaging module 3 can also be removed separately for handheld scanning.

[0160] ② The bottom background plate 5 provides a good detection background for the imaging module 3 to detect.

[0161] ③The imaging module 3 mainly consists of four parts:

[0162] 1. 850nm infrared fill light 32: emits 850nm infrared light so that the near-infrared camera 33 can obtain the vein distribution and direction characteristic map of the examined part.

[0163] 2. Binocular depth camera 31: used for real-time 3D modeling of the shape of the inspected part.

[0164] 3. Near infrared camera 33: used to receive 850nm infrared light reflected from the inspected part.

[0165] 4. Data interface: The USB3 interface 34 is used to transmit data from the binocular depth camera 31 , and the Gigabit Ethernet port 35 (1 Gbps) and the USB-C interface 36 are used to transmit data from the near-infrared camera 33 .

[0166] 2. The computing (processing module) and imaging display module composed of the display screen 2 and the host 1: The host computer software of the system of the present invention has good compatibility and high flexibility for computing and imaging display modules. The computing part can use a laptop or a desktop computer; the imaging display part can use a projector or a conventional display screen.

[0167] Invention Process (eg Figure 5a and Figure 5b As shown, Figure 5a and Figure 5b① to ⑧ correspond respectively): After connecting the detection module of the system of the present invention to the host 1, open the imaging software, wait for the self-calibration to be completed, and then start scanning directly. After clicking the "Start Scan" button, place the inspected part in the scanning area and rotate the inspected part. At this time, the hemoglobin in the surface veins of the inspected part will absorb the infrared light from the infrared fill light 32, and the body surface will reflect the infrared light that is not absorbed by other parts. The near-infrared camera 33 will capture this phenomenon and obtain the original infrared feature map of the surface veins containing noise. These original infrared features of the surface veins containing noise are transmitted to the host 1 through the USB or Gigabit network port 35 for preprocessing. The original infrared feature map of the surface veins containing noise will be converted into a segmented 2D feature map of the surface veins through preprocessing filtering and deep learning image feature extraction and segmentation algorithms. Then, the vascular mapping analysis algorithm is used to obtain the extension characteristics and thickness characteristics of each segment of the blood vessel, perform 3D reconstruction, and finally obtain a 3D model of the surface veins of the inspected part. At the same time, the original binocular depth camera image acquired by the binocular depth camera 31 will be transmitted to the host 1 via USB. The host 1 will perform parallax analysis on the image and perform deep learning bone positioning analysis to obtain key node information of the inspected part, and finally output the 3D model of the inspected part. Finally, the 3D model of the surface veins and the 3D model of the inspected part are three-dimensionally mapped to obtain the 3D imaging result of the surface veins. The final result can not only be viewed after the scan is completed, but also supports real-time viewing of the 3D imaging of the inspected part on site while scanning. The entire imaging process is real-time 3D imaging, and only one scan is required to obtain the surface vein imaging of the entire inspected part. The final 3D imaging results can not only be viewed on this system software, but also support the output of files in different formats for easy transfer to other software for comprehensive analysis.

[0168] In a specific embodiment, the hardware architecture of the system of the present invention is as follows Figure 2As shown: it includes a computing and imaging display module composed of a display screen 2 and a host 1, and a detection module composed of an imaging module 3, a bracket 4, and a bottom background plate 5. There are no specific requirements for the display screen 2 and the host 1 required by the computing and imaging display module. The host 1 can be a desktop computer, a laptop, or a tablet computer with a USB interface. The display screen 2 can be a separate display screen, a projector, or a holographic projector; the computing and imaging display module composed of the display screen 2 and the host 1 can be replaced by the user's application scenario: in the case of agile diagnosis requirements, the imaging module 3 can be removed from the bracket 4 after calibration and scanned by hand, and a tablet computer or mobile phone can be used for 3D imaging detection; in the case of joint consultation requirements, a projector + desktop host can be used for analysis. The bracket 4 of the detection module includes a base with a length of 400mm and a width of 250mm and a square column with a height of 250mm. The whole body is made of 6063 aluminum alloy CNC process. Users can also change the size of the bracket 4 according to specific needs; the bottom background plate 5 is composed of a gray frosted PC plastic flat plate, which provides a good detection background for the detection of the imaging module 3 and is connected to the base of the bracket 4 through M5 bolts; the core part of the detection module is the imaging module 3, which is composed of a near-infrared camera 33, a binocular depth camera 31, an 850nm infrared fill light 32, and a data interface. The hardware architecture is as follows Figure 3 As shown, the working diagram is as follows Figure 4As shown, the CMOS model of the near-infrared camera 33 is MT9V034, and the band is visible light + 850nm narrow band, which can obtain the infrared feature map of the vein reflected by the inspected part; the exposure mode is global exposure, and the feature map of each frame can be obtained clearly, without distortion and without jello effect under dynamic recognition; wide dynamic, the sensitivity is 4.8V / lux-sec (unit light intensity can output 4.8V voltage when it shines on the sensor for 1 second), and the sensitivity of ordinary CMOS sensors is mostly 0.7-1.0, so the CMOS sensor can still accurately obtain features under different ambient lighting; the data output interface of MT9V034 is DVP, and the control interface is SCCB. In order to interact with the host 1 using USB or Ethernet, the USB2.0 programmable Phy chip of Cypress-CY7C68013 is used for data conversion, and it also supports the use of USB to configure MT9V034 online; CY7C68013 is connected to a USB-C interface 36 to connect to the host 1, and the USB-C interface 36 also provides a 5V power supply as the total power supply for the camera and other chips. The Gigabit network uses XC7K325T as the main controller for data frame format conversion and RTL8211E as the Gigabit Ethernet Phy chip. The 850nm infrared fill light 32 uses four 850nm near-infrared 7060 type lamp beads with a power of 2W. The binocular depth camera 31 uses Leapmotion, and the specific model is Leap Motion Controller. Leapmotion is an advanced motion sensor mainly used for gesture recognition and hand movement tracking. It uses infrared sensing technology and can accurately track the movement of fingers and palms. Its core technology uses a sensor array composed of two cameras and three infrared LEDs to capture the three-dimensional spatial position and posture of the hand in real time. In this system, leapmotion is connected to the host 1 through the USB3-microA interface. The entire imaging module 3 is integrated on a 6-layer FRC-PCB with a size of 80mm*67mm, and an ABS plastic shell protects important components.

[0169] When scanning begins, the MT9V034 near-infrared camera 33 obtains the infrared characteristic images of the veins reflected by the 850nm infrared fill light 32 at the inspected part. However, these characteristic images are original infrared images of the surface veins containing noise. In order to ensure the correctness of subsequent image feature extraction and segmentation, they need to be preprocessed and detailed information retained. Figure 6As shown, the present invention first performs preprocessing filtering on the original infrared image of the noisy surface veins, and then uses a deep learning image feature extraction and segmentation algorithm to extract and filter features, and finally obtains a 2D feature map of the surface veins. The preprocessing filter uses a median filter, which reduces pixel noise by calculating the median of the pixel values ​​in the area around the pixel and can preserve the edge information of the image. The deep learning image feature extraction and segmentation algorithm uses the U-Net network, which is a fully convolutional neural network architecture for image feature extraction and segmentation. The network is characterized by a symmetrical encoder-decoder structure that can effectively capture local features and contextual information in the image. The specific process is as follows:

[0170] The input original infrared image of the surface veins with noise is a grayscale image with a size of 640x480. A 3x3 convolution kernel is used to step along each pixel in a zigzag pattern on the original infrared image of the surface veins with noise, with a step length of 1 and a fill of 0. The convolution kernel will calculate the 8 pixels around the pixel and sort them with the grayscale value of the pixel and take the median as the grayscale value of the pixel at that position in the output image. The median filter will remove salt and pepper noise and Gaussian noise. Since the median filter is based on the sorting operation, it can retain the edge information in the image to a certain extent without producing too much blurring effect.

[0171] Then the filtered image enters the U-Net encoder (contraction path) with a size of 640x480. First, a convolution operation is performed to extract the features of the image. The convolution structure of the present invention uniformly uses a 3x3 convolution kernel k=3, p=0 padding, and a step size of s=1. According to the formula for calculating the output features of the convolution layer ( is the output of the convolutional layer, is the convolution layer input, p is the padding, k is the convolution kernel size, and s is the step size):

[0172] ;

[0173] get: ;

[0174] from Figure 6 It can be seen that the encoder part of the convolution layer is composed of two 3x3 convolution layers, and each convolution layer reduces the size of the feature map by 2 pixels. After convolution, ReLU is used as the activation function to perform nonlinear mapping on the convolution output. The ReLU function is:

[0175] ;

[0176] After two convolution operations in each layer, a maximum pooling operation is required to enter the next layer of the encoder. The kernel size of each pooling layer is k=2, the padding is p=0, and the step size is s=2, so we get:

[0177] ;

[0178] from Figure 6 It can be seen that the size of the 2x2 feature map is reduced to 1 / 2 of the original size after each max pooling. After 5 layers of convolution and 4 max pooling, a 32x22 feature map with 1024 channels is obtained. There is no max pooling 2x2 operation in the 5th layer, but the feature map is directly fed into the U-Net decoder (expansion path).

[0179] Next, the feature map is restored to its original size through the U-Net decoder. This process consists of convolution, upsampling, and skipping. First, the U-Net decoder performs a 2x2 upsampling on the feature map of size 32x22x1024 from the fifth layer of the U-Net encoder. The formula is:

[0180] ;

[0181] Can be obtained from Figure 6 In the U-Net decoder, we can see that the output size is doubled to 64x44, and the number of channels is 512, which is half of the original number. Then, U-Net performs a skip connection operation, cropping the feature map of the corresponding mirror layer and concatenating it with the feature map from the upper layer to form features with more channels. Figure 6 In the fourth layer of the U-Net decoder, we can see that the size of the feature map transferred from the fifth layer of the U-Net encoder is 72x52x512, which is cropped to 64x44x512 and concatenated with the upsampled feature map from the fifth layer of the U-Net encoder to generate a feature map of size 64x44x1024. Finally, the convolution operation is the same as the convolution operation of the U-Net encoder, and the feature map is convoluted and reactivated. This operation is repeated 4 times to finally obtain a 452x292x1 feature map in the first layer. At this point, a 2D map of the surface venous blood vessel features is obtained.

[0182] After obtaining the 2D image of the surface venous blood vessel characteristics, the present invention uses a blood vessel mapping analysis algorithm to obtain the extension characteristics and thickness characteristics of each blood vessel segment, performs three-dimensional reconstruction, and finally obtains a 3D model. This is also the core of the present invention. Figure 7 As shown, the specific process is as follows:

[0183] In order to obtain the extension feature, it is necessary to obtain the central axis of the vein. First, edge detection is performed to detect the boundary of the object from the 2D feature map. The present invention uses Sobel filtering for edge detection and extraction. The Sobel filter performs convolution operation on the image through two 3x3 operators (Sobel operators), as follows:

[0184] ;

[0185] If the feature map is regarded as a two-dimensional function, then the Sobel operator is the gradient of the image in the vertical and horizontal directions. It is a two-dimensional object, and the elements of the object are the first-order derivatives of the function in the horizontal and vertical directions. About and A two-dimensional function of For function The gradient of and is a function about and The partial derivative of is as follows:

[0186] ;

[0187] The Sobel operator makes a difference in pixel values ​​in the horizontal and vertical directions to obtain an approximate value of the image gradient. By using two operators to perform convolution on the original image to obtain edge features, the two feature maps obtained are combined to obtain the comprehensive results in the vertical and horizontal directions, and the Sobel norm is calculated. The calculation formula is as follows:

[0188] ;

[0189] By calculating the approximate values ​​of the gradient in the horizontal and vertical directions, we can get the gradient size and direction of each pixel. A larger gradient value usually means that the image color changes quickly, and these points may be the edges in the image. Then binarize it and finally get Figure 7 The binary feature map shown on the upper left has clear edges.

[0190] After obtaining the image obtained through edge detection, the present invention performs skeleton processing on the image with edge features to obtain the central axis of the venous blood vessel, such as Figure 7Upper right. The purpose of skeletonization is to refine the outline of the vein to a pixel width, while retaining the topological structure and geometric features of the original vein as much as possible, so as to describe the shape and direction of the vein. The present invention extracts the central axis through the Zhang-Suen algorithm, which traverses the non-zero pixels on the image during loop iteration. When determining whether to delete or retain each central pixel to be operated (P1), the algorithm makes a judgment based on the values ​​of the 8 pixels (P2-P8) around it. The operator of the Zhang-Suen algorithm is as follows:

[0191] ;

[0192] Each iteration of the Zhang-Suen algorithm requires two stages of operation (each stage contains 4 steps). Only when all the requirements of each stage are met (the four equations from step 1 to step 4 are all established) can the pixel to be operated (P1) be cleared. The operation rules of stage 1 are:

[0193] Step 1: ;

[0194] Step 2: ;

[0195] Step 3: ;

[0196] Step 4: ;

[0197] Among them, B(P1) represents the number of non-zero neighbors among the 8 neighbors, and A(P1) refers to how many times 0-1 changes from P2 to P8.

[0198] The operation rules of stage 2 are:

[0199] Step 1: ;

[0200] Step 2: ;

[0201] Step 3: ;

[0202] Step 4: ;

[0203] Steps 1 and 2 of the two stages are the same, but steps 3 and 4 are different. Steps 3 and 4 of stage 1 are to remove the pixels on the boundary lines on the east and south sides and the corner pixels on the northwest corner. Steps 3 and 4 of stage 2 are to remove the pixels on the boundary lines on the west and north sides and the corner pixels on the southeast corner. For the binary image obtained by the Sobel filter in the previous step, the Zhang-Suen algorithm needs to continuously iterate stages 1 and 2 until no new pixels are deleted in a certain stage, and the Zhang-Suen algorithm ends, and finally obtains the central axis that can describe the characteristics of the direction of the superficial venous vessels of the examined part.

[0204] After the first two steps, we obtain the boundary information image and the direction feature information image of the surface veins of the examined part. Then, we overlap the two images according to the feature points to obtain the following: Figure 7 The central axis map at the bottom right.

[0205] like Figure 7 As shown in the lower left, sampling points are then taken in sequence on the mesh central axis feature map according to the specified step length, and the distance between the sampling point and the edge of the venous blood vessel is obtained as the radius data of the venous blood vessel at the point and stored.

[0206] Finally, the extracted central axis and radius data are used to reconstruct the 3D model. First, the 2D central axis is rotated and translated according to its direction information to determine the approximate position of each vein in the 3D space. Then, according to the 2D radius data, the central axis is stretched into a tubular structure with a certain thickness. The tubular structures of different veins are then connected. At this point, a complete 3D model of the surface veins of the tested area is obtained.

[0207] At the same time, the Leapmotion binocular depth camera 31 completes the 3D scanning and modeling of the inspected part. Its processing flow is as follows:

[0208] First, the Leapmotion binocular depth camera 31 performs self-calibration to ensure that it can accurately capture objects in three-dimensional space. The binocular depth camera 31 is calibrated by self-calibrating according to internal fixed parameters (focal length, principal point position) and a predetermined distance from the background plate.

[0209] The Leapmotion binocular depth camera 31 uses binocular vision technology to calculate the disparity (the offset of the corresponding pixels of the two cameras) using the images captured by the two cameras, and then calculates the depth information of the feature points based on the disparity for deep learning stereo matching, thereby determining the position of the corresponding feature points in the image in three-dimensional space and generating a depth map.

[0210] After obtaining the position of the feature points in three-dimensional space, the feature point detection algorithm is used to identify and locate the various feature points of the hand, such as fingertips, palm center, arm joints, etc. The depth map is converted into a 3D model of the part to be tested using a deep convolutional neural network.

[0211] Finally, the obtained feature points are bound to the skeleton of the generated 3D model, so that when the examined part moves or changes its posture, the displayed examined part model can match the transformation in real time. Thus, the 3D model of the examined part is obtained.

[0212] Finally, the obtained 3D model of the surface veins and the 3D model of the examined part need to be 3D mapped to form a model to be displayed in the software. The specific process is as follows:

[0213] First, the 3D model of the superficial venous vessels and the 3D model of the examined part need to be preprocessed by scaling, rotating, and translating them to ensure that their sizes and positions match, and finally they can be aligned in the same coordinate system.

[0214] Then, the U-Net deep learning network is used to extract features from the two 3D models. These feature points are key points, edge points, etc., so as to bind the two 3D models.

[0215] After extracting the corresponding feature points of the two models, the 3D model of the surface veins and the 3D model of the inspected part are fine-tuned (translation, rotation, deformation, etc.) to merge the two models into a model that can reflect the structure of the inspected part and the distribution direction of the veins.

[0216] Finally, the venous blood vessels of the model are colored and mapped, and the structure of the inspected part is filled with transparent materials, and then the overall model is rendered and visualized. Thus, a 3D image of the surface venous blood vessels of the inspected part is obtained.

[0217] Finally, the present invention renders the 3D image of the surface veins of the examined part on the display screen 2 so that the user can analyze and process it. At the same time, the model can also be exported to common model formats such as OBJ, DICOM, STL, VTK, PLY, etc. for analysis by other analysis software.

[0218] The above-described embodiments are used to illustrate the present invention, not to limit the present invention, so changes in the exemplified values ​​or replacement of equivalent elements should still fall within the scope of the present invention.

[0219] From the above detailed description, it can be understood by those skilled in the art that the present invention can indeed achieve the aforementioned objectives and is in compliance with the provisions of the Patent Law.

[0220] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they are aware of the basic creative concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention. The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. It should be pointed out that any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

[0221] It should be noted that the above description of the relevant process is only for example and explanation, and does not limit the scope of application of this specification. For those skilled in the art, various modifications and changes can be made to the process under the guidance of this specification. However, these modifications and changes are still within the scope of this specification.

[0222] The basic concepts have been described above. Obviously, for those of ordinary skill in the art who have read this application, the above invention disclosure is only for example and does not constitute a limitation of this application. Although not explicitly stated here, those of ordinary skill in the art may make various modifications, improvements and amendments to this application. Such modifications, improvements and amendments are suggested in this application, so such modifications, improvements and amendments still belong to the spirit and scope of the exemplary embodiments of this application.

[0223] At the same time, the present application uses specific words to describe the embodiments of the present application. For example, "one embodiment", "an embodiment", and / or "some embodiments" refer to a certain feature, structure or characteristic related to at least one embodiment of the present application. Therefore, it should be emphasized and noted that "one embodiment" or "an embodiment" or "an alternative embodiment" mentioned twice or more in different positions in this specification does not necessarily refer to the same embodiment. In addition, some features, structures or characteristics in one or more embodiments of the present application can be appropriately combined.

[0224] In addition, it will be understood by those skilled in the art that various aspects of the present application may be illustrated and described by a number of patentable categories or situations, including any new and useful process, machine, product or combination of substances, or any new and useful improvement thereof. Therefore, various aspects of the present application may be implemented entirely by hardware, entirely by software (including firmware, resident software, microcode, etc.), or by a combination of hardware and software. The above hardware or software may be referred to as a "unit", "module" or "system". In addition, various aspects of the present application may take the form of a computer program product embodied in one or more computer-readable media, wherein computer-readable program code is contained therein.

[0225] The computer program code required for the operation of each part of the present application can be written in any one or more programming languages, including object-oriented programming languages ​​such as Java, Scala, Smalltalk, Eiffel, JADE, Emerald, C++, C#, VB.NET, Python, etc., conventional procedural programming languages ​​such as C programming language, VisualBasic, Fortran2103, Perl, COBOL2102, PHP, ABAP, dynamic programming languages ​​such as Python, Ruby and Groovy, or other programming languages. The program code can be run entirely on the user's computer, or run on the user's computer as an independent software package, or run partially on the user's computer and partially on the remote computer, or run entirely on the remote computer or server. In the latter case, the remote computer can be connected to the user's computer through any network form, such as a local area network (LAN) or a wide area network (WAN), or connected to an external computer (e.g., via the Internet), or in a cloud computing environment, or used as a service such as software as a service (SaaS).

[0226] In addition, unless explicitly stated in the claims, the order of the processing elements and sequences described in this application, the use of alphanumeric characters, or the use of other names are not intended to limit the order of the processes and methods of this application. Although the above disclosure discusses some invention embodiments that are currently considered useful through various examples, it should be understood that such details are only for illustrative purposes, and the attached claims are not limited to the disclosed embodiments. On the contrary, the claims are intended to cover all modifications and equivalent combinations that are consistent with the essence and scope of the embodiments of this application. For example, although the implementation of the various components described above can be embodied in a hardware device, it can also be implemented as a pure software solution, for example, installation on an existing server or mobile device.

[0227] Similarly, it should be noted that in order to simplify the description disclosed in this application and thus help understand one or more embodiments of the invention, in the above description of the embodiments of the application, multiple features are sometimes combined into one embodiment, figure or description thereof. However, this method of the application should not be interpreted as reflecting the intention that the claimed object requires more features than those explicitly stated in each claim. On the contrary, the subject of the invention should have fewer features than the above single embodiment.

Claims

1. A 3D imaging method for surface venous vessels, It is characterized in that include: Emit infrared light to the surface of the body part being examined; Acquire the infrared characteristic image of the venous blood vessels obtained by the infrared light reflected from the surface of the examined part; Preprocessing and filtering the infrared feature map of the venous blood vessels, and using a deep learning image feature extraction and segmentation algorithm to extract and filter features, to obtain a 2D feature map of the surface venous blood vessels; Using a blood vessel mapping analysis algorithm to process the 2D feature map of the surface veins, obtain the extension characteristics and thickness characteristics of each segment of the blood vessel, perform three-dimensional reconstruction, and obtain a 3D model of the surface veins; Obtaining a 3D model of the examined area; The 3D model of the surface veins and the 3D model of the inspected part are three-dimensionally mapped into a combined model, and the vein part of the combined model is colored and mapped, and the part of the structure of the inspected part is filled with transparent material, and finally rendered and visualized to obtain a 3D imaging image of the surface veins of the inspected part; The specific process of preprocessing and filtering the infrared feature map of the venous blood vessels, and extracting and filtering the features using a deep learning image feature extraction and segmentation algorithm to obtain a 2D feature map of the surface venous blood vessels is as follows: Input the venous infrared feature map, use a 3x3 convolution kernel to step along each pixel in a zigzag pattern on the venous infrared feature map, with a step length of 1 and fill with 0, calculate the grayscale value of the 8 pixels around each pixel and the pixel, sort them, and take the median as the grayscale value of the pixel at that position in the output image; Then the filtered image enters the U-Net encoder, where a convolution operation is first performed to extract the features of the image. The convolution structure uniformly uses a 3x3 convolution kernel, k=3, p=0 padding, and a step size of s=1. The formula for calculating the output features of the convolution layer is as follows: Get: n out =n in -2; n out is the output of the convolutional layer, n in is the convolution layer input, p is padding, k is the convolution kernel size, and s is the step size; after convolution, ReLU is used as the activation function to perform nonlinear mapping on the convolution output. The ReLU function is: f(x)=max(x,0); After two convolution operations in each layer, a maximum pooling operation is performed to enter the next layer of the encoder. The kernel size of each pooling layer is k=2, the padding is p=0, and the step size is s=2, resulting in: n out =n in / 2; After 5 layers of convolution and 4 times of maximum pooling, a 32x22x1024 feature map is obtained, which is fed into the U-Net decoder; The original size of the feature map is restored through the U-Net decoder. The process consists of convolution, upsampling and skip structure. First, the feature map is upsampled by 2x2. The formula is: n out =s(n in -1)-2p+k; A 64x44x512 feature map is obtained, and a skip connection operation is performed on the feature map. The feature map of the corresponding mirror layer is cropped and spliced ​​with the feature map from the upper layer to form a feature with more channels, that is, it is spliced ​​with the upsampled feature map from the fifth layer of the U-Net encoder to generate a feature map of size 64x44x1024. The feature map is convoluted and reactivated, and this operation is repeated 4 times to obtain a 452x292x1 feature map in the first layer, that is, a 2D map of the surface venous blood vessel features is obtained; The specific process of using the blood vessel mapping analysis algorithm to process the 2D feature map of the surface veins, obtain the extension characteristics and thickness characteristics of each segment of the blood vessel, perform three-dimensional reconstruction, and obtain the 3D model of the surface veins is as follows: Sobel filtering is used to detect and extract the edges of the 2D feature map of the surface veins to obtain a binary feature map with obvious edges. The binary feature map is skeletonized, and the central axis is extracted using the Zhang-Suen algorithm to obtain a central axis map of the venous blood vessel; Overlapping the binary feature map and the central axis map to obtain a central axis mapping map; Based on the central axis mapping diagram, sampling points are sequentially taken on the diagram according to the step length, and the distance between the sampling point and the edge of the venous blood vessel is obtained as the radius data of the venous blood vessel at the point; Reconstructing a 3D model based on the central axis and radius data in the central axis diagram, first rotating and translating the 2D central axis according to its direction information, thereby determining the position of each vein in the 3D space, and then stretching the central axis into a tubular structure according to the 2D radius data, and then connecting the tubular structures corresponding to different veins to obtain a 3D model of surface veins; The process of using Sobel filtering to detect and extract edges from the 2D feature map of the surface veins to obtain a binary feature map with obvious edges is as follows: The image is convolved using two 3x3 Sobel operators as follows: The feature map is regarded as a two-dimensional function. The Sobel operator is the gradient of the image in the vertical and horizontal directions. That is, the Sobel operator is a two-dimensional object. The elements of the two-dimensional object are the first-order derivatives of the function in the horizontal and vertical directions: Among them, grad(I) represents the gradient vector of the image, including the derivatives in the horizontal and vertical directions; Represents the first-order derivative of the image in the horizontal direction, that is, the rate of change of the gray value; Represents the first-order derivative of the image in the vertical direction; T represents the transpose of the vector, which converts the row vector into a column vector; The Sobel operator makes a difference in pixel values ​​in the horizontal and vertical directions to obtain an approximate value of the image gradient. The two feature maps obtained by convolving the original image with two Sobel operators to obtain edge features are combined to obtain the comprehensive results in the vertical and horizontal directions. The Sobel norm is calculated as follows: Among them, |grad(I)| represents the norm of the image gradient; Calculate the approximate values ​​of the gradient in the horizontal and vertical directions to obtain the gradient size and direction of each pixel, and binarize it to obtain a binary feature map with obvious edges; The process of skeletonizing the binary feature map and extracting the central axis using the Zhang-Suen algorithm to obtain the central axis map of the venous blood vessel is as follows: Based on the binary feature map, the outline of the vein is refined to a pixel width, and the Zhang-Suen algorithm is used to extract the central axis. When the algorithm is iterated, the non-zero pixels on the image are traversed. When the judgment of deleting or retaining each central pixel to be operated is made, the judgment is made based on the values ​​of the 8 pixels around it. The operator of the Zhang-Suen algorithm is as follows: Each iteration goes through two steps of operation to meet all the requirements of each stage to clear the pixels to be operated. The operation rules of stage one are: 2≤B(P 1 )≤6 A(P 1 )=1 P 2 ×P 4 ×P 6 =0 P 4 ×P 6 ×P 8 =0; Where B(P 1 ) represents the number of non-zero neighbors among the 8 neighbors, A(P 1 ) refers to the 2 To P 8 , 0-1 changed several times; The operation rules of stage 2 are: 2≤B(P 1 )≤6 A(P 1 )=1 P 2 ×P 4 ×P 8 =0 P 2 ×P 6 ×P 8 =0; The algorithm ends when no new pixels are deleted, and a central axis that can describe the characteristics of the direction of the superficial venous vessels in the examined area is obtained, that is, a central axis map of the venous vessels is obtained.

2. The method for 3D imaging of surface venous vessels according to claim 1, It is characterized in that The pre-processing filter adopts median filtering to reduce the noise of the pixel by calculating the median of the pixel values ​​in the area around the pixel, and can preserve the edge information of the image.

3. The method for 3D imaging of surface venous vessels according to claim 1, It is characterized in that The deep learning image feature extraction and segmentation algorithm uses a U-Net network with a symmetric encoder-decoder structure.

4. The method for 3D imaging of surface venous vessels according to claim 1, It is characterized in that The process of obtaining the 3D model of the examined part is as follows: Self-calibrate the stereo depth camera to ensure that it can accurately capture objects in three-dimensional space; Use a binocular depth camera to scan the inspected area to obtain a depth map; Use feature point detection algorithm to identify and locate the various feature points of the hand; Use deep convolutional neural networks to convert depth maps into 3D models; The feature points are bound to the skeleton of the generated 3D model so that when the examined part moves or changes its posture, the displayed model of the examined part can match the transformation in real time to obtain a 3D model of the examined part.

5. The method for 3D imaging of surface venous vessels according to claim 1, It is characterized in that The process of obtaining the 3D imaging image of the surface venous blood vessels of the examined part is as follows: Preprocess the 3D model of the superficial venous vessels and the 3D model of the examined part by scaling, rotating and translating them to ensure that their sizes and positions match and they can be aligned in the same coordinate system; Use the U-Net deep learning network to extract features in order to bind the 3D model of the surface veins and the 3D model of the examined part; Fine-tune the 3D model of the surface veins and the 3D model of the examined part to merge them into a model that can reflect the structure of the examined part and the distribution of the veins; The venous blood vessels of the model are colored and mapped, and the structure of the inspected part is filled with transparent material, and then rendered and visualized to obtain a 3D image of the surface venous blood vessels of the inspected part.

6. A 3D imaging system for superficial venous vessels, It is characterized in that include: Host, with built-in processing module; A display screen connected to the host; The bottom background plate is provided with a bracket; An imaging module, disposed on the bracket and connected to the host; Wherein, the imaging module includes a near-infrared camera, a binocular depth camera and an infrared fill light; The infrared fill light is used to emit infrared light to the surface of the inspected part; the near-infrared camera is used to obtain the infrared characteristic map of the venous blood vessels; the binocular depth camera is used to obtain the depth map; the processing module is used to execute the 3D imaging method of the surface venous blood vessels according to any one of claims 1 to 5.