Vascular puncture reference result generation method and system based on multi-modal vision, vascular identification intelligent glasses and storage medium

Abstract

The application relates to the technical field of data processing, and discloses a blood vessel puncture reference result generation method and system based on multi-modal vision, blood vessel identification intelligent glasses and a storage medium.The method comprises the following steps: acquiring multi-modal image data, performing pretreatment to obtain target multi-modal image data, and performing feature extraction on the target multi-modal image data to obtain global blood vessel feature information; performing blood vessel identification classification to obtain a superficial vein identification result, performing blood vessel depth calculation on the superficial vein identification result to obtain blood vessel depth information; extracting a blood vessel center line according to the superficial vein identification result, obtaining a target blood vessel information according to the blood vessel center line to obtain a puncture decision feature vector, and performing position confirmation to obtain a target puncture position; and obtaining a blood vessel puncture reference result according to a target needle insertion angle and a target needle insertion depth obtained from the blood vessel depth information and the target puncture position.The application can realize accurate puncture decision at low cost.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method, system, smart glasses for vascular recognition, and computer-readable storage medium for generating reference results of vascular puncture based on multimodal vision. Background Technology

[0002] While outpatient blood collection technology has gradually incorporated venous visualization devices and automated blood collection robots, traditional manual puncture remains the mainstream method. Its core drawback lies in its high dependence on operator experience. Healthcare workers must rely on vision and touch to determine vessel location, leading to a significantly higher failure rate in obese, dehydrated, dark-skinned, or pediatric patients. This results in increased risks of repeated punctures, hematoma, and heightened patient anxiety. Furthermore, operational efficiency is affected by individual skill fluctuations, making standardization difficult. Meanwhile, although automated blood collection robots excel in puncture accuracy, single-session time (<1 minute), and contactless operation, their procurement costs are prohibitively high, far exceeding the affordability of primary healthcare institutions. The systems require deep integration with hospital information systems (HIS) and laboratory information systems (LIS), are complex to maintain, and have limited adaptability to patients with extreme body types or collapsed veins, resulting in low adoption rates and difficulty in meeting large-scale, high-throughput outpatient blood collection demands. A significant gap exists between technological advancement and clinical accessibility. While venous visualization devices can improve operator confidence, they are not recommended by authoritative guidelines as an effective means to improve success rates, resulting in low cost-effectiveness and failing to substantially alleviate the problems of reliance on manual labor and efficiency bottlenecks.

[0003] Therefore, existing technologies still need to be improved and developed. Summary of the Invention

[0004] The main objective of this invention is to provide a method, system, smart glasses for vascular puncture reference result generation based on multimodal vision, and storage medium, aiming to solve the problems of high cost and inability to make accurate puncture decisions in existing vascular puncture assistance methods.

[0005] To achieve the above objectives, the present invention provides a method for generating vascular puncture reference results based on multimodal vision, the method comprising the following steps: Multimodal image data of the puncture site of the target user is acquired, the multimodal image data is preprocessed to obtain target multimodal image data, and the target multimodal image data is convolved through spatial attention mechanism and spectral attention mechanism to obtain vascular semantic features; The target multimodal image data is segmented to obtain image segmentation results. Features are extracted from the image segmentation results to obtain a global information feature matrix. Global vascular feature information is obtained based on the vascular semantic features and the global information feature matrix. The global vascular feature information is used for vascular identification and classification to obtain superficial vein identification results. The vascular depth of the superficial vein region image corresponding to the superficial vein identification results is calculated to obtain vascular depth information. Based on the superficial vein identification results, the vessel centerline is extracted, the vessel information is calculated based on the vessel centerline to obtain the target vessel information, the puncture decision feature vector is obtained based on the target vessel information, and the location is confirmed based on the puncture decision feature vector to obtain the target puncture location. The needle insertion angle is calculated based on the blood vessel depth information to obtain the target needle insertion angle. The needle insertion depth is calculated based on the blood vessel depth information to obtain the target needle insertion depth. The blood vessel puncture reference result is obtained based on the target puncture position, the target needle insertion angle, and the target needle insertion depth.

[0006] Optionally, the method for generating vascular puncture reference results based on multimodal vision, wherein acquiring multimodal image data of the puncture site of the target user, preprocessing the multimodal image data to obtain target multimodal image data, and performing convolution processing on the target multimodal image data through spatial attention and spectral attention mechanisms to obtain vascular semantic features, specifically includes: Acquire visible light image data and near-infrared image data of the puncture site of the target user, perform data registration on the visible light image data and the near-infrared image data, and obtain image data alignment results; Multimodal image data is obtained based on the image data alignment results, and noise suppression, image enhancement, modality registration and normalization are performed on the multimodal image data to obtain target multimodal image data. The target multimodal image data is convolved using a spectral attention mechanism to obtain spectral attention feature information. The target multimodal image data is then convolved using a spatial attention mechanism to obtain spatial attention feature information. Finally, vascular semantic features are obtained based on the spectral attention feature information and the spatial attention feature information. Specifically, the convolutional processing of the target multimodal image data using the spectral attention mechanism involves: ; The convolutional processing of the target multimodal image data using a spatial attention mechanism specifically involves: ; in, For spectral attention feature information, These are the spectral channel weighting coefficients. A multilayer perceptron with weight sharing. For average pooling operation, For max pooling operation, This is the result of the encoding process of the neural network model. Spatial attention feature information, This refers to the spatial channel weighting coefficient. The result of the convolution calculation. This represents the number of convolution kernels.

[0007] Optionally, the method for generating vascular puncture reference results based on multimodal vision, wherein the step of segmenting the target multimodal image data to obtain image segmentation results, extracting features from the image segmentation results to obtain a global information feature matrix, and obtaining global vascular feature information based on the vascular semantic features and the global information feature matrix, specifically includes: The target multimodal image data is segmented to obtain image segmentation results. All sub-images in the image segmentation results are flattened to obtain multiple image feature vectors. All the image feature vectors are transformed to obtain a global information feature matrix, and global vascular feature information is obtained based on the vascular semantic features and the global information feature matrix.

[0008] Optionally, the method for generating vascular puncture reference results based on multimodal vision, wherein the step of performing vascular identification and classification on the global vascular feature information to obtain superficial vein identification results specifically includes: The global vascular feature information is classified and identified at the vascular level to obtain the vascular level identification result, and the global vascular feature information is classified and identified at the vascular region level to obtain the vascular region identification result. The superficial vein identification result is obtained based on the blood vessel level identification result and the blood vessel region identification result; Specifically, the process of performing vascular-level identification and classification on the global vascular feature information includes: ; The process of classifying and identifying blood vessel regions based on the global vascular feature information specifically involves: ; in, For blood vessel level identification results, For the first Unnormalized score of vascular-like state, For the first The score for vascular-like state, This is the number of output nodes. Results of vascular region identification. This represents the probability that the target pixel belongs to the blood vessel region.

[0009] Optionally, the method for generating vascular puncture reference results based on multimodal vision, wherein the step of calculating the vascular depth of the superficial vein region image corresponding to the superficial vein identification result to obtain vascular depth information specifically includes: The superficial vein region image is obtained by acquiring images of the superficial vein region using a binocular camera, and the baseline distance of the binocular camera is also obtained. The disparity of the target vein in the superficial vein region image is calculated to obtain the target disparity, and the depth of the target vein is calculated based on the baseline distance and the target disparity to obtain the vessel depth information. Specifically, the calculation of the vessel depth of the target vein based on the baseline distance and the target disparity is as follows: ; in, For target depth, For the focal length of the binocular camera, Baseline distance, For target parallax.

[0010] Optionally, the target blood vessel information includes blood vessel length, blood vessel curvature, number of blood vessel bifurcation points, and blood vessel diameter; The process of extracting the vessel centerline based on the superficial vein identification result, calculating vessel information based on the vessel centerline to obtain target vessel information, obtaining a puncture decision feature vector based on the target vessel information, and confirming the location based on the puncture decision feature vector to obtain the target puncture location specifically includes: The superficial vein identification results are refined to obtain an initial vascular skeleton map. The vascular path is calculated on the initial vascular skeleton map according to the fast traversal algorithm to obtain the main vascular path. The vascular centerline is extracted based on the main vascular path. The Euclidean distances between all vessel nodes along the vessel centerline are calculated to obtain the distance calculation results. The Euclidean distances in the distance calculation results are then summed to obtain the vessel length. The local curvature of the vessel centerline is calculated to obtain multiple local curvatures. The vessel curvature is obtained based on all the local curvatures. The number of bifurcation points of the vessel centerline is calculated to obtain the number of vessel bifurcation points. The vessel radius is obtained by applying Euclidean distance transformation to the vessel centerline, the vessel diameter is obtained based on the vessel radius, and the puncture decision feature vector is obtained based on the vessel depth information, the vessel length, the vessel curvature, the number of vessel bifurcation points, and the vessel diameter. The puncture decision feature vector is input into the puncture success rate prediction model to obtain the puncture success rate prediction result. The score is calculated based on the puncture success rate prediction result to obtain the target score, and the area corresponding to the target score is taken as the target puncture position.

[0011] Optionally, in the method for generating reference results for vascular puncture based on multimodal vision, the step of calculating the needle insertion angle based on the vascular depth information specifically involves: ; The calculation of the needle insertion depth based on the blood vessel depth information specifically involves: ; in, The target needle insertion angle, It is the arctangent function. For vascular depth information, This is the horizontal distance from the skin surface to the projection point corresponding to the target blood vessel. To achieve the target needle insertion depth, The compensation value is adjusted based on the patient's age, subcutaneous fat thickness, and vascular elasticity.

[0012] Optionally, the method for generating vascular puncture reference results based on multimodal vision, wherein the system for generating vascular puncture reference results based on multimodal vision includes: The image data processing module is used to acquire multimodal image data of the puncture site of the target user, preprocess the multimodal image data to obtain target multimodal image data, and perform convolution processing on the target multimodal image data through spatial attention mechanism and spectral attention mechanism to obtain vascular semantic features; The feature information extraction module is used to perform segmentation processing on the target multimodal image data to obtain image segmentation results, extract features from the image segmentation results to obtain a global information feature matrix, and obtain global vascular feature information based on the vascular semantic features and the global information feature matrix. The blood vessel recognition and classification module is used to perform blood vessel recognition and classification on the global blood vessel feature information to obtain superficial vein recognition results, and to perform blood vessel depth calculation on the superficial vein region image corresponding to the superficial vein recognition results to obtain blood vessel depth information. The puncture location confirmation module is used to extract the vessel centerline based on the superficial vein identification result, calculate the vessel information based on the vessel centerline to obtain the target vessel information, obtain the puncture decision feature vector based on the target vessel information, and confirm the location based on the puncture decision feature vector to obtain the target puncture location. The puncture result reference module is used to calculate the needle insertion angle based on the blood vessel depth information to obtain the target needle insertion angle, calculate the needle insertion depth based on the blood vessel depth information to obtain the target needle insertion depth, and obtain the blood vessel puncture reference result based on the target puncture position, target needle insertion angle, and target needle insertion depth.

[0013] Furthermore, to achieve the above objectives, the present invention also provides a blood vessel recognition smart glasses, wherein the blood vessel recognition smart glasses include: a memory, a processor, and a blood vessel puncture reference result generation program based on multimodal vision stored in the memory and executable on the processor. When the blood vessel puncture reference result generation program based on multimodal vision is executed by the processor, it implements the steps of the blood vessel puncture reference result generation method based on multimodal vision as described above.

[0014] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a vascular puncture reference result generation program based on multimodal vision, and the vascular puncture reference result generation program based on multimodal vision, when executed by a processor, implements the steps of the vascular puncture reference result generation method based on multimodal vision as described above.

[0015] This invention acquires multimodal image data of the puncture site of a target user, preprocesses the multimodal image data to obtain target multimodal image data, and performs convolution processing on the target multimodal image data using spatial attention and spectral attention mechanisms to obtain vascular semantic features; segments the target multimodal image data to obtain image segmentation results, extracts features from the image segmentation results to obtain a global information feature matrix, and obtains global vascular feature information based on the vascular semantic features and the global information feature matrix; performs vascular recognition and classification on the global vascular feature information to obtain superficial vein recognition results, and then... The superficial vein region image corresponding to the superficial vein recognition result is used to calculate the vessel depth information. The vessel centerline is extracted based on the superficial vein recognition result, and vessel information is calculated based on the vessel centerline to obtain the target vessel information. A puncture decision feature vector is obtained based on the target vessel information, and the position is confirmed based on the puncture decision feature vector to obtain the target puncture position. The needle insertion angle is calculated based on the vessel depth information to obtain the target needle insertion angle, and the needle insertion depth is calculated based on the vessel depth information to obtain the target needle insertion depth. A vessel puncture reference result is obtained based on the target puncture position, target needle insertion angle, and target needle insertion depth. This invention achieves real-time vessel detection, three-dimensional localization, and puncture path planning by fusing RGB and near-infrared images, thereby enabling accurate puncture decisions at low cost. Attached Figure Description

[0016] Figure 1 This is a flowchart of a preferred embodiment of the method for generating vascular puncture reference results based on multimodal vision according to the present invention; Figure 2 These are three views of a blood vessel recognition smart glasses according to a preferred embodiment of the present invention; Figure 3 This is a structural diagram of a preferred embodiment of the vascular puncture reference result generation system based on multimodal vision of the present invention; Figure 4 This is a structural diagram of a preferred embodiment of the blood vessel recognition smart glasses of the present invention. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0018] It should be noted that if the embodiments of the present invention involve directional indicators (such as up, down, left, right, front, back, etc.), the directional indicators are only used to explain the relative positional relationship and movement of the components in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indicators will also change accordingly.

[0019] Furthermore, if the embodiments of this invention involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by this invention.

[0020] The preferred embodiment of the present invention describes a method for generating vascular puncture reference results based on multimodal vision, such as... Figure 1 As shown, the method for generating vascular puncture reference results based on multimodal vision includes the following steps: Step S10: Obtain multimodal image data of the puncture site of the target user, preprocess the multimodal image data to obtain target multimodal image data, and perform convolution processing on the target multimodal image data through spatial attention mechanism and spectral attention mechanism to obtain vascular semantic features.

[0021] Specifically, existing methods for assisting vascular puncture are too costly and fail to provide accurate puncture decisions. Therefore, this invention proposes a method for generating vascular puncture reference results based on multimodal vision. This method is implemented using vascular recognition smart glasses, whose three-view diagrams are shown below. Figure 2 As shown, the blood vessel recognition smart glasses include a multimodal image acquisition module, a data preprocessing module, a deep learning recognition module, a puncture decision module, and an augmented reality display module. The specific processing procedure is as follows: First, multimodal image data needs to be acquired through the multimodal image acquisition module. This multimodal image data acquisition module is integrated into the front end of the blood vessel recognition smart glasses and mainly consists of a visible light camera, a near-infrared camera, and a near-infrared LED supplementary lighting array. Figure 2 (not shown in the image) and image synchronization control unit ( Figure 2 Composed of (not shown), this module can simultaneously acquire visible light image data and near-infrared image data of the puncture site of the patient (i.e., the target user). The visible light image data is used to obtain information on skin texture, color features and body surface anatomical structure, while the near-infrared image data utilizes the absorption characteristics of hemoglobin to specific wavelengths of near-infrared light to visualize subcutaneous superficial veins. Since the two images originate from different spectral bands and are acquired by different sensors, the actual imaging process is affected by factors such as camera installation position, lens focal length, field of view differences, and optical distortion, resulting in positional shifts of the same blood vessel or tissue structure in visible light and near-infrared images. Therefore, registering visible light and near-infrared image data to form pixel-level spatially aligned image data (i.e., image data alignment results) is mainly achieved by obtaining the intrinsic and extrinsic parameter matrices and distortion parameters of the visible light and near-infrared cameras, establishing a unified coordinate system, and then using feature detection algorithms such as SIFT (Scale-Invariant Feature Transform) to establish the correspondence. Finally, affine transformation or perspective transformation models are used to calculate the spatial mapping relationship between the visible light and near-infrared images, mapping the near-infrared image coordinates to the visible light image coordinate system to achieve pixel-level spatial alignment. Ultimately, a set of multimodal image data containing surface tissue information and vascular information can be formed, providing high-quality input for subsequent deep learning models. Compared with single visible light imaging or traditional near-infrared vascular imaging equipment, this invention can effectively improve the clarity and recognition accuracy of vascular boundaries, and enhance the detection capability of blood vessels in obese patients, elderly patients, children, and patients with difficult veins. At the same time, multimodal image data can reduce interference caused by changes in ambient light, skin color differences, and tissue scattering, improve the stability and robustness of vascular localization, and provide a reliable data foundation for subsequent vascular segmentation, depth estimation, and precise puncture path planning.

[0022] After acquiring multimodal image data from visible and near-infrared images, preprocessing is performed to improve the accuracy and stability of subsequent vessel recognition algorithms. The preprocessing module mainly includes noise suppression, image enhancement, modal registration, and normalization. Specifically, distortion correction algorithms are used to eliminate lens imaging errors, and random noise and environmental interference are removed through median filtering, Gaussian filtering, or bilateral filtering. To address the low contrast of vessels in near-infrared images, the Frangi vessel enhancement algorithm based on the Hessian matrix is ​​used to highlight tubular structure features, while the CLAHE (Contrast Limiting Adaptive Histogram Equalization) algorithm enhances local contrast, making the outlines of small vessels clearer. Subsequently, feature point matching and spatial transformation models are used to accurately register the visible and near-infrared images, achieving spatial correspondence between different modalities at the pixel level. Finally, image normalization processing unifies the grayscale distribution and scale range of different modalities, forming standardized input data (i.e., target multimodal image data). Preprocessing can effectively reduce the impact of illumination changes, skin color differences, motion artifacts, and equipment noise on image quality, improve the continuity and recognizability of blood vessel edges, enhance the information consistency among multimodal data, and provide a reliable data foundation for subsequent deep learning models to achieve high-precision blood vessel segmentation, depth estimation, and puncture path planning.

[0023] After preprocessing the multimodal image data, the deep learning recognition stage is entered, where blood vessels are automatically detected, segmented, and their features are extracted. To balance the ability to recognize blood vessel details with the ability to perceive global structure, this invention uses an improved Attention U-Net (attention semantic segmentation model) and Vision Transformer (visual transformer network) fusion network as the core recognition model. The encoder part uses a pre-trained ResNet34 network to extract local texture features and obtains feature maps of different scales through convolution operations. Specifically, the processing involves first using the red, green, and blue bands of the visible light image, as well as the near-infrared band, as independent input channels based on the convolution module, constructing four-channel input data, and then using four 3D... Three convolutional kernels are used to perform convolution operations on the image data of the four channels to extract features from each band, thereby obtaining the deep semantic information of the input image. In the design of this convolutional module, not only are several convolutional layers set up to extract deep information from remote sensing images, but the image information extracted by the previous layer can also be processed with the current convolutional feature layer, making fuller use of both shallow and deep information, thereby improving computational efficiency.

[0024] Subsequently, during the convolution operation on the image, a Double Attention module is introduced before deconvolution. This module includes Spatial Attention (which, when added, allows the acquisition of feature layers with spatially distributed weights from the image) and Spectral Attention (which, when added, allows the acquisition of feature layers with channel-distributed weights from the multispectral image). Using these two attention mechanisms together allows for the full utilization of the spectral and spatial information in the image. In this module, the two attention mechanisms are configured in parallel. When executing Spectral Attention, the weights of each spectrum are obtained, thus generating spectral domain attention. That is, the target multimodal image data is convolved using the spectral attention mechanism to obtain spectral attention feature information, the corresponding expression of which is: ; in, For spectral attention feature information, These are the spectral channel weighting coefficients. A multilayer perceptron with weight sharing. For average pooling operation, For max pooling operation, This is the result of the encoding process of the neural network model; unlike Spectral Attention, Spatial Attention mainly focuses on contextual location information. First, it uses MaxPooling and AveragePooling on the spectral channel dimension to obtain two different feature layers. Then, the two feature layers are merged, and convolution operations are used to generate a Spatial Attention Map (spatial attention feature information). The corresponding expression is: ; in, Spatial attention feature information, This refers to the spatial channel weighting coefficient. The result of the convolution calculation. The number of convolution kernels is used; then, vascular semantic features are obtained based on the spectral attention feature information and the spatial attention feature information.

[0025] Step S20: Perform segmentation processing on the target multimodal image data to obtain image segmentation results, extract features from the image segmentation results to obtain a global information feature matrix, and obtain global vascular feature information based on the vascular semantic features and the global information feature matrix.

[0026] Specifically, after obtaining the semantic features of blood vessels, the visual converter network module of the core recognition model focuses on acquiring features related to global information. Global information typically requires analysis and learning of the entire dataset (i.e., the target multimodal image data) to obtain global patterns and rules. The specific processing involves first segmenting the multimodal image data of the blood vessel region to improve computational efficiency. For example, the multimodal image data can be segmented... according to The image is divided into 1024 patches (sub-images) based on its size. Each patch is a small, multi-channel image with the number of channels equal to the number of spectra. The size and dimensions of each patch are as follows: Flattening it out results in a piece of length [length missing]. The vector is a one-dimensional sequence of numbers formed by arranging all pixel values ​​in a patch in a fixed order. Then, each vector is treated as a separate input, resulting in a total of 1024 vectors, which can be converted into a single... The matrix is ​​used as parallel input, where each row corresponds to the local feature representation of an image patch, and each column corresponds to a specific feature dimension. By constructing this matrix, the model can simultaneously calculate the correlation between all patches using an attention mechanism, achieving global feature interaction and information fusion, thereby establishing correlations between distant vascular regions and improving the ability to identify blood vessels. Subsequently, global vascular feature information is obtained based on the vascular semantic features and the global information feature matrix.

[0027] Step S30: Perform vascular identification and classification on the global vascular feature information to obtain superficial vein identification results, and calculate the vascular depth of the superficial vein region image corresponding to the superficial vein identification results to obtain vascular depth information.

[0028] Specifically, after obtaining the global vascular feature information, it can be input into the Sigmoid classification module and the Softmax classification module for classification calculation. For multi-class tasks involving vascular grading, the Softmax classifier is used to classify the target vascular vessels into multiple categories, outputting the probability distribution of different vascular grades to obtain vascular grade recognition results. For the segmentation task between vascular and non-vascular regions, the Sigmoid activation function is used to construct a pixel-level binary classification output layer to generate a vascular probability map to obtain vascular region recognition results. The specific processing procedure is as follows: In the Softmax classification module, the Softmax function is used to automatically identify, classify, and grade the global vascular feature information. The corresponding expression is: ; in, For blood vessel level identification results, For the first Unnormalized score of vascular-like state, For the first The score for vascular-like state, The number of output nodes; the Softmax function approaches a smooth state as it approaches positive or negative infinity, and because the output range of the Softmax function is (0, 1) and the sum of all output values ​​is 1, this function is chosen in this module for probability statistics of multi-classification (classification of blood vessels of different levels).

[0029] In the Sigmoid classification module, the Sigmoid function is used to automatically identify and classify global vascular feature information (here it is binary classification, only dividing into vascular regions and non-vascular regions). The corresponding expression is: ; in, For blood vessel level identification results, For the first Unnormalized score of vascular-like state, For the first The score for vascular-like state, This is the number of output nodes. Results of vascular region identification. The probability that the target pixel belongs to a blood vessel region is given. Then, based on the blood vessel level recognition result and the blood vessel region recognition result, the superficial vein recognition result is obtained. This invention, through an improved deep learning recognition scheme, can achieve high-precision automatic recognition of superficial veins, and its vascular detection capability for children, obese patients, and patients with difficult veins is significantly better than traditional near-infrared imaging equipment, effectively improving the accuracy of blood vessel localization.

[0030] Next, it is necessary to calculate the vessel depth of the target vein in the superficial vein recognition result. Specifically, in this embodiment of the invention, binocular vision depth perception technology is used. Images of the superficial vein recognition result area are simultaneously acquired by left and right cameras to obtain superficial vein region images. A stereo matching algorithm is used to calculate the disparity of the vessel region, and combined with the vessel depth calculation formula, the vessel depth is calculated, providing a spatial positioning basis for subsequent puncture path planning and augmented reality visualization. The vessel recognition smart glasses of this invention have a visible light and a near-infrared binocular camera installed on each of the left and right sides of the frame. The distance between the two cameras is called the baseline distance, denoted as . Suppose that a vein on the back of the target user's hand is located at a specific point in space. Then the left camera obtains the blood vessel point. The position is ,in, This represents the horizontal coordinates of the blood vessel point captured by the left camera within the left camera image. The vertical coordinates of the blood vessel point in the image captured by the left camera are: [Image 1]; the position of the blood vessel point obtained by the right camera is: [Image 2] ,in, This refers to the horizontal coordinates of the blood vessel point captured by the right camera within the image from the right camera. The vertical coordinates of the blood vessel point captured by the right camera in the image of the right camera.

[0031] Due to different viewing angles, the lateral coordinates of the same blood vessel point will shift in the left and right images. Therefore, parallax calculation is performed on the target vein in the superficial vein region image to obtain the target parallax. The corresponding expression is: ; Wherein, the greater the parallax, the closer the target; the smaller the parallax, the farther the target; then, based on the baseline distance and the target parallax, the vessel depth of the target vein is calculated to obtain vessel depth information, the corresponding expression being: ; in, For target depth, For the focal length of the binocular camera, Baseline distance, For target parallax.

[0032] Step S40: Extract the vessel centerline based on the superficial vein identification result, calculate the vessel information based on the vessel centerline to obtain the target vessel information, obtain the puncture decision feature vector based on the target vessel information, and confirm the location based on the puncture decision feature vector to obtain the target puncture location.

[0033] Specifically, after obtaining the superficial vein recognition results and blood vessel depth information, it is necessary to extract the blood vessel centerline. The corresponding extraction process is as follows: In this embodiment of the invention, the superficial vein recognition results are first refined using a skeletonization algorithm. The purpose of skeletonization is to gradually strip away the blood vessel boundary pixels while maintaining the original topology and connectivity of the blood vessels, so that the blood vessel region is compressed from the original multi-pixel width to a single-pixel width central skeleton. Then, a skeleton extraction algorithm based on distance transformation is used to iteratively delete redundant edge pixels and retain the blood vessel central axis region, thereby obtaining the initial blood vessel skeleton map. After obtaining the initial vascular skeleton map, the Fast Marching Method (FMM) is used to calculate the vascular trunk path. First, the initial vascular skeleton map is represented as a connected network composed of multiple skeleton points, and the starting endpoint of the vessel is set as the propagation source point. FMM simulates the shortest propagation path from the starting endpoint along the vascular skeleton by solving the wavefront propagation process, and calculates the cumulative propagation cost from each skeleton point to the source point. The propagation speed can be weighted according to the local radius of the vessel, the continuity of the vessel, and the confidence level in the vascular probability map, so that the wavefront preferentially propagates along the central region of the vessel. When the wavefront reaches another endpoint, the propagation trajectory is traced back according to the principle of minimum cumulative cost, thereby obtaining the optimal central path connecting the starting and ending endpoints. Finally, the vascular trunk centerline (i.e., the vascular trunk path) composed of continuous center points is obtained, and the corresponding expression is: ; Wherein, C is the set of vascular centerlines, representing the main vascular path obtained by the vascular skeleton extraction algorithm. It consists of several continuous spatial coordinate points and is used to characterize the direction, length and curvature of the blood vessel in three-dimensional space. It also serves as an important basis for subsequent vascular quality assessment, optimal puncture point selection and needle insertion path planning. The spatial coordinates of the point on the center line of the blood vessel. For the first Two-dimensional planar coordinates of each vascular node For the first The depth location of each vascular node.

[0034] The Euclidean distances between all vessel nodes along the vessel centerline are calculated to obtain the distance calculation results. Then, all Euclidean distances in the distance calculation results are summed to obtain the vessel length. The corresponding expression is: ; in, These are the two-dimensional planar coordinates of adjacent vascular nodes. This represents the depth location of adjacent vascular nodes.

[0035] vascular curvature To reflect the degree of vascular tortuosity, during puncture, the segment of the vessel with the smallest curvature and the straightest direction is usually selected first. First, the local curvature of the vessel's centerline is calculated, resulting in multiple local curvatures. The corresponding calculation process involves selecting three consecutive sampling points along the vessel's centerline, respectively... , and Construct two tangent vectors from these three sampling points. and The corresponding expression is: ; ; Next, calculate the angle between the two tangent vectors. The corresponding expression is: ; Then, the local curvature is calculated based on the included angle. The corresponding expression is: ; in, The distance between adjacent sampling points is used as the basis for determining the vessel curvature. Finally, the vessel curvature is obtained based on all the local curvatures described. The corresponding expression is: ; A greater curvature indicates a more pronounced tortuosity of the blood vessel. The number of bifurcation points along the centerline of the blood vessel is calculated to obtain the total number of bifurcation points. The corresponding expression is: ; in, This is an indicator function that takes the value 1 if the condition is met, and 0 otherwise. This represents the number of adjacent segments at a certain node on the blood vessel centerline. For example, if... This indicates that the node is an endpoint and there is no branching. This indicates that the node is a node on a normal blood vessel line and there is no branching. This indicates that the node is a three-way pipeline node, exhibiting a branching phenomenon. This indicates that the node is a cross-shaped pipeline node, and there is a branching phenomenon.

[0036] Next, the Euclidean distance is applied to the vessel centerline to obtain the vessel radius, where the vessel radius is the shortest distance from the vessel centerline to the vessel boundary. The vessel diameter is then calculated from the vessel radius, and a puncture decision feature vector is obtained based on the vessel depth, vessel length, vessel curvature, number of vessel bifurcation points, and vessel diameter. The corresponding expression is: ; in, The diameter of the blood vessel. This provides information on the depth of blood vessels.

[0037] Then, the puncture decision feature vector is input into the puncture success rate prediction model to obtain the puncture success rate prediction result. The puncture success rate prediction model is obtained by training the puncture success rate prediction network of the constructed XGBoost model using historical puncture data, vascular feature data, and clinical experience rules as training samples. The corresponding expression is: ; in, The result is a prediction of the success rate of puncture into the target blood vessel. This is the trained puncture success rate prediction model. Subsequently, the system automatically selects the vessel segment with the highest success rate that also meets the requirements for vessel diameter, depth, and continuous length as the recommended target region. To avoid selecting areas near vessel bifurcation points or areas with high curvature, a risk penalty function is set for scoring; the corresponding expression is: ; in, For the scoring results, , and All of these are weighting coefficients; finally, the overall score will be calculated. The area with the highest score (i.e., the target score) is the optimal puncture site (i.e., the target puncture site).

[0038] Step S50: Calculate the needle insertion angle based on the blood vessel depth information to obtain the target needle insertion angle; calculate the needle insertion depth based on the blood vessel depth information to obtain the target needle insertion depth; and obtain the blood vessel puncture reference result based on the target puncture position, target needle insertion angle, and target needle insertion depth.

[0039] Specifically, after obtaining the target puncture location, a recommended needle insertion angle is calculated based on the vessel centerline direction vector and spatial depth information. The target needle insertion angle is obtained by establishing the spatial geometric relationship between the puncture needle trajectory and the vessel axis, i.e., calculating the needle insertion angle based on the vessel depth information. The corresponding expression is: ; in, The target needle insertion angle, It is the arctangent function. This provides the vessel depth information; simultaneously, the needle insertion depth is calculated based on the vessel depth information to obtain the target needle insertion depth, and the corresponding expression is: ; in, For the target needle insertion depth, The compensation value is adjusted based on the patient's age, subcutaneous fat thickness, and vascular elasticity; the reference result for vascular puncture is obtained based on the target puncture location, target needle insertion angle, and target needle insertion depth.

[0040] Furthermore, this invention can transmit the vascular puncture reference results to the display of the vascular recognition smart glasses in real time, and overlay them graphically through the augmented reality (AR) display module to achieve visualization. This invention uses augmented reality display technology as the human-computer interaction terminal. Through the built-in camera, inertial measurement unit, and spatial perception module of the vascular recognition smart glasses, it acquires information on the patient's limb position and the wearer's perspective changes in real time, and uses visual synchronous positioning and mapping technology to establish a local spatial coordinate system, achieving precise spatial registration between the virtual vascular model and the real patient's limb. It also employs a layered display strategy, distinguishing the main blood vessels, secondary blood vessels, and recommended target blood vessels with different colors and transparency, while highlighting the optimal puncture position and displaying the recommended needle insertion direction and path with arrows or trajectory lines. In addition, the system introduces a dynamic target tracking and posture compensation mechanism. When the patient's limb moves slightly or the clinical medical staff's observation angle changes, the virtual image position can be automatically updated to ensure that the displayed content always remains consistent with the real blood vessel position, avoiding image drift and positioning deviation. In response to potential obstructions and changes in lighting in complex clinical environments, the system can also dynamically adjust the display brightness, transparency, and display level based on environmental information to ensure that critical puncture information is clearly visible.

[0041] Furthermore, such as Figure 3 As shown, based on the above-mentioned method for generating vascular puncture reference results based on multimodal vision, the present invention also provides a system for generating vascular puncture reference results based on multimodal vision, wherein the system for generating vascular puncture reference results based on multimodal vision includes: The image data processing module 51 is used to acquire multimodal image data of the puncture site of the target user, preprocess the multimodal image data to obtain target multimodal image data, and perform convolution processing on the target multimodal image data through spatial attention mechanism and spectral attention mechanism to obtain vascular semantic features. The feature information extraction module 52 is used to perform segmentation processing on the target multimodal image data to obtain image segmentation results, extract features from the image segmentation results to obtain a global information feature matrix, and obtain global vascular feature information based on the vascular semantic features and the global information feature matrix. The blood vessel recognition and classification module 53 is used to perform blood vessel recognition and classification on the global blood vessel feature information to obtain superficial vein recognition results, and to perform blood vessel depth calculation on the superficial vein region image corresponding to the superficial vein recognition results to obtain blood vessel depth information. The puncture location confirmation module 54 is used to extract the vessel centerline based on the superficial vein identification result, calculate the vessel information based on the vessel centerline to obtain the target vessel information, obtain the puncture decision feature vector based on the target vessel information, and confirm the location based on the puncture decision feature vector to obtain the target puncture location. The puncture result reference module 55 is used to calculate the needle insertion angle based on the blood vessel depth information to obtain the target needle insertion angle, calculate the needle insertion depth based on the blood vessel depth information to obtain the target needle insertion depth, and obtain the blood vessel puncture reference result based on the target puncture position, the target needle insertion angle, and the target needle insertion depth.

[0042] Furthermore, such as Figure 4 As shown, based on the above-mentioned method for generating reference results of vascular puncture based on multimodal vision, the present invention also provides a smart glasses for vascular recognition, which includes a processor 10, a memory 20 and a display 30. Figure 4 Only some components of the blood vessel recognition smart glasses are shown; however, it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0043] In some embodiments, the memory 20 may be an internal storage unit of the blood vessel recognition smart glasses, such as a hard drive or memory of the blood vessel recognition smart glasses. In other embodiments, the memory 20 may be an external storage device of the blood vessel recognition smart glasses, such as a plug-in hard drive, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the blood vessel recognition smart glasses. Further, the memory 20 may include both internal storage units and external storage devices of the blood vessel recognition smart glasses. The memory 20 is used to store application software and various types of data installed on the blood vessel recognition smart glasses, such as the program code for installing the blood vessel recognition smart glasses. The memory 20 may also be used to temporarily store data that has been output or will be output. In one embodiment, the memory 20 stores a blood vessel puncture reference result generation program 40 based on multimodal vision, which can be executed by the processor 10 to implement the blood vessel puncture reference result generation method based on multimodal vision in this application.

[0044] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor, or other data processing chip, used to run program code stored in the memory 20 or process data, such as executing the method for generating reference results for vascular puncture based on multimodal vision.

[0045] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display 30 is used to display information from the blood vessel recognition smart glasses and to display a visual user interface.

[0046] In one embodiment, when the processor 10 executes the multimodal vision-based vascular puncture reference result generation program 40 in the memory 20, the following steps are performed: Multimodal image data of the puncture site of the target user is acquired, the multimodal image data is preprocessed to obtain target multimodal image data, and the target multimodal image data is convolved through spatial attention mechanism and spectral attention mechanism to obtain vascular semantic features; The target multimodal image data is segmented to obtain image segmentation results. Features are extracted from the image segmentation results to obtain a global information feature matrix. Global vascular feature information is obtained based on the vascular semantic features and the global information feature matrix. The global vascular feature information is used for vascular identification and classification to obtain superficial vein identification results. The vascular depth of the superficial vein region image corresponding to the superficial vein identification results is calculated to obtain vascular depth information. Based on the superficial vein identification results, the vessel centerline is extracted, the vessel information is calculated based on the vessel centerline to obtain the target vessel information, the puncture decision feature vector is obtained based on the target vessel information, and the location is confirmed based on the puncture decision feature vector to obtain the target puncture location. The needle insertion angle is calculated based on the blood vessel depth information to obtain the target needle insertion angle. The needle insertion depth is calculated based on the blood vessel depth information to obtain the target needle insertion depth. The blood vessel puncture reference result is obtained based on the target puncture position, the target needle insertion angle, and the target needle insertion depth.

[0047] Specifically, the process of acquiring multimodal image data of the puncture site of the target user, preprocessing the multimodal image data to obtain target multimodal image data, and performing convolution processing on the target multimodal image data using spatial attention and spectral attention mechanisms to obtain vascular semantic features includes: Acquire visible light image data and near-infrared image data of the puncture site of the target user, perform data registration on the visible light image data and the near-infrared image data, and obtain image data alignment results; Multimodal image data is obtained based on the image data alignment results, and noise suppression, image enhancement, modality registration and normalization are performed on the multimodal image data to obtain target multimodal image data. The target multimodal image data is convolved using a spectral attention mechanism to obtain spectral attention feature information. The target multimodal image data is then convolved using a spatial attention mechanism to obtain spatial attention feature information. Finally, vascular semantic features are obtained based on the spectral attention feature information and the spatial attention feature information. Specifically, the convolutional processing of the target multimodal image data using the spectral attention mechanism involves: ; The convolutional processing of the target multimodal image data using a spatial attention mechanism specifically involves: ; in, For spectral attention feature information, These are the spectral channel weighting coefficients. A multilayer perceptron with weight sharing. For average pooling operation, For max pooling operation, This is the result of the encoding process of the neural network model. Spatial attention feature information, This refers to the spatial channel weighting coefficient. The result of the convolution calculation. This represents the number of convolution kernels.

[0048] Specifically, the process of segmenting the target multimodal image data to obtain image segmentation results, extracting features from the image segmentation results to obtain a global information feature matrix, and obtaining global vascular feature information based on the vascular semantic features and the global information feature matrix includes: The target multimodal image data is segmented to obtain image segmentation results. All sub-images in the image segmentation results are flattened to obtain multiple image feature vectors. All the image feature vectors are transformed to obtain a global information feature matrix, and global vascular feature information is obtained based on the vascular semantic features and the global information feature matrix.

[0049] Specifically, the step of performing vascular identification and classification on the global vascular feature information to obtain superficial vein identification results includes: The global vascular feature information is classified and identified at the vascular level to obtain the vascular level identification result, and the global vascular feature information is classified and identified at the vascular region level to obtain the vascular region identification result. The superficial vein identification result is obtained based on the blood vessel level identification result and the blood vessel region identification result; Specifically, the process of performing vascular-level identification and classification on the global vascular feature information includes: ; The process of classifying and identifying blood vessel regions based on the global vascular feature information specifically involves: ; in, For blood vessel level identification results, For the first Unnormalized score of vascular-like state, For the first The score for vascular-like state, This is the number of output nodes. Results of vascular region identification. This represents the probability that the target pixel belongs to the blood vessel region.

[0050] Specifically, the step of calculating the vessel depth of the superficial vein region image corresponding to the superficial vein identification result to obtain vessel depth information includes: The superficial vein region image is obtained by acquiring images of the superficial vein region using a binocular camera, and the baseline distance of the binocular camera is also obtained. The disparity of the target vein in the superficial vein region image is calculated to obtain the target disparity, and the depth of the target vein is calculated based on the baseline distance and the target disparity to obtain the vessel depth information. Specifically, the calculation of the vessel depth of the target vein based on the baseline distance and the target disparity is as follows: ; in, For target depth, For the focal length of the binocular camera, Baseline distance, For target parallax.

[0051] The target blood vessel information includes blood vessel length, blood vessel curvature, number of blood vessel bifurcation points, and blood vessel diameter; The process of extracting the vessel centerline based on the superficial vein identification result, calculating vessel information based on the vessel centerline to obtain target vessel information, obtaining a puncture decision feature vector based on the target vessel information, and confirming the location based on the puncture decision feature vector to obtain the target puncture location specifically includes: The superficial vein identification results are refined to obtain an initial vascular skeleton map. The vascular path is calculated on the initial vascular skeleton map according to the fast traversal algorithm to obtain the main vascular path. The vascular centerline is extracted based on the main vascular path. The Euclidean distances between all vessel nodes along the vessel centerline are calculated to obtain the distance calculation results. The Euclidean distances in the distance calculation results are then summed to obtain the vessel length. The local curvature of the vessel centerline is calculated to obtain multiple local curvatures. The vessel curvature is obtained based on all the local curvatures. The number of bifurcation points of the vessel centerline is calculated to obtain the number of vessel bifurcation points. The vessel centerline is transformed by Euclidean distance to obtain the vessel radius, the vessel diameter is obtained based on the vessel radius, and the puncture decision feature vector is obtained based on the vessel depth information, the vessel length, the vessel curvature, the number of vessel bifurcation points, and the vessel diameter. The puncture decision feature vector is input into the puncture success rate prediction model to obtain the puncture success rate prediction result. The score is calculated based on the puncture success rate prediction result to obtain the target score, and the area corresponding to the target score is taken as the target puncture position.

[0052] Specifically, calculating the needle insertion angle based on the blood vessel depth information involves: ; The calculation of the needle insertion depth based on the blood vessel depth information specifically involves: ; in, The target needle insertion angle, It is the arctangent function. For vascular depth information, This is the horizontal distance from the skin surface to the projection point corresponding to the target blood vessel. For the target needle insertion depth, The compensation value is adjusted based on the patient's age, subcutaneous fat thickness, and vascular elasticity.

[0053] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a vascular puncture reference result generation program based on multimodal vision, and the vascular puncture reference result generation program based on multimodal vision, when executed by a processor, implements the steps of the vascular puncture reference result generation method based on multimodal vision as described above.

[0054] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0055] Of course, those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware (such as a processor, controller, etc.). The program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The computer-readable storage medium can be a memory, magnetic disk, optical disk, etc.

[0056] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A method for generating reference results for vascular puncture based on multimodal vision, characterized in that, The method for generating reference results for vascular puncture based on multimodal vision includes: Multimodal image data of the puncture site of the target user is acquired, the multimodal image data is preprocessed to obtain target multimodal image data, and the target multimodal image data is convolved through spatial attention mechanism and spectral attention mechanism to obtain vascular semantic features; The target multimodal image data is segmented to obtain image segmentation results. Features are extracted from the image segmentation results to obtain a global information feature matrix. Global vascular feature information is obtained based on the vascular semantic features and the global information feature matrix. The global vascular feature information is used for vascular identification and classification to obtain superficial vein identification results. The vascular depth of the superficial vein region image corresponding to the superficial vein identification results is calculated to obtain vascular depth information. The superficial vein identification results are refined to obtain an initial vascular skeleton map. The vascular path is calculated on the initial vascular skeleton map according to the fast traversal algorithm to obtain the main vascular path. The vascular centerline is extracted according to the main vascular path. The Euclidean distance between all vascular nodes in the vascular centerline is calculated to obtain the distance calculation result. All Euclidean distances in the distance calculation result are accumulated to obtain the vascular length. The local curvature of the vessel centerline is calculated to obtain multiple local curvatures. The vessel curvature is obtained based on all the local curvatures. The number of bifurcation points of the vessel centerline is calculated to obtain the number of vessel bifurcation points. The Euclidean distance of the vessel centerline is transformed to obtain the vessel radius. The vessel diameter is obtained based on the vessel radius. Finally, a puncture decision feature vector is obtained based on the vessel depth information, the vessel length, the vessel curvature, the number of vessel bifurcation points, and the vessel diameter. The puncture decision feature vector is input into the puncture success rate prediction model to obtain the puncture success rate prediction result. The score is calculated based on the puncture success rate prediction result to obtain the target score, and the area corresponding to the target score is taken as the target puncture position. The needle insertion angle is calculated based on the blood vessel depth information to obtain the target needle insertion angle. The needle insertion depth is calculated based on the blood vessel depth information to obtain the target needle insertion depth. The blood vessel puncture reference result is obtained based on the target puncture position, the target needle insertion angle, and the target needle insertion depth.

2. The method for generating vascular puncture reference results based on multimodal vision according to claim 1, characterized in that, The process involves acquiring multimodal image data of the puncture site of the target user, preprocessing the multimodal image data to obtain target multimodal image data, and then performing convolution processing on the target multimodal image data using spatial attention and spectral attention mechanisms to obtain vascular semantic features. Specifically, this includes: Acquire visible light image data and near-infrared image data of the puncture site of the target user, perform data registration on the visible light image data and the near-infrared image data, and obtain image data alignment results; Multimodal image data is obtained based on the image data alignment results, and noise suppression, image enhancement, modality registration and normalization are performed on the multimodal image data to obtain target multimodal image data. The target multimodal image data is convolved using a spectral attention mechanism to obtain spectral attention feature information. The target multimodal image data is then convolved using a spatial attention mechanism to obtain spatial attention feature information. Finally, vascular semantic features are obtained based on the spectral attention feature information and the spatial attention feature information. Specifically, the convolutional processing of the target multimodal image data using the spectral attention mechanism involves: ； The convolutional processing of the target multimodal image data using a spatial attention mechanism specifically involves: ； in, For spectral attention feature information, These are the spectral channel weighting coefficients. A multilayer perceptron with weight sharing. For average pooling operation, For max pooling operation, This is the result of the encoding process of the neural network model. Spatial attention feature information, This refers to the spatial channel weighting coefficient. The result of the convolution calculation. This represents the number of convolution kernels.

3. The method for generating vascular puncture reference results based on multimodal vision according to claim 1, characterized in that, The process of segmenting the target multimodal image data to obtain image segmentation results, extracting features from the image segmentation results to obtain a global information feature matrix, and obtaining global vascular feature information based on the vascular semantic features and the global information feature matrix specifically includes: The target multimodal image data is segmented to obtain image segmentation results. All sub-images in the image segmentation results are flattened to obtain multiple image feature vectors. All the image feature vectors are transformed to obtain a global information feature matrix, and global vascular feature information is obtained based on the vascular semantic features and the global information feature matrix.

4. The method for generating vascular puncture reference results based on multimodal vision according to claim 1, characterized in that, The step of performing vascular identification and classification on the global vascular feature information to obtain superficial vein identification results specifically includes: The global vascular feature information is classified and identified at the vascular level to obtain the vascular level identification result, and the global vascular feature information is classified and identified at the vascular region level to obtain the vascular region identification result. The superficial vein identification result is obtained based on the blood vessel level identification result and the blood vessel region identification result; Specifically, the process of performing vascular-level identification and classification on the global vascular feature information includes: ； The process of classifying and identifying blood vessel regions based on the global vascular feature information specifically involves: ； in, For blood vessel level identification results, For the first Unnormalized score of vascular-like state, For the first The score for vascular-like state, This is the number of output nodes. Results of vascular region identification. This represents the probability that the target pixel belongs to the blood vessel region.

5. The method for generating vascular puncture reference results based on multimodal vision according to claim 1, characterized in that, The step of calculating the vessel depth of the superficial vein region image corresponding to the superficial vein identification result to obtain vessel depth information specifically includes: The superficial vein region image is obtained by acquiring images of the superficial vein region using a binocular camera, and the baseline distance of the binocular camera is also obtained. The disparity of the target vein in the superficial vein region image is calculated to obtain the target disparity, and the depth of the target vein is calculated based on the baseline distance and the target disparity to obtain the vessel depth information. Specifically, the calculation of the vessel depth of the target vein based on the baseline distance and the target disparity is as follows: ； in, For target depth, For the focal length of the binocular camera, Baseline distance, For target parallax.

6. The method for generating vascular puncture reference results based on multimodal vision according to claim 1, characterized in that, The calculation of the needle insertion angle based on the blood vessel depth information specifically involves: ； The calculation of the needle insertion depth based on the blood vessel depth information specifically involves: ； in, The target needle insertion angle, It is the arctangent function. For vascular depth information, This is the horizontal distance from the skin surface to the projection point corresponding to the target blood vessel. For the target needle insertion depth, The compensation value is adjusted based on the patient's age, subcutaneous fat thickness, and vascular elasticity.

7. A system for generating reference results for vascular puncture based on multimodal vision, characterized in that, The multimodal vision-based vascular puncture reference result generation system is used to implement the multimodal vision-based vascular puncture reference result generation method according to any one of claims 1-6, wherein the multimodal vision-based vascular puncture reference result generation system comprises: The image data processing module is used to acquire multimodal image data of the puncture site of the target user, preprocess the multimodal image data to obtain target multimodal image data, and perform convolution processing on the target multimodal image data through spatial attention mechanism and spectral attention mechanism to obtain vascular semantic features; The feature information extraction module is used to perform segmentation processing on the target multimodal image data to obtain image segmentation results, extract features from the image segmentation results to obtain a global information feature matrix, and obtain global vascular feature information based on the vascular semantic features and the global information feature matrix. The blood vessel recognition and classification module is used to perform blood vessel recognition and classification on the global blood vessel feature information to obtain superficial vein recognition results, and to perform blood vessel depth calculation on the superficial vein region image corresponding to the superficial vein recognition results to obtain blood vessel depth information. The puncture location confirmation module is used to extract the vessel centerline based on the superficial vein identification result, calculate the vessel information based on the vessel centerline to obtain the target vessel information, obtain the puncture decision feature vector based on the target vessel information, and confirm the location based on the puncture decision feature vector to obtain the target puncture location. The puncture result reference module is used to calculate the needle insertion angle based on the blood vessel depth information to obtain the target needle insertion angle, calculate the needle insertion depth based on the blood vessel depth information to obtain the target needle insertion depth, and obtain the blood vessel puncture reference result based on the target puncture position, target needle insertion angle, and target needle insertion depth.

8. A smart glasses for recognizing blood vessels, characterized in that, The blood vessel recognition smart glasses include a memory, a processor, and a program stored in the memory and executable on the processor. When the program is executed by the processor, it implements the steps of the method for generating blood vessel puncture reference results based on multimodal vision as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program thereon, and the computer-readable storage medium stores a vascular puncture reference result generation program based on multimodal vision. When the vascular puncture reference result generation program based on multimodal vision is executed by a processor, it implements the steps of the vascular puncture reference result generation method based on multimodal vision as described in any one of claims 1-6.

Images