Intelligent venipuncture blood vessel positioning method and system based on multi-modal fusion

Abstract

The application discloses a vein puncture blood vessel intelligent positioning method and system based on multi-modal fusion, relates to the technical field of medical assistance and artificial intelligence, and comprises the following steps: collecting an original blood vessel image by means of an infrared imaging device and performing pretreatment; inputting a standardized blood vessel image into a deep learning model and a multi-modal large language model respectively for parallel analysis, outputting a first candidate puncture point set and a second candidate puncture point set; performing spatial alignment on the double-path candidate points, combining depth sensor data to construct a three-dimensional blood vessel network topology structure, screening an optimal puncture point through weighted fusion and clinical rule arbitration, and calculating a needle insertion angle and depth; and projecting guide information to the surface of the skin of a patient by means of a laser projection or AR display device and performing real-time calibration. The application significantly improves the accuracy, safety and individual adaptability of puncture point selection, and is suitable for clinical puncture operation and nursing teaching scenes.

Description

Technical Field

[0001] This invention relates to the fields of medical assistance technology and artificial intelligence technology, and more specifically, to a method and system for intelligent positioning of venous puncture vessels based on multimodal fusion. Background Technology

[0002] Venipuncture is one of the most common and fundamental procedures in clinical nursing, widely used in treatments such as intravenous infusions, blood transfusions, blood collection, and drug injections. However, traditional venipuncture methods heavily rely on the operator's visual observation and tactile experience, often resulting in a low success rate, especially for novice nurses. Vein location is particularly challenging in special populations such as obese patients, patients with dark skin, infants, and elderly patients, frequently leading to repeated punctures. This not only increases patient discomfort but can also cause complications such as phlebitis and hematoma, affecting treatment outcomes.

[0003] To improve puncture success rates, various auxiliary technologies have been gradually introduced into clinical practice. Infrared vascular imaging utilizes the difference in absorption of near-infrared light by hemoglobin to achieve non-invasive visualization of blood vessels, displaying vascular contours in real time. However, it lacks intelligent analysis capabilities, requiring operators to manually select puncture points based on images, which is insufficient for precise guidance. Ultrasound-guided puncture can clearly display vascular structures, especially suitable for deep vein puncture, but the equipment is expensive and the operation is complex, limiting its widespread adoption in primary healthcare institutions. Near-infrared spectroscopy can be used to assess tissue oxygenation, but its low spatial resolution makes precise localization impossible. Traditional image processing methods such as edge detection and threshold segmentation, while simple in algorithm, are easily affected by factors such as lighting and skin color, resulting in poor adaptability. In recent years, deep learning-based vascular segmentation methods have improved automation, but still lack the integration of clinical knowledge, leading to a lack of interpretability in the results.

[0004] In summary, current venous puncture-assisted techniques still face numerous bottlenecks in terms of accuracy, adaptability, real-time performance, and cost control. How to organically combine vascular visualization with intelligent puncture point recommendation, integrate clinical experience and rules, adapt to the individualized characteristics of different patient groups, and simultaneously achieve low-cost, high-precision real-time positioning has become an urgent technical problem to be solved. Summary of the Invention

[0005] To address the aforementioned technical challenges, this invention proposes a multimodal fusion-based intelligent venous puncture vessel localization method and system. This method integrates deep learning quantitative analysis with large-scale clinical reasoning to achieve complementary advantages of both approaches and three-dimensional spatial optimization. It significantly improves the accuracy, safety, and individualized adaptability of puncture point selection, making it suitable for clinical puncture procedures and nursing education scenarios.

[0006] The first aspect of this invention provides a method for intelligent localization of venous puncture vessels based on multimodal fusion, comprising the following steps: Raw vascular images of the puncture target area are acquired using an infrared imaging device, and the raw vascular images are preprocessed to obtain standardized vascular images. The standardized vascular images are input into a deep learning model and a multimodal large language model for parallel analysis. The deep learning model outputs a first set of candidate puncture points and vascular feature parameters, while the multimodal large language model, combined with clinical puncture rules and patient information, outputs a second set of candidate puncture points and the reasons for recommendation. Spatial alignment is performed on the first candidate puncture point set and the second candidate puncture point set to identify consistent and inconsistent points. A three-dimensional vascular network topology is constructed by combining depth sensor data. The consistent points are weighted and fused, and the inconsistent points are arbitrated based on clinical rules. The optimal puncture point that meets the depth requirements and safety boundaries is selected by combining the three-dimensional vascular network topology, and the corresponding needle insertion angle and needle insertion depth are calculated. The optimal puncture point, needle insertion angle, and needle insertion depth are projected onto the patient's skin surface using a laser projection or AR display device, and the patient's micro-movements are monitored in real time and the projection position is automatically calibrated.

[0007] In this scheme, raw vascular images of the puncture target area are acquired using an infrared imaging device, and the raw vascular images are preprocessed to obtain standardized vascular images, specifically including: The infrared imaging device identifies the skin reflection characteristics and ambient light conditions of the puncture target area in real time. Based on the pre-trained classification model, it automatically determines the patient's skin color type, subcutaneous tissue density and ambient light intensity, and dynamically adjusts the acquisition parameters according to the judgment results. It also controls the infrared camera, visible light camera, and depth sensor to perform multimodal collaborative acquisition through a hardware synchronization triggering mechanism; The acquired raw blood vessel images are subjected to adaptive histogram equalization, and the equalized images are processed using a structure-preserving filtering algorithm. Edge regions in the images are identified by calculating local structure tensors, and different filtering intensities are applied to edge regions and non-edge regions. Feature points in the image sequence are detected, the motion field between adjacent image frames is calculated, the current frame is subjected to inverse transformation compensation based on the motion field, the motion-compensated infrared image, visible light image and depth image are registered, and the multimodal image is transformed to a unified coordinate system. The infrared camera, visible light camera, and depth sensor are jointly calibrated. Based on the calibration results, a mapping relationship between image pixel coordinates and actual spatial physical coordinates is established to generate standardized blood vessel images.

[0008] In this approach, a deep learning model is used to output the first candidate puncture point set and vascular feature parameters, specifically including: A deep learning model with an encoder-multi-decoder architecture is constructed. The encoder uses a deep convolutional neural network to extract multi-level feature representations of standardized blood vessel images and introduces a feature pyramid structure at each level of the encoder. The multi-decoder includes four parallel and coupled decoder branches, namely a segmentation decoder, a centerline decoder, a diameter estimation decoder, and a depth estimation decoder. Spatial attention modules are embedded after each level of the encoder to focus on the vascular region by learning spatial weight maps. Channel attention modules are embedded at the connection between the encoder and the decoder to adjust the importance of feature channels by learning channel weight vectors. Task-aware attention modules are also embedded between each decoder branch to achieve cross-task information interaction and feature enhancement. The model is trained using a multi-task joint loss function, which includes a weighted combination of segmentation loss, centerline loss, diameter estimation loss, and depth estimation loss, and the weights of each task loss are dynamically adjusted through learnable weight parameters. The standardized blood vessel image is input into the trained deep learning model, and the model outputs a blood vessel segmentation probability map, a blood vessel centerline map, a blood vessel diameter estimation map, and a blood vessel depth estimation map through four decoder branches. Based on the blood vessel segmentation probability map, blood vessel centerline map, blood vessel diameter estimation map and blood vessel depth estimation map, the blood vessel centerline is uniformly sampled to obtain several initial candidate points; For each initial candidate point, a multidimensional vascular quality assessment index is calculated. The initial candidate points are sorted according to the weighted comprehensive score of the assessment index, and the top N points with the highest comprehensive scores are selected as the first candidate puncture point set.

[0009] This solution uses a multimodal large language model combined with clinical puncture rules and patient information to output a set of second candidate puncture points and the reasons for recommendation, specifically including: Construct a hierarchical clinical knowledge base comprising a core puncture principle layer, a scenario adaptation rule layer, and a patient-specific consideration layer; Standardized vascular images are converted into structured text descriptions containing vascular distribution features, vascular morphology parameters, and key region identification information using a visual-language model. The system automatically assembles prompts based on clinical scenarios. The prompts include a system role setting module, a context injection module, a patient information fusion module, and an output format constraint module. The context injection module dynamically extracts content related to the current scenario from the hierarchical clinical knowledge base. The assembled prompts are input into a multimodal large language model, which guides the model to reason step by step according to the preset thought chain. A self-consistency verification mechanism is introduced, requiring the model to review the recommendation results and warn of potential risks. The JSON format data output by the multimodal large language model is parsed to extract the set of second candidate puncture points, the reasons for recommendation, and precautions, while recording the complete inference path of the model.

[0010] In this scheme, the first candidate puncture point set and the second candidate puncture point set are spatially aligned to identify consistent and inconsistent points, specifically including... For each point in the first set of candidate puncture points, a multi-scale spatial neighborhood is constructed based on the local blood vessel diameter; Based on geometric distance components, diameter consistency components, and semantic consistency components, a matching metric is calculated between each point in the first candidate puncture point set and each point in the second candidate puncture point set. Based on the matching metric, consistent and inconsistent points are identified, where consistent points are candidate points belonging to two sets and whose spatial locations match, and inconsistent points are candidate points that appear only in a single set or whose spatial locations do not match.

[0011] In this scheme, the optimal puncture point that meets the depth requirements and safety boundaries is selected based on the three-dimensional vascular network topology, specifically including: By fusing depth sensor data and the blood vessel depth estimation map output by a deep learning model, a two-layer three-dimensional model containing the skin surface and blood vessel location is constructed, and the three-dimensional blood vessel network topology of the puncture area is extracted based on the two-layer three-dimensional model. For the identified consistent points, a fusion score is calculated based on the comprehensive score of the first candidate puncture point and the confidence score of the second candidate puncture point; the comprehensive score of the first candidate puncture point is obtained by weighted calculation based on vascular quality assessment indicators, and the confidence score of the second candidate puncture point is output by a multimodal large language model; For the identified inconsistencies, arbitration is performed based on the clinical rule base. Candidate points that conform to the clinical rules are retained, while candidate points that violate the clinical rules are removed. The candidate points retained after clinical rule arbitration are projected into the three-dimensional vascular network topology to verify whether they meet the three-dimensional spatial constraints of depth suitability and spatial isolation of accompanying structures. The candidate points that pass the verification are used as alternative points. Consistent points that have undergone fusion scoring and alternative points that have undergone clinical rule arbitration and 3D verification are combined as a candidate point set. The candidate points are then ranked comprehensively according to the fusion score or clinical rule compliance, and the candidate point ranked first is selected as the optimal puncture point.

[0012] In this solution, the calculation of the corresponding needle insertion angle and depth includes: Based on the three-dimensional vascular network topology, the vascular depth, vascular diameter, vascular orientation and vascular cross-sectional morphology parameters at the optimal puncture point are extracted. The basic needle insertion angle is calculated based on the blood vessel depth and the blood vessel diameter, and the basic angle is adjusted for displacement compensation based on the tissue mechanical properties. The anterior wall depth, center depth, and posterior wall depth of the blood vessel are determined based on the blood vessel depth and diameter. The center depth of the blood vessel is used as the basic needle insertion depth. The safety redundancy is dynamically calculated based on the blood vessel diameter to determine the upper limit of the needle insertion depth. By using force sensors to monitor resistance changes in real time during the puncture process, skin penetration and blood vessel penetration characteristics are identified. By combining these characteristics with the upper limit of needle insertion depth, dynamic guidance and early warning are generated during the needle insertion process.

[0013] A second aspect of this invention proposes a project resource recommendation system based on the fusion of knowledge graph subgraph partitioning and representation, the system comprising: The multimodal data acquisition module is used to acquire raw vascular images of the puncture target area through infrared imaging equipment, and dynamically adjust the acquisition parameters according to patient characteristics and environmental conditions. It also works with a visible light camera and a depth sensor to perform multimodal collaborative acquisition. The data preprocessing and standardization module is used to preprocess the raw vascular images, establish the mapping relationship between image pixel coordinates and actual spatial physical coordinates, and output standardized vascular images. The deep learning vascular analysis module is used to analyze standardized vascular images using a deep learning model, and outputs a vascular segmentation probability map, a vascular centerline map, a vascular diameter estimation map, and a vascular depth estimation map. It also generates a set of first candidate puncture points based on multi-dimensional vascular quality assessment indicators. The multimodal large language model reasoning module is used to construct prompt words by combining the structured image description of standardized vascular images with a hierarchical clinical knowledge base and patient-specific information. These prompt words are then input into the multimodal large language model for multi-round thought chain reasoning, and the output includes a second set of candidate puncture points containing puncture point coordinates, confidence scores, reasons for recommendation, and precautions. The decision fusion and 3D optimization module is used to spatially align the first and second candidate puncture point sets, identify consistent and inconsistent points; fuse depth sensor data to construct a 3D vascular network topology; perform weighted fusion on consistent points and arbitrate inconsistent points based on clinical rules; combine the 3D vascular network topology to screen out the optimal puncture point that meets the depth requirements and safety boundaries, and generate multi-dimensional guidance parameters including needle insertion angle and needle insertion depth. The precision guidance execution module is used to project the optimal puncture point and multi-dimensional guidance parameters onto the patient's skin surface through laser projection or AR display device, and to monitor the patient's micro-movements in real time and automatically calibrate the projection position. The feedback learning and continuous optimization module is used to record puncture result data, analyze the deviation between the actual puncture point and the recommended point, and continuously optimize the deep learning model and the large language model based on the actual puncture results. The human-computer interaction and visualization module is used to provide an operating interface, manage patient information, visualize vascular images and recommend results, and generate puncture records and evaluation reports.

[0014] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention combines deep learning quantitative analysis with multimodal large language model clinical reasoning to form a dual-path parallel analysis architecture. The deep learning path outputs multidimensional features such as vessel segmentation, diameter, and depth, while the large model path simulates expert clinical thinking to output interpretable recommendations. The two complement each other, significantly improving the accuracy and reliability of puncture point selection. The collected parameters are automatically optimized based on the patient's skin color, age, and physical condition, providing personalized puncture plans for special populations such as obese patients, infants, and elderly patients. Simultaneously, the evaluation weights are dynamically adjusted according to different puncture purposes such as routine infusion, indwelling needle insertion, and blood draw, achieving precise guidance.

[0015] Through dual-path analysis for mutual verification and clinical rule constraints, the risk of misjudgment is effectively reduced; the construction of a three-dimensional vascular network can automatically identify and avoid dangerous areas such as arteries and nerves; real-time monitoring and dynamic calibration functions ensure that the guidance information is always precisely aligned with the target vessel, significantly reducing the incidence of complications. In addition, primary healthcare institutions can choose a low-cost solution to achieve the core functions, while advanced medical centers can expand to include advanced functions such as multispectral imaging and AR guidance to meet the clinical application needs of different levels.

[0016] A standardized nursing teaching evaluation system will be established to quantitatively assess the accuracy, standardization, and efficiency of trainees' operations and generate personalized improvement suggestions. Experts' clinical experience will be transformed into a computable knowledge base to achieve digital inheritance of experience and significantly shorten the training cycle for nursing staff. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments or examples of the present invention, the drawings used in the embodiments or examples will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained according to these drawings without creative effort.

[0018] Figure 1 A flowchart of a method for intelligent localization of vein puncture vessels based on multimodal fusion is shown. Figure 2 The flowchart for obtaining the first set of candidate puncture points is shown; Figure 3The flowchart shows the process of selecting the optimal puncture point that meets the depth requirements and safety boundaries; Figure 4 A block diagram of a smart positioning system for venous puncture vessels based on multimodal fusion is shown. Detailed Implementation

[0019] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0020] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0021] like Figure 1 As shown, this embodiment provides a method for intelligent localization of vein puncture vessels based on multimodal fusion, including: Raw vascular images of the puncture target area are acquired using an infrared imaging device, and the raw vascular images are preprocessed to obtain standardized vascular images. The standardized vascular images are input into a deep learning model and a multimodal large language model for parallel analysis. The deep learning model outputs a first set of candidate puncture points and vascular feature parameters, while the multimodal large language model, combined with clinical puncture rules and patient information, outputs a second set of candidate puncture points and the reasons for recommendation. Spatial alignment is performed on the first candidate puncture point set and the second candidate puncture point set to identify consistent and inconsistent points. A three-dimensional vascular network topology is constructed by combining depth sensor data. The consistent points are weighted and fused, and the inconsistent points are arbitrated based on clinical rules. The optimal puncture point that meets the depth requirements and safety boundaries is selected by combining the three-dimensional vascular network topology, and the corresponding needle insertion angle and needle insertion depth are calculated. The optimal puncture point, needle insertion angle, and needle insertion depth are projected onto the patient's skin surface using a laser projection or AR display device, and the patient's micro-movements are monitored in real time and the projection position is automatically calibrated.

[0022] It should be noted that the infrared imaging device identifies the skin reflection characteristics and ambient lighting conditions of the puncture target area in real time. Based on a pre-trained classification model, it automatically determines the patient's skin color type, subcutaneous tissue density, and ambient light intensity, and dynamically adjusts the acquisition parameters according to the judgment results. The acquisition parameters include near-infrared wavelength, exposure time, and filtering mode. Simultaneously, an infrared camera, a visible light camera, and a depth sensor are integrated through a hardware synchronous triggering mechanism for multimodal collaborative acquisition. The infrared camera is used to capture the hemoglobin absorption characteristics of the subcutaneous vascular network; the visible light camera is used to simultaneously acquire surface features such as skin texture, pigmentation, hair distribution, and sebaceous gland openings, providing surface positioning auxiliary information for subsequent vascular identification and can be used to eliminate interfering factors such as moles and scars on the skin surface; the depth sensor is used to obtain three-dimensional morphological information of the skin surface and coarse depth information of the subcutaneous tissue.

[0023] The acquired raw vascular images undergo adaptive histogram equalization. Image sub-blocks are divided based on the gray-level distribution characteristics of local regions, and the gray-level mapping curve is adjusted. A structure-preserving filtering algorithm is then used to process the equalized image. Edge regions are identified through local structure tensor calculation, and different filtering intensities are applied to edge and non-edge regions. Feature points in the image sequence are detected, and the motion field between adjacent image frames is calculated. Inverse transformation compensation is performed on the current frame based on the motion field. The motion-compensated infrared, visible light, and depth images are registered, transforming the multimodal images to a unified coordinate system. The infrared camera, visible light camera, and depth sensor are jointly calibrated. Based on the calibration results, a mapping relationship between image pixel coordinates and actual spatial physical coordinates is established, generating a standardized vascular image. Each two-dimensional pixel in the infrared image can be mapped to the corresponding three-dimensional spatial point on the depth sensor through calibration parameters, while also being precisely aligned with the corresponding pixel in the visible light image. This mapping relationship lays the foundation for subsequent construction of the three-dimensional vascular network topology, calculation of the true depth of the blood vessels, and ultimately, mapping the puncture point coordinates to the projection-guided device.

[0024] It should be noted that standardized vascular images are input into a pre-trained deep learning model, which integrates an attention mechanism through a multi-task learning architecture. The model simultaneously outputs a vascular segmentation probability map, vascular centerline, vascular diameter estimation, vascular depth information, and a set of first candidate puncture points generated based on vascular quality assessment metrics. For example... Figure 2As shown, a deep learning model with an encoder-multi-decoder architecture is constructed. The encoder uses a deep convolutional neural network such as ResNet as the backbone network to extract multi-level feature representations of standardized blood vessel images, and introduces a feature pyramid structure at each level of the encoder. The multi-decoder includes four parallel and coupled decoder branches: a segmentation decoder, a centerline decoder, a diameter estimation decoder, and a depth estimation decoder. The segmentation decoder is responsible for outputting a blood vessel segmentation probability map. It adopts a dense skip connection structure to directly pass the features of each layer of the encoder to the corresponding layer of the decoder, ensuring the precision of the segmentation results. The centerline decoder is responsible for extracting the blood vessel centerline. Based on the segmentation features, it introduces morphological prior constraints to enhance the recognition ability of the blood vessel skeleton structure. The diameter estimation decoder is responsible for outputting a blood vessel diameter estimation map. It shares features with the segmentation decoder and the centerline decoder and introduces a diameter regression head to predict the local blood vessel diameter for each blood vessel pixel. The depth estimation decoder is responsible for outputting blood vessel depth information. It fuses the real depth data provided by the depth sensor as a supervision signal, and extracts depth-related texture features from the infrared image to achieve a mapping from two-dimensional image to three-dimensional depth.

[0025] Spatial attention modules are embedded after each level of the encoder to focus on the vascular region by learning spatial weight maps. Channel attention modules are embedded at the connection between the encoder and decoder to adjust the importance of feature channels by learning channel weight vectors. Task-aware attention modules are also embedded between the branches of each decoder to achieve cross-task information interaction and feature enhancement. A multi-task joint loss function is used to train the model. The loss function includes a weighted combination of segmentation loss, centerline loss, diameter estimation loss, and depth estimation loss, and the weights of each task loss are dynamically adjusted through learnable weight parameters. The segmentation loss uses a weighted combination of Dice loss and cross-entropy loss. Dice loss focuses on region overlap, while cross-entropy loss focuses on pixel-level classification accuracy. The two complement each other to improve segmentation accuracy. The centerline loss uses a weighted Dice loss, giving higher weights to centerline pixels to ensure that the model prioritizes the continuity of the centerline. The diameter estimation loss uses a smoothed L1 loss to regress the difference between the predicted and true diameter values. The depth estimation loss uses a mean squared error loss, combined with the true depth value provided by the depth sensor for supervision.

[0026] Standardized vascular images are input into a trained deep learning model, which outputs a vascular segmentation probability map, a vascular centerline map, a vascular diameter estimation map, and a vascular depth estimation map through four decoder branches. Based on these maps, the vascular centerline is uniformly sampled to obtain several initial candidate points. For each candidate point, a multi-dimensional vascular quality assessment index is calculated, including diameter score, straightness score (calculating the curvature of a local vascular segment based on the vascular centerline), depth score (obtaining the depth of the vascular segment from the skin surface based on the vascular depth estimation map), bifurcation distance score (identifying the bifurcation point based on the vascular centerline and segmentation map, calculating the distance between each candidate point and the nearest bifurcation point), and venous valve distance score (identifying possible venous valve locations based on vascular morphology features and clinical prior knowledge, calculating the distance between the candidate point and the venous valve). For each sampling point, the corresponding diameter value, depth value, and local curvature value are extracted, and the distance to the nearest bifurcation point and venous valve is calculated. Based on the above multi-dimensional evaluation indicators, the comprehensive score of each sampling point is calculated. The initial candidate points are sorted according to the weighted comprehensive score of the evaluation indicators, and the top N points with the highest comprehensive scores are selected as the first candidate puncture point set. Each candidate point contains at least image coordinates, comprehensive score, blood vessel diameter, and blood vessel depth information.

[0027] Due to the high cost and small sample size of medical image annotation, transfer learning and data augmentation strategies were used for model training. The encoder was loaded with weights pre-trained on a large natural image dataset, leveraging the general feature extraction capabilities learned from natural images to accelerate model convergence and improve generalization performance. Specific data augmentation strategies were designed for the characteristics of infrared vascular images, including simulating grayscale changes against different skin tones and backgrounds, simulating brightness adjustments under different lighting conditions, and simulating elastic deformation caused by patient micro-movements, thereby enhancing the model's adaptability to diverse clinical scenarios.

[0028] It should be noted that the structured image description of standardized vascular images, combined with clinical puncture rules and patient-specific information, is used to construct prompt words which are then input into a multimodal large language model. Multimodal reasoning is used to obtain a set of second-candidate puncture points, integrating the clinical experience of senior nursing experts into the intelligent decision-making process in a computable form. A hierarchical clinical knowledge base is constructed, comprising a core puncture principle layer, a scenario adaptation rule layer, and a patient-specific consideration layer. The core puncture principle layer contains basic rules that must be followed in all clinical scenarios. The scenario adaptation rule layer sets corresponding adaptation rules according to different puncture purposes and treatment needs. The patient-specific consideration layer sets personalized rules based on individual patient characteristics, including age, physical condition, and disease status. The standardized vascular images are converted into structured text descriptions containing vascular distribution features (number, location, and direction of visible vessels), vascular morphology parameters (diameter and location of the thickest vessel, length and location of the longest straight segment), and key area identification information (suggested puncture areas and areas to be avoided) through a visual-language model.

[0029] The system automatically assembles prompts based on clinical scenarios. These prompts include a system role setting module, a context injection module, a patient information fusion module, and an output format constraint module. The context injection module dynamically extracts content relevant to the current scenario from a hierarchical clinical knowledge base. The assembled prompts are input into a multimodal large language model, guiding the model to perform step-by-step reasoning according to a pre-defined thought chain. A self-consistency verification mechanism is introduced, requiring the model to review the recommendation results and warn of potential risks. The thought chain includes five stages: global observation, candidate region screening, multi-factor comparison, patient-specific adaptation, and final recommendation. First, the overall distribution of visible blood vessels in the image is described, identifying major blood vessel regions. Based on core puncture principles, candidate regions meeting basic requirements are screened, excluding obviously unsuitable locations. Candidate regions are compared across multiple dimensions, including vessel diameter, straight segment length, depth, and relationship with surrounding tissues. Personalized adjustments are made to candidate points based on patient information. One to three optimal puncture points are determined, and detailed clinical selection reasons are generated for each recommended point.

[0030] The JSON format data output by the multimodal large language model is parsed to extract the set of second candidate puncture points and the reasons for recommendation. Each candidate point includes at least image coordinates, confidence score, structured reasons for recommendation and precautions. At the same time, the complete reasoning path of the model is recorded, including the clinical principles on which it is based, the reasons for excluding other candidate points, and the patient-specific adjustment process.

[0031] It should be noted that for each point in the first set of candidate puncture points, a multi-scale spatial neighborhood is constructed based on the local vessel diameter; for vessels with larger diameters, the neighborhood radius is appropriately increased; for densely populated vessel areas, the neighborhood radius is appropriately decreased to avoid incorrect matching. The construction of the dynamic multi-scale neighborhood ensures the adaptability of the matching process to differences in vessel morphology.

[0032] Based on geometric distance, diameter consistency, and semantic consistency components, a matching metric is calculated between each point in the first candidate puncture point set and each point in the second candidate puncture point set. The geometric distance component is obtained by calculating the Euclidean distance between the two points in the image coordinate system. The diameter consistency component compares the estimated diameter of the first candidate puncture point with the diameter description of the corresponding position of the second candidate puncture point; if they are consistent within a reasonable error range, the matching confidence is enhanced. The semantic consistency component is obtained by performing semantic association analysis based on the recommendation text of the second candidate puncture point and the multidimensional evaluation index of the first candidate puncture point. Based on the matching metric, consistent and inconsistent points are identified, where consistent points are candidate points belonging to two sets and spatially matched, and inconsistent points are candidate points that appear only in a single set or whose spatial locations cannot be matched.

[0033] It should be noted that by combining the three-dimensional vascular network topology to screen for the optimal puncture point that meets the depth requirements and safety boundaries, the introduction of three-dimensional spatial information significantly improves the accuracy and safety of puncture guidance. The raw depth map provided by the depth sensor reflects the skin surface morphology, while the depth estimation map output by the deep learning model reflects the depth of the blood vessels relative to the skin surface. By fusing the depth sensor data and the vascular depth estimation map output by the deep learning model, a two-layer three-dimensional model containing the skin surface and blood vessel locations is constructed. Based on this two-layer three-dimensional model, the three-dimensional vascular network topology of the puncture area is extracted. The topology includes vascular classification information, course continuity information, branch node location information, and accompanying structure warning information (identifying the location of potentially accompanying arteries or nerves and marking safety risk areas).

[0034] like Figure 3As shown, for identified consistent points, a fusion score is calculated based on the comprehensive score of the first candidate puncture point and the confidence score of the second candidate puncture point. The comprehensive score of the first candidate puncture point is obtained by weighted calculation based on vascular quality assessment indicators, and the confidence score of the second candidate puncture point is output by a multimodal large language model. For identified inconsistent points, arbitration is performed based on a clinical rule base, which includes rules for vascular diameter suitability, joint avoidance, straight segment length, and venous valve avoidance. Candidate points that conform to the clinical rules are retained, while those that violate the clinical rules are removed. The candidate points retained after clinical rule arbitration are projected onto the three-dimensional blood vessel. In the pipe network topology, the three-dimensional spatial constraints of depth suitability and spatial isolation of accompanying structures are verified, and the candidate points that pass the verification are used as alternative points. Consistent points that have passed the fusion score and alternative points that have passed the clinical rule arbitration and three-dimensional verification are used together as the candidate point set. They are comprehensively ranked according to the fusion score or clinical rule compliance. For each candidate point in the candidate point set, if it is a consistent point, its fusion comprehensive score is used as the ranking basis; if it is an alternative point, its rule score is calculated based on the clinical rule compliance and used as the ranking basis. The candidate point set is sorted in descending order according to the ranking basis, and the candidate point ranked first is selected as the optimal puncture point.

[0035] It should be noted that calculating the corresponding needle insertion angle and depth overcomes the technical shortcomings of traditional punctures that rely on the operator's feel and experience and lack quantitative guidance. By integrating biomechanical analysis and real-time feedback mechanisms, it provides operators with precise and personalized needle insertion parameters, significantly improving the success rate and safety of punctures.

[0036] Based on the three-dimensional vascular network topology, sub-pixel-level centerline extraction is performed on the vascular segment containing the optimal puncture point. A continuous and smooth three-dimensional centerline curve is obtained through B-spline curve fitting. This three-dimensional centerline curve not only reflects the spatial orientation of the blood vessel but also provides the first derivative (tangent direction) and second derivative (curvature information) at each point, laying the foundation for calculating the needle insertion direction. Traditional vascular models typically assume a circular cross-section, but in reality, blood vessels are affected by pressure from surrounding tissues and their own tension within the body, often exhibiting an elliptical or irregular shape. By fusing multi-angle infrared imaging information and depth sensor data, the cross-sectional contour of the blood vessel at the puncture point is reconstructed, and the optimal blood vessel depth, diameter, direction, and cross-sectional morphology parameters at the puncture point are extracted. A basic needle insertion angle is calculated based on the blood vessel depth and diameter, ensuring the needle tip is precisely centered on the blood vessel upon reaching the desired depth. The basic angle is then adjusted for displacement compensation based on tissue mechanical properties, including skin elasticity coefficient, subcutaneous fat viscosity coefficient, and blood vessel wall toughness. For highly elastic skin tissue, the displacement is greater, requiring a smaller insertion angle to reduce the downward pressure effect; for dense tissue, the angle needs to be increased to provide sufficient penetration. The risk of blood vessel rolling is predicted based on the cross-sectional morphology and mobility of the blood vessel, and the insertion angle is optimized to minimize the rolling tendency. The contact mechanics between the needle tip and the blood vessel wall at different insertion angles are simulated, and the angle direction that minimizes the rolling tendency is selected. Preferably, historical puncture data from the operator is collected to establish an operator habit model, allowing for personalized fine-tuning of the theoretical angle.

[0037] The anterior wall depth, center depth, and posterior wall depth of the blood vessel are determined based on the vessel depth and diameter. The center depth is used as the baseline needle insertion depth. A safety margin is dynamically calculated based on the vessel diameter to determine the upper limit of the needle insertion depth. The depth after the safety margin is the recommended upper limit of the needle insertion depth, and the operator should complete the puncture within this depth. A force sensor monitors the resistance changes during the puncture process in real time, and the resistance curve characteristics are analyzed to identify the resistance peaks and loss-of-motion sensations corresponding to skin penetration and blood vessel penetration. By analyzing the characteristics of the resistance curve, the anatomical layer at the needle tip is determined in real time. Combining the skin penetration and blood vessel penetration characteristics with the upper limit of the needle insertion depth, when a skin penetration resistance peak is detected, it indicates entry into the subcutaneous tissue layer. When a loss-of-motion sensation is detected, a stop-needle insertion prompt is immediately issued. If the upper limit of the needle insertion depth has been reached and no loss-of-motion sensation is detected, a deviation warning is issued.

[0038] The system performs multi-parameter collaborative optimization of needle insertion angle and depth to generate a complete personalized needle insertion plan. An AR display device overlays needle insertion direction arrows, a laser projection system projects needle insertion angle indicators, and a voice prompt system broadcasts the needle insertion angle and depth information in real time. It records the actual needle insertion angle, actual needle insertion depth, puncture results, and complications for each puncture operation, analyzing the deviation between actual and recommended parameters and their correlation with puncture results. Based on the deviation analysis results, the parameters of the needle insertion angle and depth calculation models are corrected. The operator's habit model is continuously updated based on the operator's historical puncture data. A federated learning framework aggregates puncture data from multiple institutions to uncover optimal needle insertion parameter patterns for different patient groups and clinical scenarios, which are then injected into the inference engine of a multimodal large language model.

[0039] like Figure 4 As shown, the second embodiment of the present invention provides a multimodal fusion-based intelligent positioning system for venous puncture vessels, the system comprising: The multimodal data acquisition module is used to acquire raw vascular images of the puncture target area through infrared imaging equipment, and dynamically adjust the acquisition parameters according to the patient's skin color, subcutaneous fat thickness and ambient light conditions. It also works with a visible light camera and a depth sensor to perform multimodal collaborative acquisition. The data preprocessing and standardization module is used to perform preprocessing on the raw vascular images, such as image enhancement, noise suppression, motion compensation, and multimodal registration, and to establish the mapping relationship between image pixel coordinates and actual spatial physical coordinates, and output standardized vascular images. The deep learning vascular analysis module is used to analyze standardized vascular images using a deep learning model, and outputs a vascular segmentation probability map, a vascular centerline map, a vascular diameter estimation map, and a vascular depth estimation map. It also generates a set of first candidate puncture points based on multi-dimensional vascular quality assessment indicators. The multimodal large language model reasoning module is used to construct prompt words by combining the structured image description of standardized vascular images with a hierarchical clinical knowledge base and patient-specific information. These prompt words are then input into the multimodal large language model for multi-round thought chain reasoning, and the output includes a second set of candidate puncture points containing puncture point coordinates, confidence scores, reasons for recommendation, and precautions. The decision fusion and 3D optimization module is used to spatially align the first and second candidate puncture point sets, identify consistent and inconsistent points; fuse depth sensor data to construct a 3D vascular network topology; perform weighted fusion on consistent points and arbitrate inconsistent points based on clinical rules; combine the 3D vascular network topology to screen out the optimal puncture points that meet the depth requirements and safety boundaries, and generate multi-dimensional guidance parameters including needle insertion angle, needle insertion depth, vascular orientation, and risk warning. The precision guidance execution module is used to project the optimal puncture point and multi-dimensional guidance parameters onto the patient's skin surface through laser projection or AR display device, and to monitor the patient's micro-movements in real time and automatically calibrate the projection position. The feedback learning and continuous optimization module is used to record puncture result data, analyze the deviation between the actual puncture point and the recommended point, and continuously optimize the deep learning model and the large language model based on the actual puncture results. The human-computer interaction and visualization module is used to provide an operating interface, manage patient information, visualize vascular images and recommend results, and generate puncture records and evaluation reports.

[0040] The third embodiment of the present invention provides a computer-readable storage medium, which includes a program for a method for intelligent localization of venipuncture vessels based on multimodal fusion. When the program for intelligent localization of venipuncture vessels based on multimodal fusion is executed by a processor, it implements the steps of the method for intelligent localization of venipuncture vessels based on multimodal fusion.

[0041] In the several embodiments provided in this application, it should be understood that the disclosed methods and systems can be implemented in other ways. The system embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, and can be electrical, mechanical, or other forms. Furthermore, in the various embodiments of the present invention, all functional units can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0042] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0043] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for intelligent localization of venous puncture vessels based on multimodal fusion, characterized in that, Includes the following steps: Raw vascular images of the puncture target area are acquired using an infrared imaging device, and the raw vascular images are preprocessed to obtain standardized vascular images. The standardized vascular images are input into a deep learning model and a multimodal large language model for parallel analysis. The deep learning model outputs a first set of candidate puncture points and vascular feature parameters, while the multimodal large language model, combined with clinical puncture rules and patient information, outputs a second set of candidate puncture points and the reasons for recommendation. Spatial alignment is performed on the first candidate puncture point set and the second candidate puncture point set to identify consistent and inconsistent points. A three-dimensional vascular network topology is constructed by combining depth sensor data. The consistent points are weighted and fused, and the inconsistent points are arbitrated based on clinical rules. The optimal puncture point that meets the depth requirements and safety boundaries is selected by combining the three-dimensional vascular network topology, and the corresponding needle insertion angle and needle insertion depth are calculated. The optimal puncture point, needle insertion angle, and needle insertion depth are projected onto the patient's skin surface using a laser projection or AR display device, and the patient's micro-movements are monitored in real time and the projection position is automatically calibrated.

2. The intelligent positioning method for venous puncture vessels based on multimodal fusion according to claim 1, characterized in that, Raw vascular images of the puncture target area are acquired using an infrared imaging device. These raw vascular images are then preprocessed to obtain standardized vascular images, specifically including: The infrared imaging device identifies the skin reflection characteristics and ambient light conditions of the puncture target area in real time. Based on the pre-trained classification model, it automatically determines the patient's skin color type, subcutaneous tissue density and ambient light intensity, and dynamically adjusts the acquisition parameters according to the judgment results. It also controls the infrared camera, visible light camera, and depth sensor to perform multimodal collaborative acquisition through a hardware synchronization triggering mechanism; The acquired raw blood vessel images are subjected to adaptive histogram equalization, and the equalized images are processed using a structure-preserving filtering algorithm. Edge regions in the images are identified by calculating local structure tensors, and different filtering intensities are applied to edge regions and non-edge regions. Feature points in the image sequence are detected, the motion field between adjacent image frames is calculated, the current frame is subjected to inverse transformation compensation based on the motion field, the motion-compensated infrared image, visible light image and depth image are registered, and the multimodal image is transformed to a unified coordinate system. The infrared camera, visible light camera, and depth sensor are jointly calibrated. Based on the calibration results, a mapping relationship between image pixel coordinates and actual spatial physical coordinates is established to generate standardized blood vessel images.

3. The intelligent positioning method for venous puncture vessels based on multimodal fusion according to claim 1, characterized in that, The deep learning model outputs the first candidate puncture point set and vascular feature parameters, specifically including: A deep learning model with an encoder-multi-decoder architecture is constructed. The encoder uses a deep convolutional neural network to extract multi-level feature representations of standardized blood vessel images and introduces a feature pyramid structure at each level of the encoder. The multi-decoder includes four parallel and coupled decoder branches, namely a segmentation decoder, a centerline decoder, a diameter estimation decoder, and a depth estimation decoder. Spatial attention modules are embedded after each level of the encoder to focus on the vascular region by learning spatial weight maps. Channel attention modules are embedded at the connection between the encoder and the decoder to adjust the importance of feature channels by learning channel weight vectors. Task-aware attention modules are also embedded between each decoder branch to achieve cross-task information interaction and feature enhancement. The model is trained using a multi-task joint loss function, which includes a weighted combination of segmentation loss, centerline loss, diameter estimation loss, and depth estimation loss, and the weights of each task loss are dynamically adjusted through learnable weight parameters. The standardized blood vessel image is input into the trained deep learning model, and the model outputs a blood vessel segmentation probability map, a blood vessel centerline map, a blood vessel diameter estimation map, and a blood vessel depth estimation map through four decoder branches. Based on the blood vessel segmentation probability map, blood vessel centerline map, blood vessel diameter estimation map and blood vessel depth estimation map, the blood vessel centerline is uniformly sampled to obtain several initial candidate points; For each initial candidate point, a multidimensional vascular quality assessment index is calculated. The initial candidate points are sorted according to the weighted comprehensive score of the assessment index, and the top N points with the highest comprehensive scores are selected as the first candidate puncture point set.

4. The intelligent positioning method for venous puncture vessels based on multimodal fusion according to claim 1, characterized in that, Using a multimodal large language model combined with clinical puncture rules and patient information, a set of second candidate puncture points and the reasons for recommendation are output, specifically including: Construct a hierarchical clinical knowledge base comprising a core puncture principle layer, a scenario adaptation rule layer, and a patient-specific consideration layer; Standardized vascular images are converted into structured text descriptions containing vascular distribution features, vascular morphology parameters, and key region identification information using a visual-language model. The system automatically assembles prompts based on clinical scenarios. The prompts include a system role setting module, a context injection module, a patient information fusion module, and an output format constraint module. The context injection module dynamically extracts content related to the current scenario from the hierarchical clinical knowledge base. The assembled prompts are input into a multimodal large language model, which guides the model to reason step by step according to the preset thought chain. A self-consistency verification mechanism is introduced, requiring the model to review the recommendation results and warn of potential risks. The JSON format data output by the multimodal large language model is parsed to extract the set of second candidate puncture points, the reasons for recommendation, and precautions, while recording the complete inference path of the model.

5. The intelligent positioning method for venous puncture vessels based on multimodal fusion according to claim 1, characterized in that, Spatial alignment is performed on the first candidate puncture point set and the second candidate puncture point set to identify consistent and inconsistent points, specifically including... For each point in the first set of candidate puncture points, a multi-scale spatial neighborhood is constructed based on the local blood vessel diameter; Based on geometric distance components, diameter consistency components, and semantic consistency components, a matching metric is calculated between each point in the first candidate puncture point set and each point in the second candidate puncture point set. Based on the matching metric, consistent and inconsistent points are identified, where consistent points are candidate points belonging to two sets and whose spatial locations match, and inconsistent points are candidate points that appear only in a single set or whose spatial locations do not match.

6. The intelligent positioning method for venous puncture vessels based on multimodal fusion according to claim 1, characterized in that, Based on the aforementioned three-dimensional vascular network topology, optimal puncture points that meet depth requirements and safety boundaries are selected, specifically including: By fusing depth sensor data and the blood vessel depth estimation map output by a deep learning model, a two-layer three-dimensional model containing the skin surface and blood vessel location is constructed, and the three-dimensional blood vessel network topology of the puncture area is extracted based on the two-layer three-dimensional model. For the identified consistent points, a fusion score is calculated based on the comprehensive score of the first candidate puncture point and the confidence score of the second candidate puncture point; the comprehensive score of the first candidate puncture point is obtained by weighted calculation based on vascular quality assessment indicators, and the confidence score of the second candidate puncture point is output by a multimodal large language model; For the identified inconsistencies, arbitration is performed based on the clinical rule base. Candidate points that conform to the clinical rules are retained, while candidate points that violate the clinical rules are removed. The candidate points retained after clinical rule arbitration are projected into the three-dimensional vascular network topology to verify whether they meet the three-dimensional spatial constraints of depth suitability and spatial isolation of accompanying structures. The candidate points that pass the verification are used as alternative points. Consistent points that have undergone fusion scoring and alternative points that have undergone clinical rule arbitration and 3D verification are combined as a candidate point set. The candidate points are then ranked comprehensively according to the fusion score or clinical rule compliance, and the candidate point ranked first is selected as the optimal puncture point.

7. The intelligent positioning method for venous puncture vessels based on multimodal fusion according to claim 1, characterized in that, Calculate the corresponding needle insertion angle and depth, specifically including: Based on the three-dimensional vascular network topology, the vascular depth, vascular diameter, vascular orientation and vascular cross-sectional morphology parameters at the optimal puncture point are extracted. The basic needle insertion angle is calculated based on the blood vessel depth and the blood vessel diameter, and the basic angle is adjusted for displacement compensation based on the tissue mechanical properties. The anterior wall depth, center depth, and posterior wall depth of the blood vessel are determined based on the blood vessel depth and diameter. The center depth of the blood vessel is used as the basic needle insertion depth. The safety redundancy is dynamically calculated based on the blood vessel diameter to determine the upper limit of the needle insertion depth. By using force sensors to monitor resistance changes in real time during the puncture process, skin penetration and blood vessel penetration characteristics are identified. By combining these characteristics with the upper limit of needle insertion depth, dynamic guidance and early warning are generated during the needle insertion process.

8. A multimodal fusion-based intelligent positioning system for venous puncture vessels, characterized in that, For implementing the intelligent positioning method for venous puncture vessels based on multimodal fusion as described in any one of claims 1-7, the system comprises: The multimodal data acquisition module is used to acquire raw vascular images of the puncture target area through infrared imaging equipment, and dynamically adjust the acquisition parameters according to patient characteristics and environmental conditions. It also works with a visible light camera and a depth sensor to perform multimodal collaborative acquisition. The data preprocessing and standardization module is used to preprocess the raw vascular images, establish the mapping relationship between image pixel coordinates and actual spatial physical coordinates, and output standardized vascular images. The deep learning vascular analysis module is used to analyze standardized vascular images using a deep learning model, and outputs a vascular segmentation probability map, a vascular centerline map, a vascular diameter estimation map, and a vascular depth estimation map. It also generates a set of first candidate puncture points based on multi-dimensional vascular quality assessment indicators. The multimodal large language model reasoning module is used to construct prompt words by combining the structured image description of standardized vascular images with a hierarchical clinical knowledge base and patient-specific information. These prompt words are then input into the multimodal large language model for multi-round thought chain reasoning, and the output includes a second set of candidate puncture points containing puncture point coordinates, confidence scores, reasons for recommendation, and precautions. The decision fusion and 3D optimization module is used to spatially align the first and second candidate puncture point sets, identify consistent and inconsistent points; fuse depth sensor data to construct a 3D vascular network topology; perform weighted fusion on consistent points and arbitrate inconsistent points based on clinical rules; combine the 3D vascular network topology to screen out the optimal puncture point that meets the depth requirements and safety boundaries, and generate multi-dimensional guidance parameters including needle insertion angle and needle insertion depth. The precision guidance execution module is used to project the optimal puncture point and multi-dimensional guidance parameters onto the patient's skin surface through laser projection or AR display device, and to monitor the patient's micro-movements in real time and automatically calibrate the projection position. The feedback learning and continuous optimization module is used to record puncture result data, analyze the deviation between the actual puncture point and the recommended point, and continuously optimize the deep learning model and the large language model based on the actual puncture results. The human-computer interaction and visualization module is used to provide an operating interface, manage patient information, visualize vascular images and recommend results, and generate puncture records and evaluation reports.

Images