Multimodal detection system for ultrasound-guided puncture needle navigation

Abstract

This application discloses a multimodal detection system for ultrasound-guided puncture needle navigation, belonging to the field of ultrasound-assisted diagnostic technology. It is implemented using a parallel dual-model detection architecture. During the detection phase, the system sets unified inference parameters. During system initialization, both an oriented bounding box model and a segmentation model are loaded simultaneously. The oriented bounding box model performs puncture needle directionality detection on the input frame. In the result extraction phase, puncture needle information is extracted from the oriented bounding box model detection results, and lesion masks are extracted from the segmentation results. The lesion masks are then adjusted to the same size as the input frame. When extracting the lesion mask, valid mask data for the lesion category is extracted from the multi-class segmentation results. The mask data is transferred from GPU memory to the CPU and binarized using a preset segmentation threshold, converting floating-point probability values ​​into integer binary masks. This enables high-precision puncture needle direction detection and multi-class anatomical structure segmentation.

Description

Technical Field

[0001] This invention relates to the field of ultrasound-assisted diagnostic technology, and in particular to a multimodal detection system for ultrasound-guided puncture needle navigation. Background Technology

[0002] A puncture needle is a medical device used in minimally invasive surgery to sample and inject tissue from various organs, including the kidneys, liver, lungs, breast, thyroid, prostate, pancreas, testes, uterus, ovaries, and the body surface. Ultrasound-guided navigation is a medical technique that uses ultrasound technology to achieve precise positioning and diagnosis (or identification). It assists doctors in planning puncture paths or injecting drugs through real-time imaging, significantly improving the accuracy and safety of diagnosis.

[0003] In existing technologies, ultrasound-guided puncture needle navigation systems mostly use a single YOLO detection model, which cannot simultaneously detect directional puncture needle structures and segmented anatomical targets with high precision. In addition, there are the following technical defects: traditional bounding box detection is difficult to accurately capture the direction of the puncture needle, which can easily lead to trajectory calculation errors; and existing segmentation models cannot distinguish between multiple types of anatomical structures in real time, which poses many safety hazards. Summary of the Invention

[0004] This application provides a multimodal detection system for ultrasound-guided puncture needle navigation. It can simultaneously perform puncture needle orientation detection and segmentation of multiple anatomical structures during real-time ultrasound imaging. It integrates YOLOv11-OBB (Directed Bounding Box) detection for puncture needle structure with YOLOv11-SEG (Segmentation) for seven types of medical structures or target regions, avoiding model interference and providing real-time processing capabilities. The system's detection accuracy is verified using standard target detection evaluation methods, and the processing speed is dynamically adjusted based on hardware configuration and algorithm complexity.

[0005] This application provides a multimodal detection system for ultrasound-guided puncture needle navigation, implemented using a parallel dual-model detection architecture, including a detection phase and a result extraction phase. In the detection phase, the detection system sets unified inference parameters. During system initialization, two YOLOv11 models are loaded simultaneously. The first is an oriented bounding box model used for directional detection of the puncture needle in the input frame; the second is a segmentation model used for segmenting seven types of medical structures or target regions.

[0006] During the result extraction stage, the detection system extracts puncture needle information from the detection results of the directional bounding box model, extracts lesion masks from the segmentation results, and extracts masks or contour / center information of all target structures from the segmentation results for risk field construction, and adjusts the lesion mask to the same size as the input frame.

[0007] The segmentation model establishes a hierarchical security system based on medical priorities, with category mapping consistent with a preset category mapping table. The correspondence between the seven types of medical structures or target regions is as follows:

[0008] Category 0: Artery, Level 1 contraindication area, danger threshold 5.0 mm, warning threshold 8.0 mm, priority 9;

[0009] Category 1: Vein, Level II contraindication area, danger threshold 3.0 mm, warning threshold 5.0 mm, priority 7;

[0010] Category 2: Lymph node, target area, danger threshold 0.0 mm, warning threshold 0.0 mm, priority 1;

[0011] Category 3: Skeleton, obstacle area, danger threshold 0.0mm, warning threshold 1.0mm, priority 8;

[0012] Category 4: Lesion, Target area, Danger threshold 0.0 mm, Warning threshold 0.0 mm, Priority 1;

[0013] Category 5: Esophagus, secondary contraindication area, danger threshold 3.0 mm, warning threshold 5.0 mm, priority 7;

[0014] Category 6: Diaphragm, Level II contraindication area, danger threshold 2.0 mm, warning threshold 4.0 mm, priority 6;

[0015] Note: The risk thresholds and safe distances mentioned above are used for safety constraint modeling in path planning and do not constitute a medical diagnosis or treatment conclusion. Their values ​​can be parameterized according to different clinical standards, equipment types or application scenarios.

[0016] When extracting lesion masks, the validity of the segmentation results is first verified. Then, valid mask data (one or more) for the lesion category (category 4) is extracted from the multi-class segmentation results. When multiple valid masks exist, they are sorted by mask area from largest to smallest and then sorted by confidence / average probability as a secondary sort. The primary lesion mask is selected. When no lesion category mask exists, an empty mask is output and a message indicating that no lesion target was detected is displayed. The mask data is transferred from GPU memory to CPU and binarized using a preset segmentation threshold to convert the floating-point probability value into an integer binary mask. When the mask size does not match the target frame size, the system uses a nearest neighbor interpolation algorithm to adjust the size.

[0017] In one possible implementation, navigation path planning is performed using an enhanced A* algorithm with medical constraints. During system initialization, a medical grid space is established as the computational basis for the A* algorithm. The continuous image space is discretized into a searchable node network through gridding. Pixel spacing parameters are set for physical-to-pixel coordinate transformation. The medical constraint parameter system includes:

[0018] For angle change constraint parameters, it is recommended to set the maximum angle change (comfort / operability limit) to 30.0 degrees to apply strong penalty guidance for path turning; the clinical limit angle threshold is set to 45.0 degrees, and when the angle change is ≥45.0 degrees, an infinite penalty value is set to prevent the path from expanding;

[0019] Step size control, with a minimum step size of 1 grid unit and a maximum step size of 3 grid units, ensures the operability of the navigation path;

[0020] Grid resolution, 4 pixels / grid cell, balancing computational accuracy and processing efficiency;

[0021] The connectivity method employs 8-connected neighborhood search, providing path selection in 8 directions;

[0022] The weighting strategy uses a risk weight factor of 5.0 and a smoothness weight factor of 2.0.

[0023] In one possible implementation, a continuous risk field is constructed in the pixel space of the original ultrasound image, and then pixel-level risk values ​​are aggregated to grid nodes through spatial mapping to realize the transformation from the continuous risk field to the discrete search space. The resulting risk field then provides guidance for the navigation path planning algorithm.

[0024] In one possible implementation, the detection system first creates a zero-value-initialized floating-point risk field matrix based on the input image size, and then constructs boundary risk regions and anatomical structure risk regions in layers;

[0025] The boundary risk area is defined as follows: boundary constraints are established within a range of 5 pixels from the top edge, bottom edge, left edge, and right edge of the image, and a preset boundary risk value (configurable) is assigned to it.

[0026] The anatomical structure risk zone is defined as follows: For each detected anatomical structure, the system extracts its center coordinates, danger threshold, warning threshold (all in millimeters), and avoidance priority parameters. A concentric circle risk classification model is used to establish a two-level risk zone system. The high-risk zone (within the danger threshold range) is assigned an extremely high risk value of 1000.0 × priority. The warning zone (within the danger threshold to the warning threshold range) is assigned a medium risk value of 200.0 × priority. For non-risk structural areas, the system initializes with a zero value, with a risk value of 0.0.

[0027] In one possible implementation, a movement cost architecture is integrated into the navigation path planning process. The detection system establishes a three-layer cost structure, namely enhanced movement cost, which includes basic movement cost, risk cost, and smoothness cost. The basic movement cost calculates the geometric distance from the current grid point to adjacent grid points using the Euclidean distance formula, ensuring the spatial efficiency of the path. The risk cost is calculated by multiplying the risk value of adjacent points by the risk weight factor in medical constraints and applying a scale factor of 0.001 for numerical adjustment. The smoothness cost is based on the angle change of navigation path segments. When the current grid point has a parent node, the smoothness cost calculation algorithm calculates the angle change between the previous and next path segments: the system first calculates the cosine of the angle between the two path segments using vector dot product and vector magnitude, then applies the inverse cosine function to calculate the angle change and converts it from radians to degrees. The system evaluates all angle changes: when the angle change is ≤10 degrees, a negative reward is given. Smooth paths are encouraged; a linearly increasing penalty is applied when the angle change is between 10 and 20 degrees; a quadratic penalty is applied when the angle change is between 20 and 30 degrees; when the angle change is greater than 30 degrees, the penalty intensity increases sharply, applying a cubic penalty or an infinite penalty to reinforce the cost of uncomfortable turning; when the angle change is ≥45 degrees, an infinite penalty value is set to completely block the path expansion. The weighting factors in the three-layer cost structure are optimized based on clinical practice: a risk weighting factor of 5.0 ensures that the algorithm prioritizes avoiding high-risk areas; the scale factor is preferably 0.001, used to map the risk field numerical system (preset risk value for image boundaries (preferably 100.0), warning area 200.0 × priority, high-risk area 1000.0 × priority) to a cost range of the same magnitude as the basic movement cost to facilitate A* cost comparison; this scale factor supports configuration adjustment, and under the preset evaluation weights (e.g., a weighted combination of safety and efficiency) and the test scenario set, 0.001 can achieve a good trade-off between safety and efficiency.

[0028] In one possible implementation, the enhanced mobility cost calculation formula is: total_cost = base_cost + risk_cost + smoothness_cost;

[0029] Where, base_cost: basic movement cost, with a diagonal movement cost of √2≈1.414;

[0030] risk_cost=neighbor.risk_value×risk_weight×scale_factor,

[0031] Where risk_cost is the risk cost, neighbor.risk_value is the risk value of the neighbor node, risk_weight is the risk weight factor, and scale_factor is the scale factor.

[0032] smoothness_cost: Smoothness cost, calculated based on a five-level medical safety threshold system.

[0033] Among them, 30 degrees is the recommended maximum angle change limit (comfort / operability limit). The system further subdivides the 0-30 degree range into the optimal operation range (0-10 degrees), the clinically acceptable range (10-20 degrees), and the high-risk operation range (20-30 degrees). When 30 degrees < angle change < 45 degrees, it enters the extremely dangerous operation range and is subject to stronger penalties. When the angle change is ≥ 45 degrees, it is the clinical limit threshold, and an infinite penalty value is set to completely block it.

[0034] When the angle change is ≤10 degrees (comfortable operating range):

[0035] smoothness_cost = -0.5 × adjustment_factor, giving a negative value to encourage the generation of smooth paths;

[0036] When 10 degrees < angular change ≤ 20 degrees (clinically acceptable range):

[0037] smoothness_cost = (angle_change - 10) × smoothness_weight × adjustment_factor × 0.5, with a linearly increasing penalty;

[0038] When 20 degrees < angle change ≤ 30 degrees (high-risk operation range):

[0039] smoothness_cost = base_penalty + (angle_change - 20)² × smoothness_weight × adjustment_factor × 2.0, applying an exponential growth penalty;

[0040] When 30 degrees < angle change < 45 degrees (extremely dangerous operating range):

[0041] smoothness_cost = accumulated_base_penalty + (angle_change - 30)³ × smoothness_weight × adjustment_factor × 10.0, applying a cubic growth penalty;

[0042] When the angle change is ≥45 degrees (technical prohibition zone):

[0043] `smoothness_cost=∞` sets an infinite penalty value to completely block such paths.

[0044] Among them, angle_change: the change in path turning angle (degrees), smoothness_weight: smoothness weight factor, adjustment_factor: adjustment coefficient 0.1, and accumulated_base_penalty (cumulative connection penalty) is the cumulative connection base value used to ensure the continuity of the penalty between the 20-30 degree quadratic interval and the 30-45 degree cubic interval at the 30-degree boundary.

[0045] base_penalty=(20-10)×smoothness_weight×adjustment_factor×0.5, used to ensure continuous penalty in the 10-20 degree and 20-30 degree intervals;

[0046] accumulated_base_penalty = base_penalty + (30-20)²×smoothness_weight×adjustment_factor×2.0, used to ensure continuous penalty in the 20-30 degree and 30-45 degree intervals; when the angle change is ≥45 degrees, the penalty is taken as infinity, or infinity is approximated by a preset large number M in the software implementation.

[0047] In one possible implementation, an angle change penalty mechanism is integrated into the navigation path planning process. The detection system establishes an angle change threshold mechanism based on medical constraints. When the angle change between path segments exceeds the preset recommended maximum angle change limit (30 degrees), the angle change penalty strategy algorithm applies a penalty to the turn, and the penalty intensity increases sharply with the angle change. When the angle change reaches or exceeds the preset clinical limit angle threshold (45 degrees), an infinite penalty value is set to completely block the path extension. The baseline strength of the penalty value can be determined by multiplying the angle change by a smoothness weighting factor and then numerically optimized using an adjustment coefficient of 0.1.

[0048] In one possible implementation, the angle change penalty strategy algorithm specifically includes: constructing a five-level medical safety threshold system, establishing a graded penalty mechanism based on the physical limitations of clinical puncture operations, and setting five assessment levels according to the medical safety level of the angle change: comfortable operation range (0-10 degrees), clinically acceptable range (10-20 degrees), high-risk operation range (20-30 degrees), extremely dangerous operation range (30-45 degrees), and technically prohibited range (≥45 degrees). The corresponding graded penalty strategy is as follows:

[0049] Among them, 30 degrees is the recommended maximum angle change limit (comfort / operability limit). Searching is still allowed if the angle change exceeds 30 degrees, but the penalty intensity increases sharply. 45 degrees is the clinical limit threshold. When the angle change is ≥45 degrees, an infinite penalty value is set to prohibit the path from expanding.

[0050] Comfortable operating range processing: When the angle change is within the optimal operating range, the system provides a negative reward factor to actively encourage the generation of smooth paths;

[0051] Clinically acceptable range treatment: A linearly increasing function is used, and the intensity of the penalty is proportional to the angle that exceeds the comfort range;

[0052] Handling of high-risk operating ranges: Implement an exponential growth penalty model, with the penalty value increasing by the square of the angle deviation;

[0053] Handling of extremely dangerous operating ranges: Implement an exponential growth penalty mode, with the penalty value increasing cubically according to the angle deviation;

[0054] Technically prohibited areas: Set an infinite penalty value.

[0055] In one possible implementation, the unified inference parameters include a unified specification for dual-model inputs: 640×640 pixel normalized input (supporting dynamic model size adaptation), confidence threshold of 0.25, intersection-over-union (IoU) threshold of 0.45, segmentation binarization threshold of 0.5, and global pixel pitch of 0.208998 mm / pixel (obtained via DICOM metadata, supports system-level configuration adjustment, and can be configured to other values ​​such as 0.15 to adapt to different devices).

[0056] The system parameters offer excellent flexibility, supporting dynamic configuration of grid resolution, pixel spacing, detection threshold, and medical constraint parameters. These can be optimized and adjusted according to specific application scenarios, hardware conditions, and clinical needs. This configurability ensures the system's adaptability to different devices and application environments.

[0057] In one possible implementation, the puncture needle information includes needle tip position, angle, and bounding box parameters.

[0058] Beneficial Effects: Compared with existing technologies, the multimodal detection system for ultrasound-guided puncture needle navigation provided in this application employs a YOLOv11 dual-model architecture for target detection and anatomical structure segmentation. Combined with the medical A* path planning algorithm, it can achieve puncture needle direction detection and recognition of multiple types of anatomical structures. The system verifies detection accuracy using standard evaluation methods. Angle calculation is based on geometric relationships, and distance measurement is converted through pixel spacing calibration, supporting multi-scale detection of puncture needles of different specifications. System performance indicators may vary depending on the actual test environment and hardware configuration; specific detection accuracy, angle measurement accuracy, and segmentation quality indicators should be based on test results from the actual deployment environment. In terms of safety, through risk field construction and path constraint optimization, the probability of the path passing through high-risk areas can be effectively reduced, improving the safety and operability of surgical navigation.

[0059] These and other objects, features and advantages of the present invention will become fully apparent from the following detailed description. Attached Figure Description

[0060] Figure 1 A system block diagram of the parallel dual-model detection architecture of the present invention is shown.

[0061] Figure 2 The flowchart of multimodal detection and result extraction in this invention is shown.

[0062] Figure 3 A schematic diagram of risk field construction and gridded mapping in this invention is shown. Detailed Implementation

[0063] The following description is intended to disclose the present invention and enable those skilled in the art to implement it. The preferred embodiments described below are merely examples, and other obvious variations will occur to those skilled in the art. The basic principles of the invention defined in the following description can be applied to other embodiments, modifications, improvements, equivalents, and other technical solutions that do not depart from the spirit and scope of the invention.

[0064] Those skilled in the art should understand that, in the disclosure of this specification, the terms "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the above terms should not be construed as limiting the present invention.

[0065] It is understood that the term "a" should be understood as "at least one" or "one or more", that is, in one embodiment, the number of an element can be one, while in another embodiment, the number of the element can be multiple, and the term "a" should not be understood as a limitation on the number.

[0066] refer to Figures 1 to 3 Example 1: Detailed technical implementation of the parallel dual-model detection architecture:

[0067] 1.1 Technical Implementation of the YOLOv11-OBB Oriented Bounding Box Detection Model

[0068] The backbone network adopts a hierarchical feature extraction architecture, which includes five main downsampling stages: P1 layer output stride 2 (feature map size is 1 / 2 of the input), P2 layer output stride 4 (feature map size is 1 / 4 of the input), P3 layer output stride 8 (feature map size is 1 / 8 of the input), P4 layer output stride 16 (feature map size is 1 / 16 of the input), and P5 layer output stride 32 (feature map size is 1 / 32 of the input).

[0069] The feature fusion network adopts a feature pyramid network design, which achieves multi-scale feature fusion through upsampling and feature stitching. It integrates the Spatial Pyramid Pooling Module (SPPF) to expand the receptive field and introduces the Position Aware Attention Module (C2PSA) to enhance the detection capability of slender objects.

[0070] Special adjustments for puncture needle detection include: setting the number of categories to 1 for single-class detection of puncture needles; expanding the output parameters to 6 dimensions including center point coordinates x and y, width w, height h, rotation angle θ, and confidence; limiting the angle parameter θ to a range of 0 to π radians, corresponding to 0 to 180 degrees; and optimizing the receptive field for feature extraction of slender objects.

[0071] 1.2 Technical Implementation of YOLOv11-SEG Seven Types of Medical Structure or Target Region Segmentation Model

[0072] Category Definitions and Medical Stratification: Arteries (Category 0) are the highest priority medical contraindication areas, veins (Category 1) are the second-level contraindication areas, lymph nodes (Category 2) are the target tissues, bones (Category 3) are the areas of physical barriers, lesions (Category 4) are the primary treatment targets, esophagus (Category 5) are the second-level contraindication areas, and diaphragm (Category 6) are the second-level contraindication areas.

[0073] The segmentation head network adopts a prototype network architecture with 32 prototypes and 256 mask feature dimensions. It outputs multi-level segmentation masks to support instance-level accurate segmentation.

[0074] Medical priority weighting can be set based on clinical guidelines or expert experience, and can be configured and adjusted in the system parameters. Risk weighting coefficients for different anatomical structures can be set: artery weighting coefficient 9.0 (highest risk), vein weighting coefficient 7.0, lymph node weighting coefficient 1.0 (target tissue), bone weighting coefficient 8.0 (physical obstacle), lesion weighting coefficient 1.0 (primary target), esophagus weighting coefficient 7.0, and diaphragm weighting coefficient 6.0.

[0075] 1.3 Parallel Inference Architecture Implementation

[0076] The dual-model parallel computing adopts an independent model loading method to avoid parameter conflicts. The unified inference parameter settings include a confidence threshold of 0.25 and an IoU threshold of 0.45. The input image is uniformly standardized to 640×640 pixels. FP32 precision is used to ensure the computational accuracy of medical applications.

[0077] The parallel execution mechanism utilizes a thread pool executor to achieve parallel inference, with a maximum number of worker threads set to 2 to handle OBB detection and segmentation tasks respectively. Parallel efficiency is ensured through asynchronous submission and result retrieval mechanisms.

[0078] Memory optimization strategies include pre-allocating GPU memory buffers to reduce dynamic memory allocation overhead, and using CUDA streams for asynchronous data transfer and computation overlap.

[0079] Example 2: Detailed implementation of training dataset composition and annotation methods:

[0080] 2.1 Design of the Puncture Needle Detection Dataset

[0081] Dataset size statistics: a total of 3230 images, 2584 images in the training set (80%), and 646 images in the validation set (20%); the image size distribution includes primary size 1920×1080 pixels (ultra-high-definition ultrasound images), secondary size 1256×894 pixels (high-definition ultrasound images), and standard size 720×576 pixels (standard ultrasound images).

[0082] Statistics on the distribution of puncture needle specifications: 14G needles account for 35%, 16G needles account for 35%, 18G needles account for 20%, and 20G needles account for 10%; Needle length distribution: short needles (30-60 mm) account for 25%, medium length needles (60-120 mm) account for 45%, and long needles (120-180 mm) account for 30%.

[0083] X-AnyLabeling annotation tool application: Supports directional bounding box annotation, provides an angle adjustment interface to precisely control the annotation direction, supports batch annotation and quality inspection, and outputs YOLO format annotation files to directly adapt to the training process;

[0084] OBB annotation format specification: Each annotation file contains a puncture needle category identifier 0 (representing a puncture needle) and the normalized coordinates of the four vertices of the oriented bounding box arranged in counterclockwise order. Annotation example: 0 0.4005 0.3301 0.6907 0.2489 0.6866 0.2261 0.3964 0.3072, where the meaning of each parameter is as follows: 0 is the puncture needle category ID (puncture needle), followed by the normalized coordinates of the four vertices arranged in a counterclockwise direction, forming an oriented bounding box polygon that accurately describes the direction and position information of the puncture needle;

[0085] 2.2 Design of Seven Types of Medical Structure or Target Region Segmentation Datasets

[0086] Dataset size and distribution: Total number of images: 5960, training set: 4768 images (80%), validation set: 1192 images (20%); Distribution of samples by category: arteries: 2949 cases, veins: 1872 cases, lymph nodes: 4790 cases, bones: 3496 cases, lesions: 5409 cases, esophagus: 2104 cases, diaphragm: 1355 cases;

[0087] X-AnyLabeling segmentation and annotation process: Pixel-level precise annotation standards are adopted. A conservative annotation strategy is used to avoid over-annotation in areas with blurred boundaries. Overlapping areas are annotated according to medical priorities. The annotation is carried out by three senior physicians to reduce subjective differences in annotation.

[0088] YOLO segmentation format output: The annotation output adopts the YOLO segmentation format, which includes a category ID to identify one of the seven medical structures or target regions, and a normalized polygon vertex coordinate sequence of multiple vertex coordinate pairs arranged in order. Each vertex contains normalized x and y coordinates. The annotation example format is category ID x1 y1 x2 y2 x3 y3 ... xn yn. All coordinate values ​​are normalized relative to the image size and range from 0 to 1.

[0089] 2.3 Medical-Specific Data Augmentation Strategies

[0090] Enhancement configuration tailored to the characteristics of ultrasound images: Geometric transformation parameters include rotation angle ±5 degrees (to maintain correct anatomical orientation), translation ratio 5% (to reduce displacement amplitude), scaling range 20% (to adapt to different patient body types), shear angle ±2 degrees (for slight deformation), and fluoroscopic transformation 0.0001 (for minimal fluoroscopic distortion).

[0091] Color space adjustment: Hue adjustment 1% (ultrasound images are mainly grayscale), saturation adjustment 30%, brightness adjustment 20% (to adapt to different device gain settings);

[0092] Ultrasonic-specific noise simulation: Base noise level 5%, speckle noise enabled (simulating ultrasound-specific artifacts), gain change 15% (simulating equipment parameter changes), probe shadow probability 10% (simulating probe occlusion effect).

[0093] Spatial enhancement strategies: mosaic enhancement 80% probability, mixed enhancement 10% probability (reduced to maintain the realism of medical images), copy and paste 5% probability (only used for lesion target enhancement);

[0094] Example 3: Detailed implementation of key training parameters and loss function design:

[0095] 3.1 Training Hyperparameter Configuration

[0096] Basic training parameter settings: 300 training rounds (medical applications require more thorough training), batch size 32 (to balance GPU memory and training stability), input image size 640×640 pixels, early stop patience value 50 rounds (to increase patience to avoid stopping too early), checkpoint save interval save every 10 rounds;

[0097] Learning rate scheduling strategy: Initial learning rate 0.001 (reduced to improve training stability), final learning rate ratio 0.01, SGD momentum 0.937, weight decay 0.0005 (L2 regularization coefficient), 5 warm-up rounds (adding a warm-up phase), warm-up momentum 0.8, cosine learning rate scheduling enabled;

[0098] Optimizer selection: Optimizer type AdamW (more suitable for medical image features), mixed precision training enabled, gradient cropping enabled and threshold set to 10.0;

[0099] 3.2 Implementation of Independent Training Mode

[0100] The training strategy employs a two-stage independent approach: The first stage involves training the OBB (Obstruction Bypass) detection model for puncture needles, lasting 200 epochs. The dedicated dataset is the puncture needle detection dataset (3230 images), and the loss function is the OBB-specific loss function. Validation metrics include metrics / precision(B), metrics / recall(B), metrics / mAP50(B), and metrics / mAP50-95(B). The second stage involves training the anatomical structure segmentation model, lasting 300 epochs. The dedicated dataset is the anatomical structure segmentation dataset (5960 images), and the loss function is the medical segmentation-specific loss function. Validation metrics include metrics / precision(B), metrics / recall(B), metrics / mAP50(B), metrics / mAP50-95(B), metrics / precision(M), metrics / recall(M), metrics / mAP50(M), and metrics / mAP50-95(M).

[0101] Independence guarantee mechanism: The two models use different weights to initialize the seeds, the training process is completely separated with no parameter sharing, and each has its own independent verification and testing process, and saves the optimal model weights separately;

[0102] 3.3 Detailed Design of Loss Function:

[0103] 3.3.1 Loss function for OBB detection of puncture needle

[0104] The overall loss function is: L_total = λ_box × L_box + λ_cls × L_cls + λ_dfl × L_dfl, where the weight parameters include λ_box as the bounding box loss weight set to 7.5, λ_cls as the classification loss weight set to 0.5, and λ_dfl as the distribution focus loss weight set to 1.5.

[0105] The regression loss for rotated bounding boxes uses probabilistic IoU loss: L_box=1-ProbIoU(pred_rboxes,target_rboxes), where ProbIoU is the probabilistic intersection-union ratio that specifically handles the overlap calculation of rotated bounding boxes and can effectively handle angle information without a separate angle loss term;

[0106] The classification loss uses binary cross-entropy loss: L_cls=BCEWithLogitsLoss(pred_scores,target_scores);

[0107] The distributed focus loss uses DFL loss to improve boundary localization accuracy: L_dfl=DFLoss(pred_dist,target_dist), where DFLoss transforms continuous bounding box regression into a discrete distribution learning problem to improve localization accuracy;

[0108] 3.3.2 Anatomical Structure Segmentation Loss Function

[0109] The overall loss function is composed of: L_total = λ_box × L_box + λ_seg × L_seg + λ_cls × L_cls + λ_dfl × L_dfl, where the weight parameters include λ_box as the bounding box loss weight of 7.5, λ_seg as the segmentation loss weight of 7.5 (same as box), λ_cls as the classification loss weight of 0.5, and λ_dfl as the distribution focus loss weight of 1.5;

[0110] The segmentation mask loss is calculated using binary cross-entropy loss to determine the mask prediction error: L_seg=BCEWithLogitsLoss(pred_masks, gt_masks), where pred_masks is obtained by multiplying the prototype mask and coefficients: pred_masks=torch.einsum('in,nhw->ihw', pred_coeffs, proto_masks);

[0111] Other loss components: The calculation methods for bounding box loss, classification loss, and DFL loss are the same as those for the detection task, ensuring consistency between detection and segmentation tasks;

[0112] 3.4 Verification Index System

[0113] Validation metrics for OBB detection of puncture needles: Training process monitoring metrics / precision(B) bounding box precision represents the proportion of true positives among those predicted as positives; metrics / recall(B) bounding box recall represents the proportion of true positives correctly predicted; metrics / mAP50(B) average precision with an IoU threshold of 0.5; metrics / mAP50-95(B) average precision with an IoU threshold from 0.5 to 0.95 (step size 0.05). These metrics are calculated using the ap_per_class function based on statistics of true positives, confidence, predicted class, and target class.

[0114] Anatomical segmentation validation metrics: Training process monitoring bounding box metrics (labeled B) include metrics / precision (B) bounding box precision, metrics / recall (B) bounding box recall, metrics / mAP50 (B) bounding box mAP at IoU=0.5, and metrics / mAP50-95 (B) bounding box mAP at IoU=0.5:0.95; Segmentation mask metrics (labeled M) include metrics / precision (M) mask precision, metrics / recall (M) mask recall, metrics / mAP50 (M) mask mAP at IoU=0.5, and metrics / mAP50-95 (M) mask mAP at IoU=0.5:0.95. All metrics are calculated using standard methods of the Metric class to ensure consistency and comparability of the evaluation.

[0115] In ultrasound-guided puncture needle navigation detection systems, a single YOLO detection model is often used. This model has the problem of not being able to simultaneously detect directional puncture needle structures and segmented anatomical targets with high precision. In particular, traditional bounding box detection cannot accurately capture the direction of the puncture needle, which makes the trajectory calculation error easy. Furthermore, existing segmentation models cannot distinguish between multiple types of anatomical structures in real time, which poses a safety hazard and makes it difficult to achieve precise puncture needle positioning and navigation.

[0116] In view of this, embodiments of this application provide a multimodal detection system for ultrasound-guided puncture needle navigation, which can simultaneously realize puncture needle direction detection and segmentation of multiple anatomical structures in real-time ultrasound imaging. It integrates YOLOv11-OBB (directed bounding box) detection for puncture needle structure with YOLOv11-SEG (segmentation) for identification of seven types of medical structures or target regions, avoiding model interference and possessing real-time processing capabilities. In addition, it can also implement a medical A* pathfinding algorithm that combines anatomical structure risk field, multi-objective optimization, and real-time constraint satisfaction for ultrasound-guided surgery. Furthermore, it can construct a dynamic risk field that reflects different safety requirements of different anatomical structures (arteries, veins, esophagus, diaphragm, bones, lesions, lymph nodes) and has an appropriate distance-based risk gradient.

[0117] Specifically, the multimodal detection system (hereinafter referred to as the detection system) for ultrasound-guided puncture needle navigation is implemented using a parallel dual-model detection architecture. The collaborative operation of two YOLOv11 models is achieved through the ModelDetector class, including a detection phase and a result extraction phase. In the detection phase, a parallel inference strategy is employed to achieve optimal performance. The detection system sets unified inference parameters, including a confidence threshold of 0.25 (optimized for medical imaging), an intersection-over-union (IoU) threshold of 0.45 (to prevent duplicate detection), and detailed output control. During system initialization, two dedicated YOLOv11 models are loaded simultaneously. The first is an Oriented Bounding Box (OBB) model used for directional detection of the puncture needle in the input frame; the second is a Segmentation model used for segmenting seven types of medical structures or target regions. The parallel execution of these two processes further improves processing efficiency.

[0118] During the result extraction stage, the detection system extracts puncture needle information from the directional bounding box (OBB) model detection results, including needle tip position, angle, and bounding box parameters; it extracts lesion masks from the segmentation results and adjusts the lesion masks to the same size as the input frame. The mask extraction process uses a segmentation threshold of 0.5 to generate a binary mask to ensure the accuracy of the segmentation boundaries.

[0119] The relevant technical parameters (based on the actual configuration of the project code) include:

[0120] - Dual-model input unified specification: 640×640 pixel standardized input (supports automatic detection and adaptation of dynamic model size through the _get_model_input_size method);

[0121] - Confidence threshold: 0.25, set in detector initialization and main program parameter configuration, specifically optimized for low-contrast features in medical imaging;

[0122] - Cross-Union Ratio (IoU) threshold: 0.45, set in the detector parameter configuration to balance detection accuracy and repetition suppression;

[0123] - Segmentation binarization threshold: 0.5, set in the detector segmentation parameter configuration to ensure accurate segmentation of anatomical structure boundaries;

[0124] - Global pixel pitch: 0.208998 mm / pixel.

[0125] Pixel spacing parameters can be configured in the main configuration file and in each functional module.

[0126] Detailed output control includes:

[0127] 1. Control of the level of detail in test results;

[0128] -Basic mode: Outputs only bounding box coordinates and confidence level;

[0129] - Detailed mode: Outputs complete detection information, including:

[0130] - The precise coordinates of the detection box (x, y, width, height);

[0131] - Confidence score for each detected target;

[0132] - Category labels and category probability distribution;

[0133] - Rotation bounding box angle information (for OBB models);

[0134] -Pixel-level information of the segmentation mask (for the SEG model);

[0135] 2. Information output control during the reasoning process;

[0136] - Timestamp records: Time information for processing each frame;

[0137] -Performance metrics: inference time, memory usage;

[0138] -Model status: GPU utilization, model loading status;

[0139] - Anomaly localization: Detailed error information when inference fails;

[0140] 3. Medical-specific output control;

[0141] For the specific needs of ultrasonic navigation systems:

[0142] - Anatomical structure identification: Detailed classification of anatomical structures (arteries, veins, esophagus, diaphragm, lesions, lymph nodes, bones);

[0143] - Risk level information: Medical risk classification for each detection target;

[0144] - Spatial relationship description: the distance and relative positional relationship between targets;

[0145] - Puncture suitability assessment: Feasibility analysis of the puncture procedure;

[0146] 4. Debugging and development support output;

[0147] - Intermediate Feature Map: Visualization of features at each layer of the network;

[0148] - Confidence heatmap: Detects the spatial distribution of confidence levels;

[0149] - Anchor box information: Candidate box generation and filtering process;

[0150] - Post-processing details: Detailed steps of the NMS (Non-maximum suppression) process;

[0151] 5. Control of clinical record output;

[0152] - Operation log: Complete record of detection sequences;

[0153] - Keyframe markers: timestamps of important medical events;

[0154] -Quality assessment: Image quality and detection reliability scores;

[0155] -DICOM compatible output: Metadata conforming to medical imaging standards;

[0156] Technical implementation level

[0157] In the inference parameters of the YOLO model:

[0158] -verbose=True / False: Controls the verbosity of the terminal output;

[0159] -save_txt=True: Saves the test results to a text file;

[0160] -save_conf=True: Include confidence information in the output;

[0161] -save_crop=True: Saves the cropped image of the detected target;

[0162] -show_labels=True: Display label information in the visualization;

[0163] -show_conf=True: Display confidence levels in the visualization;

[0164] In medical navigation systems, detailed output control ensures that the system can: provide clinicians with complete test outputs and navigation reference information, support post-operative medical analysis and quality control, facilitate system debugging and performance optimization, meet medical record and audit requirements, and support continuous algorithm improvement and validation.

[0165] Therefore, detailed output control is a comprehensive parameter setting mechanism that determines the richness and format of the AI ​​detection model's output information to meet the specific needs of different application scenarios (real-time navigation, offline analysis, clinical recording, system debugging).

[0166] Specific implementation examples:

[0167] Assuming the input ultrasound image size is 1920×1080 pixels, the system processing flow is as follows:

[0168] 1. Image preprocessing: Scale the 1920×1080 image to a standard 640×640 input;

[0169] 2. Dual-model parallel inference: The OBB model and the SEG model process the normalized image simultaneously;

[0170] 3. Confidence Score Screening: Detection results with a confidence score > 0.25 are retained. The calculation formula is as follows:

[0171] Conf_valid={detection|conf(detection)>0.25};

[0172] in:

[0173] Conf_valid: The set of valid detection results;

[0174] detection: A single detection result object;

[0175] conf(detection): The confidence score of the detection result, with a value range of [0,1].

[0176] 0.25: Confidence threshold; detection results below this value are discarded.

[0177] 4. Non-maximum suppression: An IoU threshold of 0.45 eliminates duplicate detections. The calculation formula is as follows:

[0178] IoU=Area_intersection / Area_union;

[0179] in:

[0180] -IoU: Intersection over Union (IoU), used to measure the degree of overlap between two bounding boxes;

[0181] -Area_intersection: The area of ​​the intersection region between the two bounding boxes;

[0182] -Area_union: The area of ​​the union region of the two bounding boxes;

[0183] - When IoU > 0.45, retain the detection boxes with higher confidence and suppress duplicate detection;

[0184] 5. Post-segmentation processing: A binary mask is generated with a segmentation threshold of 0.5.

[0185] Mask_binary(x,y)={1,if prob(x,y)>0.5;0,otherwise};

[0186] in:

[0187] -Mask_binary(x,y): The binary mask value at pixel coordinates (x,y);

[0188] prob(x,y): The probability value that the pixel coordinate (x,y) belongs to the target category, with a value range of [0,1].

[0189] -0.5: Segmentation threshold; pixels with values ​​greater than this are marked as target regions.

[0190] 6. Result Restoration: Restore the 640×640 detection result to the original size of 1920×1080.

[0191] In a further optimization scheme, the detection system integrates TensorRT engine optimization technology, automatically determining whether to enable hardware acceleration based on the detected model file format. When a TensorRT-optimized OBB model is detected, the system automatically loads the GPU-accelerated orientation detection engine; when an optimized segmentation model is detected, the corresponding segmentation acceleration engine is activated, which can significantly improve inference speed.

[0192] The detection system can accurately detect and identify seven key anatomical structures. The segmentation model establishes a hierarchical safety system based on medical priorities, with category mapping consistent with a pre-defined category mapping table. The correspondence between the seven types of medical structures or target regions is as follows:

[0193] Category 0: Artery, Level 1 Contraindication Area, Danger Threshold 5.0 mm, Warning Threshold 8.0 mm, Priority 9 (Highest Risk) (defined in the Anatomical Structure Risk Configuration Module);

[0194] Category 1: Vein, Level 2 contraindication area, danger threshold 3.0 mm, warning threshold 5.0 mm, priority 7 (defined in the anatomical structure risk configuration module);

[0195] Category 2: Lymph node, target area, danger threshold 0.0 mm, warning threshold 0.0 mm, priority 1 (treatment target) (defined in the anatomical structure risk configuration module);

[0196] Category 3: Bone, obstacle area, danger threshold 0.0mm, warning threshold 1.0mm, priority 8 (defined in the anatomical structure risk configuration module).

[0197] Category 4: Lesion / mass, target area, danger threshold 0.0 mm, warning threshold 0.0 mm, priority 1 (primary treatment target) (defined in the anatomical structure risk configuration module);

[0198] Category 5: Esophagus, Level 2 contraindication area, danger threshold 3.0 mm, warning threshold 5.0 mm, priority 7 (defined in the anatomical structure risk configuration module);

[0199] Category 6: Diaphragm, Level 2 contraindication area, danger threshold 2.0 mm, warning threshold 4.0 mm, priority 6 (defined in the anatomical structure risk configuration module).

[0200] Specific implementation examples of segmentation algorithms for segmentation models:

[0201] Taking the detection of arterial structures using 1920×1080 ultrasound images as an example:

[0202] 1. Input processing: The original image is scaled to 640×640 pixels;

[0203] 2. Segmentation reasoning: YOLOv11-SEG outputs a 640×640×7 multi-class probability graph;

[0204] 3. Thresholding: The segmentation threshold for artery category (class_id=0) is 0.5.

[0205] Artery_mask(x,y)={1, if P_artery(x,y)>0.5;0,otherwise};

[0206] in:

[0207] Artery_mask(x,y): The binary mask of the artery at pixel coordinates (x,y);

[0208] P_artery(x,y): The probability value that the pixel coordinate (x,y) belongs to the artery category;

[0209] 0.5: Segmentation threshold to ensure clear boundaries;

[0210] 4. Geometric Analysis: Calculate the contour area and equivalent radius:

[0211] Area=Σ(Artery_mask(x,y)=1);

[0212] R_equiv=√(Area / π)×pixel_spacing;

[0213] in:

[0214] Area: Total number of pixels in the arterial region;

[0215] R_equiv: Equivalent radius, in millimeters;

[0216] π: Pi, approximately equal to 3.14159;

[0217] pixel_spacing: pixel spacing, 0.208998 mm / pixel;

[0218] 5. Risk parameter settings:

[0219] Risk_radius=R_equiv+Safety_margin;

[0220] in:

[0221] Risk_radius: Arterial risk radius;

[0222] Safety_margin: Safety margin distance, 8.0mm for arteries.

[0223] Mask extraction and size adaptation algorithm: When extracting lesion masks, the validity of the segmentation results is first verified. Then, valid mask data (one or more) for the lesion category (category 4) is extracted from the multi-class segmentation results. When multiple valid masks exist, they are sorted by mask area from largest to smallest, with confidence / average probability as the secondary sort, and the primary lesion mask is selected. When no lesion category mask exists, an empty mask is output and an indication that no lesion target was detected is given. The mask data is transferred from GPU memory to CPU and binarized using a preset segmentation threshold, converting the floating-point probability value into an integer binary mask. When the mask size does not match the target frame size, the system uses a nearest neighbor interpolation algorithm for size adjustment to maintain the sharpness and accuracy of the segmentation boundaries. This method is particularly suitable for medical image processing, avoiding the boundary blurring problem that may be introduced by bilinear interpolation.

[0224] In one embodiment, navigation path planning is performed using an enhanced A* (MedicalAStar) algorithm with medical constraints. During system initialization, a medical grid space is established as the computational basis for the A* (MedicalAStar) algorithm. The continuous image space is discretized into a searchable node network through gridding. A pixel spacing parameter (globally configured with a default of 0.208998 mm / pixel) is set for physical-to-pixel coordinate transformation. The medical constraint parameter system includes:

[0225] The recommended maximum angle change (comfort / operability limit) parameter is set to 30.0 degrees, defined in the medical constraint parameter configuration, and used to apply strong penalty guidance for path turning; the clinical limit angle threshold is set to 45.0 degrees, and when the angle change is ≥45.0 degrees, an infinite penalty value is set to prevent the path from expanding.

[0226] Step size control, with a minimum step size of 1 grid unit and a maximum step size of 3 grid units, is configured in the medical constraint parameters to ensure the operability of the navigation path;

[0227] The grid resolution is set to 4 pixels per grid cell to balance computational accuracy and processing efficiency;

[0228] In terms of connectivity, the medical A* algorithm module employs 8-connected neighborhood search, providing path selection in 8 directions;

[0229] The weighting strategy, with a risk weight factor of 5.0 and a path smoothness weight factor of 2.0, is set in the medical constraint parameters to achieve a balance between safety and operability.

[0230] The Medical Grid Space is a discretized computational grid space that implements the MedicalGrid class to divide ultrasound images into regular grids. Its function is to serve as a data structure for the A* path search algorithm. The coordinate system is a grid coordinate system (grid_x, grid_y), and the resolution is 4 pixels per grid cell.

[0231] In one embodiment, a continuous risk field is constructed in the pixel space of the original ultrasound image, and then the pixel-level risk values ​​are aggregated to grid nodes through spatial mapping to realize the transformation from the continuous risk field to the discrete search space, thereby providing guidance for the navigation path planning algorithm through the final generated risk field.

[0232] The pixel space, also known as the image space (Risk Field image space), is a continuous pixel-level risk field space. It implements the risk_field array, which corresponds one-to-one with the pixels of the original ultrasound image. Its function is to store the risk value of each pixel (based on the distance to the anatomical structure). The coordinate system is the pixel coordinate system (pixel_x, pixel_y). The resolution is the same as that of the original ultrasound image (1:1 pixel correspondence).

[0233] The relationship and transformation between grid space and image space are as follows:

[0234] Image space -> Grid space;

[0235] grid_x, grid_y=self.grid.pixel_to_grid(pixel_x, pixel_y);

[0236] Grid space -> Image space;

[0237] pixel_x, pixel_y=self.grid.grid_to_pixel(grid_x, grid_y);

[0238] Risk field mapped to grid;

[0239] def set_risk_field(self, risk_field: np.ndarray):

[0240] Mapping continuous risk field values ​​to discrete grid points

[0241] risk_region=risk_field[pixel_y_start:pixel_y_end, pixel_x_start:pixel_x_end];

[0242] avg_risk = np.mean(risk_region) (takes the average value of the region);

[0243] self.grid[x][y].risk_value=avg_risk.

[0244] In simple terms, image space is a continuous risk field at the pixel level, while grid space is a discrete computational grid used for path search. The two are linked through coordinate transformation and value mapping.

[0245] More preferably, the detection system first creates a zero-value-initialized floating-point risk field matrix based on the input image size, and then constructs boundary risk regions and anatomical structure risk regions in layers;

[0246] The boundary risk area is defined as follows: The system sets a preset boundary risk value (configurable) at the edge of the image. In the preferred embodiment, it is 100.0. Specifically, boundary constraints are established within a range of 5 pixels from the top edge, 5 pixels from the bottom edge, 5 pixels from the left edge, and 5 pixels from the right edge of the image. This is implemented in the medical grid risk field construction module to prevent the planned path from exceeding the effective imaging area.

[0247] The anatomical structure risk zone is defined as follows: For each detected anatomical structure object, the system extracts the center coordinates, danger threshold, warning threshold (all in millimeters), and avoidance priority parameters for each object. A concentric circle risk classification model is used to establish a two-level risk zone system. The high-risk zone (within the danger threshold range) is assigned an extremely high risk value of 1000.0 × priority, while the warning zone (within the danger threshold to the warning threshold range) is assigned a medium risk value of 200.0 × priority. For non-risk structural areas, the system initializes with a zero value, with a risk value of 0.0.

[0248] Both the danger threshold and the warning threshold are expressed in millimeters (mm), used to physically define the risk radius and form a closed-loop mapping:

[0249] (1) The warning threshold T_warn defines the outer boundary radius R_warn of the warning area, which is calculated by physical-pixel conversion based on the pixel spacing;

[0250] (2) The danger threshold T_crit defines the boundary radius R_crit of the high-risk zone;

[0251] (3) Two-level risk zone system: the radius R_crit of the high-risk zone corresponds to the danger threshold, and the outer boundary R_warn of the warning zone corresponds to the warning threshold. The absolute distance threshold is used for risk assessment to avoid the complexity of percentage classification.

[0252] Specific implementation examples of risk field construction algorithms:

[0253] Taking an arterial structure detected by a 1920×1080 ultrasound image as an example (center coordinates (500, 400), risk radius 8.0 mm, priority 9):

[0254] 1. Physical to pixel conversion:

[0255] Risk radius in pixels = 8.0mm ÷ 0.208998mm / pixel ≈ 38.3 pixels;

[0256] 2. Tiered risk calculation:

[0257] High-risk area: r≤5.0mm÷0.208998mm / pixel≈24.0 pixels, risk value=1000.0×9=9000.0;

[0258] Warning zone: 24.0 < r ≤ 8.0 mm ÷ 0.208998 mm / pixel ≈ 38.3 pixels, risk value = 200.0 × 9 = 1800.0;

[0259] 3. Distance Calculation: Calculate the Euclidean distance from each pixel in the 1920×1080 image to the center of the artery:

[0260] distance(x,y)=√[(x-center_x)²+(y-center_y)²];

[0261] in:

[0262] distance(x,y): The distance from pixel (x,y) to the center of the artery;

[0263] center_x, center_y: Coordinates of the artery center, which are (500, 400) in this example;

[0264] √: Square root operation, calculates Euclidean distance;

[0265] Specific calculation example: distance(520,420)=√[(520-500)²+(420-400)²]=√[400+400]=28.3 pixels;

[0266] 4. Risk Allocation: Assign corresponding risk levels based on distance values;

[0267] 5. Obstacle Marking: The system comprehensively determines obstacles based on anatomical structure type and risk value. Areas with a risk value exceeding the preset obstacle threshold T_obs and which are not treatment target structures are marked as obstacles in the medical grid system. Treatment target structures (lesions, lymph nodes) have a priority of 1, and their warning area risk value (200.0 × 1 = 200) is lower than the obstacle threshold, therefore they will not be incorrectly marked as obstacles, ensuring that the path can be correctly planned to the treatment target. The obstacle threshold T_obs is a configurable parameter, preferably consistent with the risk value system of this specification (boundary risk 100, warning area 200 × priority, high-risk area 10). The T_obs value (00×priority) maintains a consistent dimension and magnitude and can be determined as follows: T_obs = β × (200.0 × p_target), where p_target is the priority of the treatment target structure (p_target = 1 in this embodiment), and β is a scaling factor greater than 1 (preferably 2.0 to 5.0, more preferably β = 2.5, corresponding to T_obs = 500). Through this design, when the priority configuration, risk benchmark value, or clinical safety strategy is adjusted, β can be adjusted simultaneously or T_obs can be configured directly, thereby maintaining the consistency of the passage judgment strategy of "target tissue is accessible, and non-target high-risk areas are inaccessible".

[0268] 6. Grid Mapping: In the optimized Navigator, the risk field is mapped to the grid system in 4×4 pixel blocks.

[0269] This grading system is implemented through polar coordinate distance calculation. For each pixel on the image plane, the Euclidean distance from it to the center of the anatomical structure is calculated. The risk level of the point is determined according to the ratio of distance to risk radius, and the corresponding risk value is assigned to the corresponding position in the risk field matrix. The resulting risk field provides detailed safety guidance information for the path planning algorithm.

[0270] Optional Implementation: Construction of a non-circular risk field based on distance transformation (nearest distance field):

[0271] For slender or curved anatomical structures such as blood vessels, esophagus, and diaphragm, the system does not approximate modeling using concentric circles of "structure center point + equivalent radius." Instead, it directly calculates the Euclidean distance transformation based on the binary mask S output by the segmentation model, obtaining the nearest distance field D(x,y)=min_{(u,v)∈S}√[(xu)²+(yv)²], and converts it into physical distance d_mm=D(x,y)×pixel_spacing by combining the pixel spacing parameter. Subsequently, it assigns tiered values ​​according to the corresponding structure's danger threshold d_danger and warning threshold d_warn (in mm): when d_mm≤d_danger, it is assigned a value of 1000×priority; when d_danger<d_mm≤d_warn, it is assigned a value of 200×priority; and when d_mm>d_warn, it is assigned a value of 0. The non-circular risk field is merged with the boundary risk area (preset risk value (preferably 100.0)) (e.g., taking the maximum value point by point), thereby maintaining the consistency of the risk value system of "boundary 100 / warning 200×priority / high risk 1000×priority" while generating risk contour lines along the true shape of the anatomical structure, avoiding geometric distortion of non-circular structures by the concentric circle model, and improving the safety and applicability of path planning.

[0272] In one embodiment, a mobility cost architecture is integrated into the navigation path planning process. The detection system establishes a three-layer cost structure, namely enhanced mobility cost, which includes basic mobility cost, risk cost, and smoothness cost. The basic mobility cost calculates the geometric distance from the current grid point to adjacent grid points using the Euclidean distance formula to ensure the spatial efficiency of the path. The risk cost is calculated by multiplying the risk value of adjacent points by the risk weight factor in the medical constraints and applying a scaling factor of 0.001 for numerical adjustment. The smoothness cost is based on the angle change of the navigation path segment. When the current grid point has a parent node, the smoothness cost calculation algorithm calculates the angle change between the previous path segment and the next path segment. The system first calculates the cosine of the angle between two path segments using vector dot product and vector magnitude. Then, it applies the inverse cosine function to calculate the angle change and converts it from radians to degrees for evaluation. The system evaluates all angle changes: a negative reward is given when the angle change is ≤10 degrees to encourage smooth paths; a linearly increasing penalty is applied when the angle change is between 10 and 20 degrees; a quadratic penalty is applied when the angle change is between 20 and 30 degrees; when the angle change is >30 degrees, the penalty intensity increases sharply, applying a cubic or infinite penalty to reinforce the cost of uncomfortable turning. The weighting factors in the three-layer cost structure are optimized based on clinical practice; when the angle change is ≥45 degrees, an infinite penalty value is set to completely block the path extension.

[0273] More preferably, the enhanced mobility cost calculation formula is: total_cost = base_cost + risk_cost + smoothness_cost;

[0274] Where, base_cost: basic movement cost, with a diagonal movement cost of √2≈1.414;

[0275] risk_cost=neighbor.risk_value×risk_weight×scale_factor,

[0276] Where risk_cost is the risk cost, neighbor.risk_value is the risk value of the neighbor node, risk_weight is the risk weight factor, and scale_factor is the scale factor.

[0277] smoothness_cost: Smoothness cost, calculated based on a five-level medical safety threshold system.

[0278] Among them, 30 degrees is the recommended maximum angle change limit (comfort / operability limit). The system further subdivides the 0-30 degree range into the optimal operation range (0-10 degrees), the clinically acceptable range (10-20 degrees), and the high-risk operation range (20-30 degrees). When 30 degrees < angle change < 45 degrees, it enters the extremely dangerous operation range and is subject to stronger penalties. When the angle change is ≥ 45 degrees, it is the clinical limit threshold, and an infinite penalty value is set to completely block it.

[0279] When the angle change is ≤10 degrees (comfortable operating range):

[0280] smoothness_cost = -0.5×adjustment_factor, giving a negative value to encourage the generation of smooth paths;

[0281] When 10 degrees < angular change ≤ 20 degrees (clinically acceptable range):

[0282] smoothness_cost = (angle_change - 10) × smoothness_weight × adjustment_factor × 0.5, with a linearly increasing penalty;

[0283] When 20 degrees < angle change ≤ 30 degrees (high-risk operation range):

[0284] smoothness_cost = base_penalty + (angle_change - 20)² × smoothness_weight × adjustment_factor × 2.0, applying an exponential growth penalty;

[0285] When 30 degrees < angle change < 45 degrees (extremely dangerous operating range):

[0286] smoothness_cost = accumulated_base_penalty + (angle_change - 30)³ × smoothness_weight × adjustment_factor × 10.0, applying a cubic growth penalty;

[0287] When the angle change is ≥45 degrees (technical prohibition zone):

[0288] `smoothness_cost=∞` sets an infinite penalty value to completely block such paths.

[0289] Among them, angle_change: the change in path turning angle (degrees), smoothness_weight: smoothness weight factor, adjustment_factor: adjustment coefficient 0.1, and accumulated_base_penalty (cumulative connection penalty) is the cumulative connection base value used to ensure the continuity of the penalty between the 20-30 degree quadratic interval and the 30-45 degree cubic interval at the 30-degree boundary.

[0290] base_penalty (interval continuity penalty) = (20-10) × smoothness_weight × adjustment_factor × 0.5, used to ensure the continuity of the penalty between the 10-20 degree and 20-30 degree intervals;

[0291] accumulated_base_penalty = base_penalty + (30-20)² × smoothness_weight × adjustment_factor × 2.0, which is used to ensure that the penalty is continuous in the 20-30 degree and 30-45 degree intervals; when the angle change is ≥ 45 degrees, the penalty is taken as infinity, or infinity is approximated by a preset large number M in the software implementation.

[0292] Basic moving cost calculation example:

[0293] When evaluating a move from grid point A (x=10, y=15) to adjacent grid point B (x=11, y=16), the geometric distance is calculated using the Euclidean distance formula. Specifically, the base movement cost is calculated as follows: Base movement cost = √[(11-10)² + (16-15)²] = √[1² + 1²] = √2 ≈ 1.414. For orthogonal movements (e.g., from A to C (x=11, y=15)), the base cost is 1.0; for diagonal movements, the base cost is 1.414, reflecting the additional distance cost of diagonal movement.

[0294] Example of risk cost calculation:

[0295] Assuming target grid point B is located in the artery warning region with a risk value of 1800 (200.0 × 9 = 1800), and the risk weight factor in the medical constraints is set to 5.0, then the risk cost is calculated as: Risk Cost = 1800 × 5.0 × 0.001 = 9.0. This value represents the additional path cost incurred due to proximity to high-risk anatomical structures, guiding the algorithm to avoid danger zones.

[0296] Example of comprehensive cost calculation:

[0297] Combining the two components mentioned above, the total movement cost from grid point A to grid point B is: Total Cost = Basic Movement Cost + Risk Cost = 1.414 + 9.0 = 10.414. In contrast, the cost of moving to the image boundary region (risk value 100.0) is: 1.414 + (100.0 × 5.0 × 0.001) = 1.414 + 0.5 = 1.914; the cost of moving to the non-risk region (risk value 0) is only: 1.414 + (0 × 5.0 × 0.001) = 1.414, reflecting the algorithm's preference for safe paths.

[0298] The complete risk value distribution of the arterial region is as follows: high risk area 9000 (1000.0×9), warning area 1800 (200.0×9). The system also includes a preset risk value for the image boundary (preferably 100.0) and a zero value of 0 for the non-risk area, ensuring that the risk field covers all possible spatial locations.

[0299] Example of a path smoothness evaluation mechanism:

[0300] Example of angle change calculation:

[0301] Note: The following is only for demonstrating the calculation method of angle change; in actual path planning, when the angle change is ≥45 degrees, it will be considered a clinical limit and prohibited (infinite penalty for complete block).

[0302] Consider three consecutive grid points: parent node P (x=8, y=10), current node C (x=10, y=15), and next node N (x=13, y=15). The system first constructs vectors: the preceding path segment vector V1=(10-8, 15-10)=(2,5), and the subsequent path segment vector V2=(13-10, 15-15)=(3,0).

[0303] Example of vector dot product and magnitude calculation:

[0304] Vector dot product: V1·V2=2×3+5×0=6+0=6;

[0305] The magnitude of vector V1 is: |V1| = √(2² + 5²) = √29 ≈ 5.385, and the magnitude of vector V2 is: |V2| = √(3² + 0²) = √9 = 3.0.

[0306] Example of quantifying angle change:

[0307] Cosine value: cos(θ) = 6 / (5.385 × 3.0) ≈ 6 / 16.155 ≈ 0.372;

[0308] Angle change: θ = arccos(0.372) ≈ 68.1° × (180 / π) ≈ 68.1°;

[0309] This example shows that there is a large angle change between the two path segments; since the angle change (68.1°) exceeds 45 degrees, it is only for calculation demonstration and will be prohibited in actual path planning (infinite penalty completely blocks it).

[0310] Examples of non-zero angle changes:

[0311] If the next node is N(x=12,y=13), then V2=(2,-2);

[0312] Dot product: V1·V2=2×2+5×(-2)=4-10=-6,|V2|=√8≈2.828;

[0313] cos(θ)=-6 / (5.385×2.828)≈-0.394;

[0314] Angle change: θ = arccos(-0.394) ≈ 113.2° (example value, for demonstration purposes only; in actual path planning, angle changes ≥ 45 degrees will be prohibited).

[0315] In one embodiment, an angle change penalty mechanism is also integrated into the navigation path planning process. The detection system establishes an angle change threshold mechanism based on medical constraints. When the angle change between path segments exceeds the preset recommended maximum angle change limit (30 degrees), the angle change penalty strategy algorithm applies a penalty to the turn, and the penalty intensity increases sharply with the amount of angle change. When the angle change reaches or exceeds the preset clinical limit angle threshold (45 degrees), an infinite penalty value is set to completely block the path extension. The baseline strength of the penalty value can be determined by multiplying the angle change amount by the smoothness weight factor, and numerical optimization is performed using an adjustment coefficient of 0.1.

[0316] More preferably, the angle change penalty strategy algorithm specifically includes: constructing a five-level medical safety threshold system, establishing a graded penalty mechanism based on the physical limitations of clinical puncture operations, and setting five assessment levels according to the medical safety level of the angle change: comfortable operation range (0-10 degrees), clinically acceptable range (10-20 degrees), high-risk operation range (20-30 degrees), extremely dangerous operation range (30-45 degrees), and technically prohibited range (≥45 degrees). The corresponding graded penalty strategy is as follows:

[0317] Among them, 30 degrees is the recommended maximum angle change limit (comfort / operability limit). Searching is still allowed if the angle change exceeds 30 degrees, but the penalty intensity increases sharply. 45 degrees is the clinical limit threshold. When the angle change is ≥45 degrees, an infinite penalty value is set to prohibit the path from expanding.

[0318] Comfortable operating range processing: When the angle change is within the optimal operating range, the system provides a negative reward factor to actively encourage the generation of smooth paths;

[0319] Clinically acceptable range treatment: A linearly increasing function is used, and the intensity of the penalty is proportional to the angle that exceeds the comfort range;

[0320] Handling of high-risk operating ranges: Implement an exponential growth penalty model, with the penalty value increasing by the square of the angle deviation;

[0321] Handling of extremely dangerous operating ranges: Implement an exponential growth penalty mode, with the penalty value increasing cubically according to the angle deviation;

[0322] Technically prohibited areas: Set an infinite penalty value.

[0323] A complete implementation of the angle change penalty strategy algorithm includes specific calculation methods for five evaluation levels:

[0324] Comfortable operating range processing (0-10 degrees): penalty_cost = -0.5 × 0.1 = -0.05, negative values ​​indicate positive encouragement for smooth paths;

[0325] Clinically acceptable range of treatment (10-20 degrees): penalty_cost = (angle_change - 10) × 2.0 × 0.1 × 0.5, the penalty intensity is proportional to the amount of angle exceeding the comfort range;

[0326] High-risk operation range handling (20-30 degrees): penalty_cost = base_penalty + (angle_change - 20)² × 2.0 × 0.1 × 2.0, implement exponential growth penalty mode, the penalty value increases by the square of the angle deviation;

[0327] Extremely dangerous operating range handling (30-45 degrees): penalty_cost = accumulated_base_penalty +(angle_change - 30)³×2.0×0.1×10.0, implement cubic growth penalty mode, the penalty value increases cubically according to the angle deviation;

[0328] Technically prohibited interval processing (above 45 degrees): penalty_cost = ∞, setting an infinite penalty value.

[0329] In addition, the system also applies medical safety correction factors, including additional constraints for deep punctures (additional penalty for angle changes > 15 degrees) and additional constraints for adjacent critical structures (additional penalty for angle changes > 12 degrees).

[0330] Examples of comfort range penalties:

[0331] When the angle change is 5°, the system determines that it is within the comfortable operating range (0°-10°) and awards a negative reward: penalty value = -0.5 × adjustment coefficient 0.1 = -0.05. The negative value indicates a positive encouragement for a smooth path, effectively guiding the algorithm to generate the clinically optimal path.

[0332] Acceptable range of linear penalty instances:

[0333] When the angle change is 15°, exceeding the comfort range by 5°, the system applies a linear penalty:

[0334] The excess angle = 15° - 10° = 5°;

[0335] Penalty value = 5 × 2.0 (smoothness weight factor) × 0.1 × 0.5 = 0.5;

[0336] The penalty value is moderate, allowing the path to proceed but increasing the corresponding cost.

[0337] Example of high-risk range index penalty:

[0338] When the angle changes to 25°, the system enters a high-risk operating range:

[0339] Base penalty = (20° - 10°) × 2.0 × 0.1 × 0.5 = 1.0;

[0340] Exponential penalty = (25° - 20°)² × 2.0 × 0.1 × 2.0 = 5² × 0.4 = 10.0;

[0341] Total penalty value = 1.0 + 10.0 = 11.0;

[0342] Extremely dangerous range (30°-45°), base penalty = cumulative penalty of preceding range;

[0343] Cubic penalty = (angle change - 30°)³ × smoothness weighting factor × adjustment coefficient × 10.0;

[0344] Total penalty = base penalty + cubed penalty;

[0345] Forbidden range (≥45°), penalty value = ∞ (complete blockage).

[0346] A significant penalty value can effectively prevent the algorithm from choosing a high-risk path.

[0347] Examples of medical safety modifications:

[0348] In addition to the aforementioned 25° angle change, the system also considers medical constraints:

[0349] - Deep puncture correction: (25-15°)×0.3×1.2=3.6;

[0350] - Correction for proximity to critical structures: (25°-12°)×0.2×1.0=2.6;

[0351] - Needle specification correction (16G): (25°-20°)×0.4×1.8=3.6;

[0352] Total medical safety correction: 3.6 + 2.6 + 3.6 = 9.8;

[0353] Final penalty value example:

[0354] Total penalty value = Total angle penalty + Medical safety correction = 11.0 + 9.8 = 20.8.

[0355] The medical safety correction calculation formula, implemented through a medical safety correction algorithm, achieves a triple correction formula:

[0356] Deep puncture correction, if the angle change is greater than 15°: Deep correction = (angle change - 15°) × 0.3 × depth factor (1.2);

[0357] For adjacent critical structures, if the angle change is greater than 12°: Adjacency correction = (angle change - 12°) × 0.2 × adjacency factor (1.0);

[0358] Needle specification correction, if the angle change is greater than 20°: Specification correction = (angle change - 20°) × 0.4 × needle flexibility factor;

[0359] The formula for calculating the needle flexibility factor is implemented by using the needle flexibility factor acquisition algorithm to achieve the specification mapping formula: Needle flexibility factor = {14G: 2.0, / / Coarse needle, difficult to turn;}

[0360] 16G: 1.8, 18G: 1.5, 20G: 1.2, 22G: 1.0, / / Standard reference;

[0361] 25G: 0.8 / / Fine needle, relatively flexible};

[0362] Risk field mapping calculation formula

[0363] Implement the risk value mapping formula using the risk field mapping algorithm:

[0364] Grid risk value = mean(pixel risk field[y_start:y_end,x_start:x_end]);

[0365] in:

[0366] x_start = grid x × grid resolution;

[0367] y_start = grid y × grid resolution;

[0368] x_end = min(x_start + grid resolution, risk field width);

[0369] y_end=min(y_start+mesh resolution, risk field height);

[0370] The final enhanced A* cost function is: F(n) = G(n) + H(n); where: G(n): the actual cumulative cost function, representing the actual cumulative path cost from the starting point to the current node n, calculated as follows:

[0371] G(n) = G(parent) + move_cost(parent, n); This can be further elaborated as follows:

[0372] G(n) = Parent node G value + Enhanced movement cost = Parent node G value + (Basic movement distance + Medical risk cost + Smoothness penalty cost);

[0373] Here, the parent node G value is defined as the cumulative actual cost from the starting point s to the parent node parent, that is, G(parent) represents the total cost of the path already traversed. In particular, the starting point G value is initialized to G(s) = 0, and the G value of each subsequent node is calculated recursively.

[0374] The base distance in move_cost(parent, n) refers to the geometric distance between two adjacent grid nodes. For an 8-connected grid, the base distance for orthogonal movement is 1.0, and the base distance for diagonal movement is √2≈1.414. The calculation formula is: base_distance=√[(x_n - x_parent)²+(y_n-y_parent)²];

[0375] Where (x_parent, y_parent) and (x_n, y_n) are the grid coordinates of the parent node and the current node, respectively, and the two are adjacent (distance ≤ √2).

[0376] H(n): Heuristic cost estimation function, representing the estimated residual cost from the current node n to the target node goal. The Euclidean distance is used as the heuristic function, and the calculation formula is: H(n)=√[(x_goal-x_current)²+(y_goal-y_current)²];

[0377] Where (x_current, y_current) are the grid coordinates of the current node n, and (x_goal, y_goal) are the grid coordinates of the target node. H(n) represents the straight-line distance from the current position to the target position, used to estimate the minimum possible cost of the remaining path and guide the search towards the target.

[0378] F(n): The total estimated cost function combines the actual cumulative cost and the estimated residual cost, used for priority queue sorting in the A* algorithm: F(n) = G(n) + H(n) = actual cumulative cost + estimated residual cost. The A* algorithm improves search efficiency by prioritizing the expansion of nodes with the smallest F(n) value, ensuring the optimal path is found. This value effectively reflects the comprehensive impact of multi-dimensional medical constraints on path selection, ensuring the clinical safety and operability of the generated path.

[0379] Assume the starting point is s(0,0), the current node is n(10,10), the neighboring node is m(11,11), and the target node is goal(100,100):

[0380] 1. The G value of the current node n: G(n) represents the cumulative actual cost from the starting point (0,0) to node (10,10), assuming G(n) = 20.5;

[0381] 2. Calculate the cost of moving to neighbor node m:

[0382] - Basic movement distance: √[(11-10)²+(11-10)²]=√2≈1.414 (adjacent diagonal movement);

[0383] - Enhanced movement cost: move_cost = 1.414 + risk cost + smoothness cost, assumed to be 2.0;

[0384] - The G value of the neighboring node: G(m) = G(n) + move_cost = 20.5 + 2.0 = 22.5;

[0385] 3. Calculate the heuristic cost of neighbor node m:

[0386] - H(m)=√[(100-11)²+(100-11)²]=√(89²+89²)≈125.9 (straight-line distance to the target);

[0387] 4. Calculate the total estimated cost of neighbor node m:

[0388] - F(m)=G(m)+H(m)=22.5+125.9=148.4.

[0389] In the example above, the base moving distance of 1.414 and the heuristic distance of 125.9 describe completely different geometric relationships. The former is used to accumulate the G value, and the latter is used to estimate the H value. Together, they constitute the F value to guide the A* search.

[0390] The base distance calculation for move_cost is the distance between adjacent nodes, with a typical value of 1.0 or 1.414. Its function is to accumulate it into G(n) to represent the path already traversed.

[0391] The heuristic distance calculation of H(n) is the straight-line distance from the current point to the target. Its typical value can be very large (e.g., 125.9). Its purpose is to estimate the remaining path and guide the search direction.

[0392] Therefore, the two do not overlap, but together constitute the "estimated total mileage" F(n).

[0393] Performance was verified using actual code configuration parameters in a standard testing environment (NVIDIA RTX 3080, CUDA 11.8):

[0394] Verification of puncture needle detection accuracy:

[0395] - Test configuration: confidence threshold 0.25, IoU threshold 0.45 (set in detector parameter configuration);

[0396] - Input image: 1920×1080 ultrasound image, scaled down to 640×640 for inference;

[0397] - Evaluation method: Standard precision, recall, and F1 score were used as evaluation metrics;

[0398] - Precision calculation formula: Precision=TP / (TP+FP), where TP is the number of true positives and FP is the number of false positives;

[0399] - Recall formula: Recall=TP / (TP+FN), where FN is the number of false negatives;

[0400] - F1 score formula: F1 = 2 × (Precision × Recall) / (Precision + Recall);

[0401] Seven types of medical structure or target region segmentation verification:

[0402] - Segmentation threshold: 0.5;

[0403] - Evaluation metrics: The standard mean intersection-union ratio (mIoU) is used for evaluation.

[0404] -mIoU calculation formula: mIoU=(1 / 7)×Σ[Area_intersection / Area_union];

[0405] -Specific calculation: The intersection-union ratio of each category is determined by the overlap between the segmentation mask and the actual annotation;

[0406] - Overall performance: The overall segmentation performance index of the system is obtained by averaging the intersection-union ratios of seven types of medical structures or target regions;

[0407] Real-time processing performance verification:

[0408] - Pixel pitch setting: 0.208998 mm / pixel (set in the main configuration module);

[0409] -TensorRT engine acceleration: Supports acceleration of best.engine and 7_seg.engine models (loaded during detector initialization);

[0410] -Performance monitoring: The system has real-time performance monitoring capabilities for each processing stage, including image preprocessing, dual-model parallel inference, result postprocessing, anatomical structure analysis, risk field construction, path planning, and visualization rendering.

[0411] - Processing efficiency: Actual processing speed is dynamically adjusted based on hardware configuration (GPU model, video memory capacity) and algorithm complexity;

[0412] -FPS calculation: The real-time frame rate is calculated based on the average processing time of the last ten frames;

[0413] Oriented bounding box detection accuracy:

[0414] - Puncture needle angle measurement: calculated based on the geometric relationship between the extension line of the needle shaft and the line connecting the target; accuracy is limited by image resolution.

[0415] - Needle tip position positioning: determined by bounding box coordinate calculation with pixel-level accuracy; physical accuracy is converted from pixel spacing calibration value.

[0416] - Multi-scale adaptability: The system supports the detection of puncture needles of different lengths, the specific range of which depends on the coverage of the training dataset;

[0417] Path planning efficiency gains, performance verification of the medical A* algorithm:

[0418] Specific path planning performance implementation and verification:

[0419] Path planning performance analysis using actual parameter configuration in code:

[0420] Path generation speed characteristics:

[0421] - Grid resolution: 4 pixels / grid cell (set in medical grid configuration);

[0422] -8 Connectivity Search: Supports path selection in 8 directions (implemented in the Medical A* Algorithm module);

[0423] - Computational complexity: Reducing the mesh size can significantly reduce the search space size and improve computational efficiency;

[0424] - Time complexity: The algorithm has a time complexity of O(n log n), where n is the number of grid nodes;

[0425] -Performance analysis: Search time is positively correlated with path complexity and the number of risky areas;

[0426] Path optimization verification:

[0427] Medical constraint parameters: Angle change limit 30°, risk weight 5.0, smoothing weight 2.0 (set in medical constraint parameter configuration);

[0428] - Optimization algorithm: The A* algorithm with medical constraints is adopted, which integrates risk field and smoothness constraints;

[0429] - Path evaluation: Performance evaluation is performed by comparing the path quality of the standard A* algorithm with that of the medical enhanced A* algorithm;

[0430] - Optimization mechanism: A balance between path safety and efficiency is achieved through angle change penalties and risk weight adjustments;

[0431] - Safety analysis: The system can effectively reduce the number of times the path passes through high-risk areas, improving surgical safety;

[0432] Mathematical verification example of path planning:

[0433] Optimized mesh path planning from needle tip (100,75) to lesion (150,113):

[0434] - Heuristic function: H(n) = √[(150-100)² + (113-75)²] = √(2500 + 1444) = 62.8;

[0435] - Actual path cost: G(n) = Parent node G value + Movement cost = Parent node G value + (Basic movement cost + Risk cost + Smoothness cost);

[0436] -Specific calculation: G(n) = parent node G value + (basic movement cost + risk cost + smoothness cost), where the basic movement cost is calculated based on Euclidean distance;

[0437] - Total cost: F(n) = G(n) + H(n), used for A* search priority sorting;

[0438] Effectiveness of risk field construction:

[0439] - Risk assessment method: A hierarchical risk field is established based on the anatomical structure detection results. In one embodiment, a concentric circle risk model is used; in another embodiment, the nearest distance field (distance transformation) is calculated for each type of anatomical structure based on the segmentation mask to generate a non-circular risk field.

[0440] - Computational efficiency: In one implementation, computational optimization strategies for the risk field construction process can be adopted to reduce computational load and improve real-time performance. These optimization strategies include, but are not limited to, caching / incremental updates, resolution adaptive processing, or priority queues in open set management.

[0441] - Adaptive design: Supports dynamic adjustment of risk thresholds for different surgical types and anatomical structures;

[0442] - Evaluation mechanism: The effectiveness of risk field construction can be evaluated and verified through clinical expert assessment, retrospective case analysis, or simulation testing.

[0443] Those skilled in the art should understand that the embodiments of the present invention described above and shown in the accompanying drawings are merely examples and do not limit the invention. The advantages of the present invention have been fully and effectively realized. The functional and structural principles of the present invention have been demonstrated and explained in the embodiments; any variations or modifications can be made to the implementation of the present invention without departing from these principles.

Claims

1. A multimodal detection system for ultrasound-guided needle navigation, characterized in that, The parallel double model detection architecture is implemented, including a detection stage and a result extraction stage, wherein in the detection stage, the detection system sets unified inference parameters, simultaneously loads two YOLOv11 models at system initialization, and the two YOLOv11 models are respectively a directional bounding box model and a segmentation model, wherein the directional bounding box model is used for directional detection of a puncture needle on an input frame; and the segmentation model is used for segmentation of at least seven types of medical structures or target regions; In the result extraction stage, the detection system extracts puncture needle information from the detection result of the directional bounding box model, extracts a lesion mask from the segmentation result, and extracts a mask or contour / center information of all target structures from the segmentation result for risk field construction, and adjusts the lesion mask to the same size as the input frame; The segmentation model establishes a hierarchical safety system based on medical priorities, the category mapping is consistent with the preset category mapping table, and the corresponding relationship of the seven types of medical structures or target regions is as follows: Category 0: artery, first-level contraindicated region, danger threshold 5.0 mm, warning threshold 8.0 mm, priority 9; Category 1: vein, second-level contraindicated region, danger threshold 3.0 mm, warning threshold 5.0 mm, priority 7; Category 2: lymph node, target region, danger threshold 0.0 mm, warning threshold 0.0 mm, priority 1; Category 3: bone, obstacle region, danger threshold 0.0 mm, warning threshold 1.0 mm, priority 8; Category 4: lesion, target region, danger threshold 0.0 mm, warning threshold 0.0 mm, priority 1; Category 5: esophagus, second-level contraindicated region, danger threshold 3.0 mm, warning threshold 5.0 mm, priority 7; Category 6: diaphragm, second-level contraindicated region, danger threshold 2.0 mm, warning threshold 4.0 mm, priority 6; The danger threshold and the warning threshold are preset safety constraint modeling parameters for risk field construction and path planning, and can be configured and adjusted according to device calibration and clinical specifications; When extracting the lesion mask, the validity of the segmentation result is first verified, and then the valid mask data of the lesion category is extracted from the multi-class segmentation result; when the lesion category mask does not exist, an empty mask is output and it is prompted that the lesion target is not detected, the mask data is transmitted from the GPU memory to the CPU, and a preset segmentation threshold is applied for binary processing to convert the floating-point probability value into an integer binary mask; when the mask size and the target frame size do not match, the system uses a nearest neighbor interpolation algorithm for size adjustment; The enhanced A* algorithm with medical constraints is used for navigation path planning, and a medical grid space is established as the calculation basis of the A* algorithm at system initialization, the continuous image space is discretized into a searchable node network through gridding, and a pixel spacing parameter is set for physical-pixel coordinate conversion, wherein the medical constraint parameter system includes: An angle change constraint parameter, a recommended maximum angle change is set to 30.0 degrees, which is used to apply a strong penalty to guide the path turning; and a clinical limit angle threshold is set to 45.0 degrees, and an infinite penalty value is set when the angle change is greater than or equal to 45.0 degrees, so as to prohibit path expansion; Step length control, minimum step length 1 grid unit, maximum step length 3 grid units, to ensure the operability of the navigation path; Grid resolution, 4 pixels per grid unit, to balance the calculation accuracy and processing efficiency; Connectedness, 8-connected neighborhood search is adopted to provide 8 directions of path selection; Weight allocation strategy, risk weight factor 5.0, smoothness weight factor 2.

0.

2. The multi-modal detection system for ultrasound-guided needle navigation of claim 1, wherein, A continuous risk field is constructed in the pixel space of the original ultrasound image, and then the pixel-level risk values are aggregated into grid nodes through spatial mapping to realize the conversion from the continuous risk field to the discrete search space, and then the finally generated risk field provides guidance for the navigation path planning algorithm.

3. The multi-modal detection system for ultrasound-guided needle navigation of claim 2, wherein, The detection system first creates a zero-value initialized floating-point risk field matrix according to the input image size, and then constructs the boundary risk area and the anatomical structure risk area in layers; The boundary risk area is: within the range of 5 pixels on the top edge, 5 pixels on the bottom edge, 5 pixels on the left edge and 5 pixels on the right edge of the image, a boundary constraint is established and a preset boundary risk value is assigned; The anatomical structure risk area is: for each detected anatomical structure object, the system extracts the center coordinates, danger threshold, warning threshold and avoidance priority parameters, adopts a concentric circle hierarchical risk model to establish a two-level risk area system, in which the high-risk area is assigned an extremely high risk value of 1000.0× priority, and the warning area is assigned a medium risk value of 200.0× priority; for non-risk structure area, the system adopts zero-value initialization, and the risk value is 0.

0.

4. The multi-modal detection system for ultrasound-guided needle navigation of claim 1, wherein, In the navigation path planning process, the mobile cost architecture is integrated, and the detection system establishes a three-layer cost structure including basic mobile cost, risk cost and smoothness cost, i.e. enhanced mobile cost, in which the basic mobile cost calculates the geometric distance between the current grid point and the adjacent grid point through the Euclidean distance formula to ensure the spatial efficiency of the path, the risk cost is calculated by multiplying the risk value of the adjacent point with the risk weight factor in the medical constraint, and a scale factor of 0.001 is applied for numerical adjustment, and the smoothness cost is based on the angle change of the navigation path segment: when the current grid point has a parent node, the smoothness cost calculation algorithm calculates the angle change between the previous path segment and the next path segment: the system first calculates the cosine value of the included angle of the two path segments through vector dot product and vector length, then applies the inverse cosine function to calculate the angle change, and converts it from radian to degree; the system evaluates all angle changes: when the angle change ≤10 degrees, a negative reward is given to encourage smooth path; when the angle change is between 10-20 degrees, a linearly increasing penalty is applied; when the angle change is between 20-30 degrees, a quadratic penalty is applied; when the angle change >30 degrees, the penalty intensity increases sharply, a cubic penalty or infinite penalty is applied to strengthen the cost of uncomfortable turning, the weight factors in the three-layer cost structure are optimized based on clinical practice: the risk weight factor 5.0 ensures that the algorithm prioritizes avoiding high-risk areas based on the highest priority principle of medical safety; The standardization coefficient 0.001 is used for numerical range unification to ensure the reasonable superposition of different dimension costs.

5. The multi-modal detection system for ultrasound-guided needle navigation of claim 4, wherein, The enhanced mobile cost calculation formula is: total_cost = base_cost + risk_cost + smoothness_cost; Wherein, base_cost: base mobile cost, diagonal movement is √2≈1.414; risk_cost = neighbor.risk_value × risk_weight × scale_factor, Wherein risk_cost: risk cost, neighbor.risk_value: risk value of neighbor node, risk_weight: risk weight factor, scale_factor: scale factor; Smoothness_cost: smoothness cost, based on five-level medical safety threshold system for hierarchical calculation: When the angle change ≤10 degrees: Smoothness_cost = -0.5 × adjustment_factor, give negative value reward to encourage smooth path generation; When 10 degrees < angle change ≤20 degrees: Smoothness_cost = (angle_change-10) × smoothness_weight × adjustment_factor × 0.5, linearly increasing penalty is adopted; When 20 degrees < angle change ≤30 degrees: smoothness_cost = base_penalty + (angle_change - 20) 2 × smoothness_weight × adjustment_factor × 2.0, implementing an exponential growth penalty; When 30 degrees < angle change < 45 degrees: smoothness_cost = accumulated_base_penalty + (angle_change - 30) 3 smoothness_weight x adjustment_factor x 10.0, implementing a cubic growth penalty; When the angle change ≥45 degrees: smoothness_cost = ∞, set infinite penalty value, completely block such path; Wherein, angle_change: path turning angle change amount, smoothness_weight: smoothness weight factor, adjustment_factor: adjustment factor 0.1, accumulated_base_penalty, that is, accumulated connection penalty, is used to ensure the accumulated connection reference value of the penalty continuity at the 30-degree boundary between the 20-30-degree quadratic interval and the 30-45-degree cubic interval; Base_penalty = (20-10) × smoothness_weight × adjustment_factor × 0.5, used to ensure the penalty continuity between 10-20 degrees and 20-30 degrees interval; accumulated_base_penalty = base_penalty + (30 - 20) 2 × smoothness_weight × adjustment_factor × 2.0, to ensure that the penalties for the 20-30 degree and 30-45 degree intervals are continuous; when the angle change is ≥ 45 degrees, the penalty is infinite, or in software implementation, a large number M is used to approximate the infinite.

6. The multi-modal detection system for ultrasound-guided needle navigation of claim 4, wherein, In the navigation path planning process, the angle change penalty mechanism is also integrated. The detection system establishes an angle change threshold mechanism based on medical constraints. The angle change penalty strategy algorithm evaluates the angle change of all path segments and applies corresponding punishment or reward. The punishment intensity is determined according to the medical safety threshold interval to which the angle change amount belongs, and the smoothness weight factor and the adjustment coefficient of 0.1 are applied for numerical optimization. The punishment intensity is determined by the product of the angle change amount and the smoothness weight factor, and the adjustment coefficient of 0.1 is applied for numerical optimization.

7. The multi-modal detection system for ultrasound-guided needle navigation of claim 6, wherein, The angle change penalty strategy algorithm specifically comprises: constructing a five-level medical safety threshold system, establishing a hierarchical penalty mechanism based on the physical limitations of clinical puncture operations, and detecting the system according to the medical safety degree of the angle change amount to set five evaluation levels of a comfortable operation interval of 0-10 degrees, a clinically acceptable interval of 10-20 degrees, a high-risk operation interval of 20-30 degrees, an extremely dangerous operation interval of 30-45 degrees, and a technical prohibited interval of greater than or equal to 45 degrees, and the corresponding hierarchical penalty strategy is: Comfortable operation interval processing: when the angle change is in the optimal operation range, the system gives a negative reward factor, actively encouraging smooth path generation; Clinically acceptable interval processing: a linear increasing function is used, and the punishment intensity is in direct proportion to the angle deviation beyond the comfortable interval; High-risk operation interval processing: an exponential growth penalty mode is implemented, and the penalty value grows in the square of the angle deviation; Extremely dangerous operation interval processing: an exponential growth penalty mode is implemented, and the penalty value grows in the cube of the angle deviation; Technical prohibited interval processing: an infinite penalty value is set.

8. The multi-modal detection system for ultrasound-guided needle navigation of claim 1, wherein, The unified reasoning parameters at least include: double-model input unified specifications, confidence threshold, intersection over union threshold, segmentation binary threshold, and pixel spacing parameter.

9. The multi-modal detection system for ultrasound-guided needle navigation of claim 1, wherein, The puncture needle information includes needle tip position, angle, and bounding box parameters.

Images