A Deep Learning-Based Method for Real-Time Ultrasound Lesion Localization and Auxiliary Diagnosis of Benign and M

Abstract

This invention discloses a deep learning-based method for real-time ultrasound lesion localization and benign / malignant auxiliary diagnosis. The method acquires ultrasound video streams in real time via API or video interface, uses a YOLOv8 model optimized based on CIoU loss for lesion detection, and introduces a boundary-aware auxiliary head and K-means clustering anchor boxes to improve the localization accuracy of small lesions. Subsequently, morphological features such as lesion boundary clarity, shape regularity, and echo texture are extracted. Based on these comprehensive features, benign / malignant classification is performed, and Monte Carlo Dropout is used for uncertainty estimation and conservative dynamic calibration. The system outputs the calibrated malignancy probability, confidence level, and lesion boundary, providing structured reports and visual alerts to support physician decision-making. The processing speed is ≥15 frames / second, combining high accuracy, real-time performance, and clinical robustness.

Description

Technical Field

[0001] This invention relates to the field of medical diagnostic platform technology, and more specifically, to a method for real-time ultrasound lesion localization and benign / malignant auxiliary diagnosis based on deep learning. Background Technology

[0002] Ultrasound imaging, also known as ultrasound examination or ultrasound scanning, is a non-invasive medical diagnostic technique that uses high-frequency sound waves (ultrasound waves) to image the internal structures of the human body. Its basic principle is to emit ultrasound waves into the body; when these sound waves encounter interfaces between tissues of different densities, they are reflected, scattered, or refracted. The reflected sound waves are received by the probe and converted into electrical signals, which are then processed to form an image on a screen.

[0003] However, the interpretation of ultrasound images is highly dependent on the doctor's experience and skill level, and there may be diagnostic differences between different doctors. In addition, lesions in ultrasound images are often small in size and have blurred boundaries, which increases the risk of missed detection and misdiagnosis. Moreover, doctors have no reference when making a diagnosis and can only make judgments based on experience, which undoubtedly increases the difficulty of diagnosis.

[0004] Therefore, in order to solve the above-mentioned technical problems, this application proposes a method for real-time ultrasound lesion localization and auxiliary diagnosis of benign and malignant lesions based on deep learning. Summary of the Invention

[0005] To address the shortcomings of existing technologies, the present invention aims to provide a deep learning-based method for real-time ultrasound lesion localization and auxiliary diagnosis of benign and malignant lesions. This method includes the following steps: Step 1: Acquire the current frame image and corresponding patient clinical information from the real-time ultrasound video stream; Step 2: Input the current frame image into the YOLOv8 lesion detection model trained based on CIoU loss, and output the bounding box coordinates of the lesion region; During the model training phase, in addition to the main detection head, a lightweight boundary-aware auxiliary head is introduced. This auxiliary head outputs a single-channel boundary probability map, and its supervision signal is generated by the edge map extracted from the real mask by the Sobel operator. The CIoU loss of the main detection head and the binary cross-entropy loss of the auxiliary head are jointly optimized, so that the model can implicitly perceive the gradient distribution of the lesion boundary while learning to locate the lesion. The auxiliary head's supervision signal is generated by convolving the real mask with the Sobel operator to produce an edge map, which is then binarized to obtain the target edge map, used to supervise the output of the single-channel boundary probability map; K-means++ clustering was performed on all lesions in the training set to obtain three sets of scale-adaptive prior box sizes, which were then used as the initial prediction biases for the three detection layers of PAN-FPN. After each forward propagation, the system clips the ROI based on the predicted bounding box and calculates its compactness C = P² / (4πA) in real time. If this value exceeds the typical range of benign lesions, but the true label is benign, a penalty term is added to the total loss to guide the model to avoid generating excessively irregular predicted boxes. The regularization term is only enabled during the training phase. During the deployment phase, when no suspected lesions are detected in 10 consecutive frames of the video stream, the input resolution is automatically reduced to 320×320 to improve processing speed; once a suspected lesion is detected, it is immediately switched back to the 640×640 high-resolution mode for accurate localization. Step 3: Based on the bounding box, crop the region of interest of the lesion and extract morphological feature vectors including boundary clarity, shape regularity and echo texture. Perform structured encoding on the patient's clinical information to generate clinical feature vectors. Step 4: Using a dual-path fusion strategy combining attention and gating mechanisms, the morphological feature vectors and clinical feature vectors are fused into comprehensive diagnostic features. ; Step 5: Based on the comprehensive diagnostic features, classify the malignancy into benign and malignant, output the malignancy probability, and perform conservative dynamic calibration on the malignancy probability by combining Monte Carlo uncertainty estimation; Among them, the Monte Carlo Dropout strategy is used to process the input... Perform 50 forward propagations, each time applying Dropout to generate independent prediction probabilities. Calculate the mean As a preliminary probability of malignancy, For the first The probability of the prediction is calculated. Simultaneously, the standard deviation is calculated. As a measure of the uncertainty of the prediction; where K=50, if If the preset threshold is exceeded, a conservative dynamic calibration formula will be used. right Adjustments were made, including This is the calibration coefficient, with a value range of [0,1]. To predict the standard deviation of uncertainty, when Calibration is triggered when the variance is >0.1 to suppress high-variance prediction results; Step 6: Output the calibrated malignancy probability, corresponding confidence level, and lesion bounding box coordinates.

[0006] Preferably, the YOLOv8 lesion detection model is optimized using CIoU bounding box regression loss during the training phase. CIoU loss considers the overlap area between the predicted and ground truth bounding boxes, the distance between their center points, and the consistency of their aspect ratios. Its calculation formula is as follows:

[0007] Where loU is the intersection-union ratio. For Euclidean distance, and These are the center points of the predicted bounding box and the ground truth bounding box, respectively. The length of the diagonal of the minimum bounding box. These are the weighting coefficients. This is a measure of aspect ratio consistency.

[0008] Preferably, the boundary sharpness is obtained by calculating the mean value of pixel gradient magnitudes within the lesion region; the shape regularity includes compactness C and roundness R, wherein roundness... density Where P is the perimeter of the lesion outline and A is the area of ​​the lesion. The echo texture is calculated using the gray-level co-occurrence matrix (GLCM) and includes contrast, energy, and homogeneity.

[0009] Preferably, the patient's clinical information includes age, gender, BI-RADS classification, family history, and laboratory indicators; wherein age and laboratory indicators are standardized, gender and family history are encoded in binary, and BI-RADS classification is mapped to a vector through an embedding layer.

[0010] Preferably, the attention mechanism uses morphological feature vectors as queries and clinical feature vectors as keys and values, and calculates cross-modal attention weights through scaled dot product attention; the gating mechanism generates adaptive gating weights based on the concatenated bimodal feature vectors through a fully connected layer and a sigmoid activation function, and performs weighted fusion of image features and clinical features.

[0011] Preferably, the Monte Carlo uncertainty estimation calculates the variance of the malign probability as an uncertainty measure by enabling random deactivation and performing multiple forward propagations during the inference phase. The conservative dynamic calibration reduces the malign probability and biases towards benign judgments when the uncertainty is high.

[0012] A deep learning-based real-time ultrasound lesion localization and benign / malignant auxiliary diagnosis system includes: The image acquisition module is used to acquire the current frame image in the real-time ultrasound video stream; The clinical information input module is used to receive patient clinical information; The lesion detection module includes a YOLOv8 lesion detection model trained based on CIoU loss optimization, which is used to output the bounding box coordinates of the lesion region; The YOLOv8 lesion detection model is configured as follows: During the model training phase, in addition to the main detection head, a lightweight boundary-aware auxiliary head is introduced. This auxiliary head outputs a single-channel boundary probability map, and its supervision signal is generated from the edge map extracted from the ground truth mask using the Sobel operator. The CIoU loss of the main detection head and the binary cross-entropy loss of the auxiliary head are jointly optimized, enabling the model to implicitly perceive the gradient distribution of the lesion boundary while learning localization. The supervision signal of the auxiliary head is generated from the ground truth mask through convolution using the Sobel operator to produce an edge map, which is then binarized to obtain the target edge map, used to supervise the output of the single-channel boundary probability map. K-means++ clustering is performed based on the true aspect ratio of all lesions in the training set to obtain three sets of scale-adaptive prior box sizes, which are used as the initial prediction biases for the three detection layers of PAN-FPN. After each forward propagation, the system clips the ROI based on the predicted bounding boxes and calculates its compactness C in real time. P² / (4πA); If this value exceeds the typical range of benign lesions, but the true label is benign, a penalty term is added to the total loss to guide the model to avoid generating excessively irregular prediction boxes; the regularization term is only enabled during the training phase; during the deployment phase, when no suspected lesions are detected in 10 consecutive frames of the video stream, the input resolution is automatically reduced to 320×320 to improve processing speed; once a suspected lesion is detected, it is immediately switched back to the 640×640 high-resolution mode for accurate localization; The feature extraction module is used to extract morphological feature vectors from the lesion area and encode clinical information to generate clinical feature vectors. The feature fusion module uses a dual-path fusion strategy combining attention and gating mechanisms to fuse morphological and clinical feature vectors into comprehensive diagnostic features. ; The classification calibration module employs a Monte Carlo Dropout strategy to classify and calibrate the input data. Perform 50 forward propagations, each time applying Dropout to generate independent prediction probabilities. Calculate the mean As a preliminary probability of malignancy, For the first The probability of the prediction is calculated. Simultaneously, the standard deviation is calculated. As a measure of the uncertainty of the prediction; where K=50, if If the preset threshold is exceeded, a conservative dynamic calibration formula will be used. right Adjustments were made, including The calibration coefficient has a value range of [0,1], and is preferably 0.5; To predict the standard deviation of uncertainty, when Calibration is triggered when the variance is >0.1 to suppress high-variance prediction results; The results output module is used to output the calibrated malignancy probability, corresponding confidence level, and lesion bounding box coordinates.

[0013] Preferably, the deep learning-based real-time ultrasound lesion localization and benign / malignant auxiliary diagnosis system runs on a computing device equipped with a GPU, and can process ultrasound video streams in real time and simultaneously display lesion localization results and benign / malignant probabilities.

[0014] A computer-readable storage medium storing a computer program, which, when executed by a processor, performs a deep learning-based method for real-time ultrasound lesion localization and auxiliary diagnosis of benign and malignant lesions, with a processing frequency of not less than 15 frames per second, and synchronously outputs key morphological feature values ​​of the lesions for physician review.

[0015] Compared with the prior art, the present invention has the following beneficial effects: 1. By introducing a boundary-aware auxiliary head and a cluster-based scale-adaptive anchor box, the problems of missed detection and localization deviation caused by small lesion size and blurred boundaries in ultrasound images are effectively solved, and the detection IoU is significantly improved.

[0016] 2. A dual-path fusion strategy combining attention and gating mechanisms is adopted to enable adaptive weighting of morphological features and clinical information in diagnosis, overcoming the shortcomings of fixed feature fusion weights and lack of interaction in traditional methods. Attached Figure Description

[0017] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings: Figure 1 This is a flowchart of the present invention; Figure 2 This is a flowchart of the lesion detection module in this invention. Detailed Implementation

[0018] like Figure 1-2 As shown, this invention provides a method for real-time ultrasound lesion localization and benign / malignant auxiliary diagnosis based on deep learning, comprising the following steps: Step 1: Acquire the current frame image and corresponding patient clinical information from the real-time ultrasound video stream. During clinical scanning, the ultrasound equipment continuously outputs a B-mode ultrasound video stream, which is accessed through the software development kit (SDK) provided by the equipment manufacturer or a standard video interface. In short, the system calls the ultrasound equipment's open APIs, such as the eGate API for Philips devices or the KRETZ API for GE devices, to capture each frame of ultrasound image in real time using a frame callback mechanism. If the equipment does not support the SDK, a video capture card is connected via HDMI or SDI video output interfaces to convert analog or digital video signals into digital image frames, or via RTSP, DICOM RT... The network streaming protocols such as Image receive and decode video data; the acquired images are single-channel grayscale images, preferably with a resolution of 640×480 or 768×576, and a frame rate between 15 and 30 frames per second, which can meet the timing requirements of real-time diagnosis. The pixel values ​​of the original ultrasound images are linearly normalized to the integer range of 0 to 255 to adapt to the input format of the deep learning model. The patient information includes age, gender, BI-RADS classification, family history, and laboratory indicators; age and laboratory indicators are standardized, gender and family history are encoded in binary, and the BI-RADS classification is mapped to a vector through an embedding layer. Step 2: Input the current frame image into the YOLOv8 lesion detection model trained based on CIoU loss, and output the bounding box coordinates of the lesion region. When constructing this YOLOv8 lesion detection model, it is first necessary to train the model, and then perform the following process: A training dataset was constructed, collecting ultrasound video frames from multiple centers. Radiologists not only annotated the bounding boxes but also simultaneously delineated pixel-level masks of the lesions to generate boundary guidance signals. During model training, in addition to the main detection head, a lightweight boundary-aware auxiliary head was introduced. This auxiliary head outputs a single-channel boundary probability map, and its supervision signal is generated from the edge map extracted from the real mask using the Sobel operator. The CIoU loss of the main detection head and the binary cross-entropy loss of the auxiliary head were jointly optimized, enabling the model to implicitly perceive the gradient distribution of the lesion boundary while learning localization. This results in the generation of bounding boxes that more closely match the actual lesion contour during inference, improving the localization accuracy of malignant lesions with blurred boundaries. The auxiliary head's supervision signal is generated by convolving the real mask with a Sobel operator (3×3 kernel) to produce an edge map, which is then binarized (gradient strength > 0.5) to obtain the target edge map, used to supervise the output of the single-channel boundary probability map; To address the small and dense lesion size (e.g., 30×30) in ultrasound images, the default anchor box mechanism of YOLOv8 is reconstructed. Before training, K-means++ clustering (K=9) is performed based on the actual aspect ratio of all lesions in the training set to obtain three sets of scale-adaptive prior box sizes, which are then used as the initial prediction biases for the three detection layers of PAN-FPN. This strategy enables the model to focus on the actual size distribution of ultrasound lesions in the early training stage, thereby improving the recall rate of small lesions. Furthermore, after each forward propagation, the system clips the ROI based on the predicted bounding box and calculates its density in real time. If the value exceeds the typical range for benign lesions (e.g., C>2.5), but the true label is benign, a penalty term is added to the total loss to guide the model to avoid generating excessively irregular prediction boxes. The regularization term is only enabled during the training phase to improve the geometric rationality of the detection boxes. After the model training is completed, dynamic input resolution is used during the deployment phase: when no suspected lesions are detected in 10 consecutive frames of the video stream, the input resolution is automatically reduced to 320×320 to improve processing speed; once a suspected lesion is detected (IoU>0.3), it is immediately switched back to the 640×640 high-resolution mode for accurate localization. The YOLOv8 lesion detection model is optimized using CIoU bounding box regression loss during the training phase. CIoU loss considers the overlap area between the predicted and ground truth bounding boxes, the center point distance, and aspect ratio consistency. Its calculation formula is as follows:

[0019] Where loU is the intersection-union ratio. For Euclidean distance, and These are the center points of the predicted bounding box and the ground truth bounding box, respectively. The length of the diagonal of the minimum bounding box. These are the weighting coefficients. To measure aspect ratio consistency, the system optimizes the YOLOv8 model using the CIoU loss function, and defines the CIoU similarity index using the following formula: ; Step 3: Based on the bounding box, crop the region of interest of the lesion and extract morphological feature vectors, including boundary sharpness, shape regularity, and echo texture. Perform structured encoding on the patient's clinical information to generate clinical feature vectors. Boundary sharpness is obtained by calculating the mean value of pixel gradient magnitudes within the lesion region; shape regularity includes density. and roundness Where P is the perimeter of the lesion outline and A is the area of ​​the lesion. The echo texture is calculated by the gray-level co-occurrence matrix (GLCM), including contrast, energy and homogeneity. Based on the lesion region mask output in step two, the geometric properties of the lesion are first calculated; for example, the circularity can be calculated by measuring the area A and the perimeter P of the lesion. This index reflects the regularity of the lesion shape; a value closer to 1 indicates a shape closer to a circle. Simultaneously, to quantify the complexity of the boundary contour, the Fourier descriptor method is used to perform high-frequency component analysis on the boundary point sequence, generating a boundary irregularity BI, whose value is positively correlated with the fractal degree of the boundary. Furthermore, the ratio of the longer side L to the shorter side W of the lesion's minimum bounding rectangle is calculated. This can assess the flatness of the lesion; for the homogeneity of internal echoes, the standard deviation 0 is calculated based on the gray-level histogram of the lesion area and mapped to... The degree of dispersion of echo distribution is numerically represented; The aforementioned features (including density, boundary irregularity, aspect ratio, and echo uniformity) together constitute the morphological feature vector. As input to the diagnostic model, it is combined with clinical feature vectors to participate in the final prediction.

[0020] Step 4: Using a dual-path fusion strategy that combines attention and gating mechanisms, the morphological feature vectors and clinical feature vectors are fused into comprehensive diagnostic features; Based on the morphological feature vector output in step three (Including boundary sharpness, shape regularity, and echo texture) and clinical feature vectors (Including structured information such as age and medical history), a dual-path fusion strategy combining attention and gating mechanisms is used to generate comprehensive diagnostic features Fdiag. Specifically: Attention mechanism: based on morphological feature vectors As a query, clinical feature vector Cross-modal attention weights are computed using scaled dot product attention, with the weights serving as both keys and values. ,in For feature dimension, It was used to quantify the correlation strength between morphological feature vectors and clinical feature vectors. Gating mechanism: Concatenates morphological feature vectors and clinical feature vectors into a bimodal feature vector. The input is a fully connected layer and the adaptive gating weights are generated by the Sigmoid activation function. Ultimately, through weighted fusion Output comprehensive diagnostic features.

[0021] Step 5: Based on the comprehensive diagnostic features, classify malignancy into benign and malignant, output the malignancy probability, and perform conservative dynamic calibration on the malignancy probability in combination with Monte Carlo uncertainty estimation. Monte Carlo uncertainty estimation calculates the variance of the malignancy probability as an uncertainty measure by enabling random deactivation and performing multiple forward propagations during the inference stage. The conservative dynamic calibration reduces the malignancy probability when the uncertainty is high, biasing towards benign judgment. Based on step four Construct a fully connected neural network (FCN) for benign / malignant classification, input... The malignancy probability is output after passing through 3 layers of FCN (128 nodes per layer, ReLU activation). The loss function is cross-entropy; Uncertainty calibration is achieved by statistically analyzing the mean and standard deviation of these prediction distributions, using a Monte Carlo Dropout strategy on the input. Perform 50 forward propagations, each time applying Dropout to generate independent prediction probabilities. ,calculate As a preliminary probability of malignancy, For the first The probability of the next prediction. Simultaneously, calculate... As a measure of the uncertainty of prediction; if If the preset threshold is exceeded, a conservative dynamic calibration formula will be used. right Adjustments were made, including The calibration coefficient has a value range of [0,1], and is preferably 0.5; To predict the standard deviation of uncertainty, when Calibration is triggered when the variance is >0.1 to suppress high-variance prediction results; Step Six: After completing the dynamic calibration in Step Five, the system enters the result output stage. At this point, the system will output the calibrated malignancy probability. Corresponding confidence level The lesion bounding box coordinates obtained in step two are integrated into a complete diagnostic result. These outputs are transmitted to the medical terminal through a unified interface so that doctors can make decisions based on their clinical experience. For example, this deep learning-based real-time ultrasound lesion localization and benign / malignant auxiliary diagnosis method acquires ultrasound video streams in real time via a medical imaging API. If no lesions are detected in the first 5 frames, the system automatically reduces the input resolution to 320×320. After a suspected lesion is detected in the 6th frame, the system switches to a 640×640 high-resolution mode. At this point, the input model is a 640×640 high-resolution image, and the main detection head outputs the lesion bounding box coordinates. The actual lesion size is 28×24mm. The boundary perception auxiliary head outputs a single-channel boundary probability map, displaying the lesion edge ambiguity. The mean gradient strength is 0.35. Then, anchor boxes generated by K-means clustering are used. The small-size group (26×22) highly matches the lesion size, improving the recall rate. The platform then uses a formula... The scale and orientation deviations between the predicted bounding boxes and the ground truth bounding boxes were corrected, and the final detection IoU was 0.89. Then, the bounding box region of the lesion (28×24mm) was input. The area ratio of the lesion to the minimum bounding rectangle was 0.72, the aspect ratio AR was 1.17, and the echo homogeneity (EH) was 0.68. This indicates low shape regularity, an elliptical shape, and heterogeneous internal echoes, suggesting a possible malignancy. The morphological output vector... , and then with and As the input vector, calculate the cross-modal attention weights. The result is This means that morphological features have higher weights, and then features are concatenated through a gating mechanism. Input the fully connected layer and generate gate weights By weighted fusion of comprehensive diagnostic features ; enter After three layers of FCN (ReLU activation), the initial malignancy probability p=0.72 is output. After 50 forward propagations, the predicted probability distribution is obtained. Statistical calculation of mean Standard deviation , here because <0.1, no calibration required, final output At this point, the system indicates a 71% probability of malignancy for the lesion, which is very high, and recommends a biopsy. The diagnostic result also includes... This indicates that the result is reliable.

[0022] In summary, the overall workflow of this deep learning-based method for real-time ultrasound lesion localization and auxiliary diagnosis of benign and malignant lesions is as follows: First, the system acquires ultrasound video streams in real time via API or SDK interfaces and dynamically adjusts the input resolution based on the video content—reducing to 320×320 for improved processing efficiency when multiple consecutive frames are lesion-free, and immediately switching to 640×640 high resolution for precise localization once a suspected lesion is detected. Subsequently, the input image is fed into a YOLOv8 lesion detection model trained with CIoU loss optimization. This model significantly improves the localization accuracy of lesions with blurred boundaries by introducing a boundary-aware auxiliary head (outputting a single-channel boundary probability map) and scale-adaptive anchor boxes generated by K-means++ clustering (30×30 level ultrasound small lesions). After lesion detection, the system extracts morphological feature vectors from the bounding boxes and simultaneously acquires structured clinical information such as patient age and medical history. These two types of features interact through a dual-path fusion strategy: using morphological feature vectors as queries and clinical feature vectors as keys and values, cross-modal associations are quantified through a scaled dot product attention mechanism; further, the concatenated bimodal features are input into a fully connected layer to generate adaptive gating weights, achieving dynamic weighted fusion of morphological and clinical feature vectors to generate comprehensive diagnostic features. Finally, the comprehensive diagnostic features are input into a fully connected neural network for benign / malignant classification, and a Monte Carlo Dropout strategy is used for 50 forward propagations to calculate the mean and standard deviation of the predicted probabilities. If the standard deviation exceeds a preset threshold, a conservative dynamic calibration formula is applied. This effectively suppresses the risk of misdiagnosis in samples with high uncertainty. The entire process, through the synergistic effect of boundary-aware detection, cross-modal attention fusion, and uncertainty calibration, solves technical challenges such as ambiguous localization of small lesions in ultrasound images, fixation of multimodal feature fusion, and high-risk misdiagnosis, forming an innovative diagnostic system that combines real-time performance, accuracy, and clinical robustness.

[0023] A deep learning-based real-time ultrasound lesion localization and benign / malignant auxiliary diagnosis system includes: The image acquisition module is used to acquire the current frame image in the real-time ultrasound video stream; The clinical information input module is used to receive patient clinical information; The lesion detection module includes a YOLOv8 lesion detection model trained based on CIoU loss optimization, which is used to output the bounding box coordinates of the lesion region; The YOLOv8 lesion detection model is configured as follows: During the model training phase, in addition to the main detection head, a lightweight boundary-aware auxiliary head is introduced. This auxiliary head outputs a single-channel boundary probability map, and its supervision signal is generated from the edge map extracted from the ground mask by the Sobel operator. The CIoU loss of the main detection head and the binary cross-entropy loss of the auxiliary head are jointly optimized, so that the model implicitly perceives the gradient distribution of the lesion boundary while learning localization. The supervision signal of the auxiliary head is generated from the ground mask by convolution with the Sobel operator (3×3 kernels) to produce an edge map, which is then binarized (gradient strength > 0.5) to obtain the target edge map, which is used to supervise the output of the single-channel boundary probability map. K-means++ clustering (K=9) is performed based on the true aspect ratio of all lesions in the training set to obtain three sets of scale-adaptive prior box sizes, which are used as the initial prediction biases of the three detection layers of PAN-FPN. After each forward propagation, the system clips the ROI based on the predicted bounding box and calculates its compactness C in real time. P² / (4πA); If this value exceeds the typical range for benign lesions (e.g., C>2.5), but the true label is benign, a penalty term is added to the total loss to guide the model to avoid generating excessively irregular prediction boxes; the regularization term is only enabled during the training phase; during the deployment phase, when no suspected lesions are detected in 10 consecutive frames of the video stream, the input resolution is automatically reduced to 320×320 to improve processing speed; once a suspected lesion is detected (IoU>0.3), it is immediately switched back to the 640×640 high-resolution mode for accurate localization; The feature extraction module is used to extract morphological feature vectors from the lesion area and encode clinical information to generate clinical feature vectors. The feature fusion module uses a dual-path fusion strategy combining attention and gating mechanisms to fuse morphological and clinical feature vectors into comprehensive diagnostic features. ; The classification calibration module employs a Monte Carlo Dropout strategy to classify and calibrate the input data. Perform 50 forward propagations, each time applying Dropout to generate independent prediction probabilities. Calculate the mean As a preliminary probability of malignancy, For the first The probability of the prediction is calculated. Simultaneously, the standard deviation is calculated. As a measure of the uncertainty of the prediction; where K is 50, if If the preset threshold is exceeded, a conservative dynamic calibration formula will be used. right Adjustments were made, including This is the calibration coefficient, with a value range of [0,1]. To predict the standard deviation of uncertainty, when Calibration is triggered when the variance is >0.1 to suppress high-variance prediction results; The results output module is used to output the calibrated malignancy probability, corresponding confidence level, and lesion bounding box coordinates.

[0024] A computer-readable storage medium storing a computer program, which, when executed by a processor, performs a deep learning-based method for real-time ultrasound lesion localization and auxiliary diagnosis of benign and malignant lesions, with a processing frequency of not less than 15 frames per second, and synchronously outputs key morphological feature values ​​of the lesions for physician review.

[0025] Preferably, the deep learning-based real-time ultrasound lesion localization and benign / malignant auxiliary diagnosis system runs on a computing device equipped with a GPU, and can process ultrasound video streams in real time and simultaneously display lesion localization results and benign / malignant probabilities.

[0026] The foregoing has only described certain exemplary embodiments of the present invention by way of illustration. Undoubtedly, those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the foregoing drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

Claims

1. A method for real-time ultrasound lesion localization and auxiliary diagnosis of benign and malignant lesions based on deep learning, characterized in that, Includes the following steps: Step 1: Acquire the current frame image and corresponding patient clinical information from the real-time ultrasound video stream; Step 2: Input the current frame image into the YOLOv8 lesion detection model trained based on CIoU loss, and output the coordinates of the lesion region bounding box; During the model training phase, in addition to the main detection head, a lightweight boundary-aware auxiliary head is introduced. This auxiliary head outputs a single-channel boundary probability map, and its supervision signal is generated by the edge map extracted from the real mask by the Sobel operator. The CIoU loss of the main detection head and the binary cross-entropy loss of the auxiliary head are jointly optimized, so that the model can implicitly perceive the gradient distribution of the lesion boundary while learning to locate the lesion. The auxiliary head's supervision signal is generated by convolving the real mask with the Sobel operator to produce an edge map, which is then binarized to obtain the target edge map, used to supervise the output of the single-channel boundary probability map; K-means++ clustering was performed on all lesions in the training set to obtain three sets of scale-adaptive prior box sizes, which were then used as the initial prediction biases for the three detection layers of PAN-FPN. After each forward propagation, the system clips the ROI based on the predicted bounding box and calculates its compactness C = P² / (4πA) in real time. If this value exceeds the typical range of benign lesions, but the true label is benign, a penalty term is added to the total loss to guide the model to avoid generating excessively irregular predicted boxes. The regularization term is only enabled during the training phase. During the deployment phase, when no suspected lesions are detected in 10 consecutive frames of the video stream, the input resolution is automatically reduced to 320×320 to improve processing speed. Once a suspected lesion is detected, immediately switch back to the 640×640 high-resolution mode for precise localization; Step 3: Based on the bounding box, crop the region of interest of the lesion and extract morphological feature vectors including boundary clarity, shape regularity and echo texture. At the same time, perform structured encoding on the patient's clinical information to generate clinical feature vectors. Step 4: Using a dual-path fusion strategy combining attention and gating mechanisms, the morphological feature vectors and clinical feature vectors are fused into comprehensive diagnostic features. ; Step 5: Based on the comprehensive diagnostic features, classify the malignancy into benign and malignant, output the malignancy probability, and perform conservative dynamic calibration on the malignancy probability by combining Monte Carlo uncertainty estimation; Among them, the Monte Carlo Dropout strategy is used to process the input... Perform 50 forward propagations, each time applying Dropout to generate independent prediction probabilities. Calculate the mean As a preliminary probability of malignancy, For the first The probability of the prediction is calculated, and the standard deviation is also calculated. As a measure of the uncertainty of the prediction; where K is 50, if If the preset threshold is exceeded, a conservative dynamic calibration formula will be used. right Adjustments were made, including This is the calibration coefficient, with a value range of [0,1]. To predict the standard deviation of uncertainty, when Calibration is triggered when the variance is >0.1 to suppress high-variance prediction results; Step 6: Output the calibrated malignancy probability, corresponding confidence level, and lesion bounding box coordinates.

2. The method for real-time ultrasound lesion localization and benign / malignant auxiliary diagnosis based on deep learning according to claim 1, characterized in that: The YOLOv8 lesion detection model is optimized using CIoU bounding box regression loss during the training phase. CIoU loss considers the overlap area between the predicted and ground truth bounding boxes, the distance between their center points, and the consistency of their aspect ratios. Its calculation formula is as follows: Where loU is the intersection-union ratio. For Euclidean distance, and These are the center points of the predicted bounding box and the ground truth bounding box, respectively. The length of the diagonal of the minimum bounding box. These are the weighting coefficients. This is a measure of aspect ratio consistency.

3. The method for real-time ultrasound lesion localization and benign / malignant auxiliary diagnosis based on deep learning according to claim 2, characterized in that: The boundary sharpness is obtained by calculating the mean value of pixel gradient magnitudes within the lesion region; the shape regularity includes compactness C and roundness R, wherein roundness... density Where P is the perimeter of the lesion outline and A is the area of ​​the lesion. The echo texture is calculated using the gray-level co-occurrence matrix (GLCM) and includes contrast, energy, and homogeneity.

4. The method for real-time ultrasound lesion localization and benign / malignant auxiliary diagnosis based on deep learning according to claim 3, characterized in that: The patient's clinical information includes age, gender, BI-RADS classification, family history, and laboratory indicators; among which age and laboratory indicators are standardized, gender and family history are encoded in binary, and BI-RADS classification is mapped to a vector through an embedding layer.

5. The method for real-time ultrasound lesion localization and benign / malignant auxiliary diagnosis based on deep learning according to claim 4, characterized in that: The attention mechanism uses morphological feature vectors as queries and clinical feature vectors as keys and values, and calculates cross-modal attention weights by scaling dot product attention. The gating mechanism generates adaptive gating weights based on the concatenated bimodal feature vectors through a fully connected layer and a sigmoid activation function, and performs weighted fusion of image features and clinical features.

6. The method for real-time ultrasound lesion localization and benign / malignant auxiliary diagnosis based on deep learning according to claim 5, characterized in that: The Monte Carlo uncertainty estimation calculates the variance of the malign probability as an uncertainty measure by enabling random deactivation and performing multiple forward propagations during the inference phase. The conservative dynamic calibration reduces the malign probability and biases towards benign judgments when the uncertainty is high.

7. A deep learning-based real-time ultrasound lesion localization and benign / malignant auxiliary diagnosis system, characterized in that, include: The image acquisition module is used to acquire the current frame image in the real-time ultrasound video stream; The clinical information input module is used to receive patient clinical information; The lesion detection module includes a YOLOv8 lesion detection model trained based on CIoU loss optimization, which is used to output the bounding box coordinates of the lesion region; The YOLOv8 lesion detection model is configured as follows: During the model training phase, in addition to the main detection head, a lightweight boundary-aware auxiliary head is introduced. This auxiliary head outputs a single-channel boundary probability map, and its supervision signal is generated by the edge map extracted from the ground mask using the Sobel operator. The CIoU loss of the main detection head and the binary cross-entropy loss of the auxiliary head are jointly optimized, enabling the model to implicitly perceive the gradient distribution of the lesion boundary while learning localization. The supervision signal of the auxiliary head is generated by convolving the ground mask with the Sobel operator to produce an edge map, which is then binarized to obtain the target edge map, used to supervise the output of the single-channel boundary probability map. K-means++ clustering is performed on all lesions in the training set to obtain three sets of scale-adaptive prior box sizes, which are used as the initial prediction biases for the three detection layers of PAN-FPN. After each forward propagation, the system clips ROIs based on the predicted bounding boxes and calculates their compactness C = P² / (4πA) in real time. If this value exceeds the typical range of benign lesions, but the true label is benign, a penalty term is added to the total loss to guide the model to avoid generating excessively irregular prediction boxes. The regularization term is only enabled during the training phase. During the deployment phase, when no suspected lesions are detected in 10 consecutive frames of the video stream, the input resolution is automatically reduced to 320×320 to improve processing speed. Once a suspected lesion is detected, immediately switch back to the 640×640 high-resolution mode for precise localization; The feature extraction module is used to extract morphological feature vectors from the lesion area and encode clinical information to generate clinical feature vectors. The feature fusion module uses a dual-path fusion strategy combining attention and gating mechanisms to fuse morphological and clinical feature vectors into comprehensive diagnostic features. ; The classification calibration module employs a Monte Carlo Dropout strategy to classify and calibrate the input data. Perform 50 forward propagations, each time applying Dropout to generate independent prediction probabilities. Calculate the mean As a preliminary probability of malignancy, For the first The probability of the prediction is calculated, and the standard deviation is also calculated. As a measure of the uncertainty of the prediction; where K is 50, if If the preset threshold is exceeded, a conservative dynamic calibration formula will be used. right Adjustments were made, including This is the calibration coefficient, with a value range of [0,1]. To predict the standard deviation of uncertainty, when Calibration is triggered when the variance is >0.1 to suppress high-variance prediction results; The results output module is used to output the calibrated malignancy probability, corresponding confidence level, and lesion bounding box coordinates.

8. The deep learning-based ultrasound real-time lesion localization and benign / malignant auxiliary diagnosis system according to claim 7, characterized in that: The deep learning-based real-time ultrasound lesion localization and benign / malignant auxiliary diagnosis system runs on a computing device equipped with a GPU, and can process ultrasound video streams in real time and simultaneously display lesion localization results and benign / malignant probabilities.

9. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the deep learning-based ultrasound real-time lesion localization and benign / malignant auxiliary diagnosis method according to claim 1, with a processing frequency of not less than 15 frames / second, and synchronously outputs key morphological feature values ​​of the lesion for physician review.

Images