Breast ultrasonic image processing method, system and device and storage medium

Abstract

The invention provides a mammary gland ultrasonic image processing method, system and device and a storage medium, and belongs to the field of medical image processing, and the method comprises the steps: collecting mammary gland ultrasonic images and multi-source data of the same patient at two different dates, and carrying out the preprocessing; extracting a plurality of morphological features, a plurality of acoustic features and a depth visual feature of the breast ultrasound image; performing element-by-element subtraction on the depth visual features of the two different images, and performing L2 normalization processing on a result obtained by subtraction to obtain difference features; performing weighting processing on the difference features, and splicing the plurality of morphological features and the plurality of acoustic features with the weighted difference features to form a multi-modal preliminary feature set; splicing the multi-modal preliminary feature set with multi-source data to form a time sequence multi-modal fusion feature vector; and inputting the time sequence multi-modal fusion feature vector into a pre-trained multi-task classification network, and outputting a tumor morphological change classification result. The feasibility of clinical application is improved, and the dilemma that in the prior art, result errors are large, and the popularization rate is low is effectively improved.

Description

Technical Field

[0001] This invention belongs to the field of medical image processing technology, specifically relating to a method, system, device, and storage medium for processing breast ultrasound images. Background Technology

[0002] Breast cancer is one of the most common malignant tumors in women, and its early diagnosis has always been a challenge in clinical medicine. Imaging examinations are an important means of early diagnosis of breast cancer. These examinations include breast ultrasound, mammography, and breast magnetic resonance imaging (MRI). Ultrasound, due to its non-invasiveness, convenience, and lack of radiation, has become the preferred method for screening and preliminary diagnosis. The analysis of ultrasound images provides a reliable reference for the early diagnosis of breast cancer.

[0003] Ultrasound image analysis primarily relies on morphological features and blood flow signal assessment, commonly performed manually. Physicians interpret ultrasound results by observing morphological indicators such as lesion size, boundaries, shape, internal echoes, calcifications, and posterior echo attenuation, combined with blood flow richness (RI and PI values). The limitations of manual analysis include its susceptibility to physician experience and subjective judgment, leading to missed or misdiagnosed early, small lesions or atypical lesions; insufficient diagnostic consistency among physicians; and significant differences in diagnostic capabilities among primary healthcare institutions. To address these issues, a popular approach is to utilize computer-aided intelligent technology for more precise ultrasound image analysis. This involves image segmentation, feature extraction, and machine learning algorithms (such as support vector machines and neural networks) to quantitatively analyze ultrasound images, assisting physicians in identifying suspicious lesions and improving diagnostic efficiency.

[0004] Due to the unique structure of the breast, ultrasound images often exhibit blurred boundaries and microcalcifications. Furthermore, images vary among different individuals; for example, obese patients have denser glandular tissue, and differences in probe operation can lead to variations in ultrasound images. Therefore, existing computer-aided technologies employ algorithms with weak generalization capabilities, are easily affected by image quality, and are prone to errors in their output, limiting their widespread clinical application. Summary of the Invention

[0005] To address the shortcomings of existing computer-aided technologies, such as weak generalization ability, susceptibility to image quality issues, and erroneous output results, this invention provides a breast ultrasound image processing method. This invention focuses on capturing temporal dynamic changes, fusing multimodal information, and adaptively optimizing image quality. It analyzes breast ultrasound images of the same patient at different times, extracting temporal feature differences through a Siamese network and combining this with clinical data to enhance analytical specificity. Simultaneously, image quality standardization addresses the insufficient generalization ability caused by low-quality images, ultimately achieving simultaneous multi-task output and providing comprehensive reference for clinical diagnosis.

[0006] To achieve the above objectives, the present invention provides the following technical solution: A method for processing breast ultrasound images includes the following steps: Breast ultrasound images of the same patient from two different dates were acquired, and the patient's clinical data, molecular marker data, and lifestyle data were acquired simultaneously. Multi-level ROI division was performed on the breast ultrasound images from different dates to obtain multi-level ROI images. The clinical data, molecular marker data, and lifestyle data were then normalized. Multiple morphological features characterizing tumor structural attributes and multiple acoustic features characterizing tumor ultrasound signal attributes are extracted from multi-level ROI images. The multi-level ROI images are input into a temporal twin network, and the two branches of the temporal twin network process multi-level ROI images from two different dates, respectively, and output the depth visual features of the two different images. The depth visual features of two different images are subtracted element-wise, and the result is L2 normalized to obtain the difference features. The difference features are then weighted, and multiple morphological features and multiple acoustic features are concatenated with the weighted difference features to form a preliminary multimodal feature set. The preliminary multimodal feature set is then concatenated with normalized clinical data, molecular marker data, and lifestyle data to form a temporal multimodal fusion feature vector. The temporal multimodal fusion feature vector is input into a pre-trained multi-task classification network, which outputs the tumor morphological change classification results.

[0007] Preferably, the temporal Siamese network uses the Siamese Siamese network as a framework, comprising two branch networks M1 with identical structures and real-time synchronized parameters. A pre-trained ResNet50 is used as the backbone network of each branch. A multi-scale feature fusion module and a convolutional block attention module (CBAM) are sequentially inserted after the conv5_x layer of each ResNet50 branch, forming the temporal Siamese-ResNet50 network. The branch networks process multi-level ROI images to output deep visual features, including the following steps: The pre-trained ResNet50 layers conv3_x, conv4_x, and conv5_x output low-dimensional detail features, mid-dimensional structural features, and high-dimensional semantic features, respectively. The multi-scale feature fusion module fuses the low-dimensional detail features, mid-dimensional structural features, and high-dimensional semantic features into multi-scale integrated features. Then, the CBAM module applies attention weights to the multi-scale integrated features, and finally outputs deep visual features.

[0008] Preferably, before processing multi-level ROI images through a temporal Siamese network, the process further includes training the temporal Siamese network, the training comprising the following steps: Collect breast ultrasound images from multiple patients at at least two time points, with an interval of 1-6 months; simultaneously collect the patient's clinical data, molecular marker data, and lifestyle data; All breast ultrasound images were scaled to a uniform size of 512×512 pixels using bilinear interpolation. The contrast of these uniformly sized images was then enhanced using the CLAHE algorithm. After contrast enhancement, the images were first subjected to a 5×5 Gaussian filter, followed by a 3×3 median filter. The filtered images were then divided into multi-level regions of interest (ROIs) to obtain multi-level ROI images. Clinical and molecular biomarker data were normalized using Z-Score, while lifestyle data were normalized using Min-Max. A basic dataset was constructed using the normalized clinical, molecular biomarker, and lifestyle data, along with the multi-level ROI images. The constructed temporal Siamese network is trained using a base dataset.

[0009] Preferably, the loss function for training the temporal Siamese network is: ; Where represents the loss function of the i-th classification task, and represents the uniformity of the label value distribution in the i-th classification task.

[0010] Preferably, the clinical data includes patient age, breast density grade, family history of breast cancer, and previous imaging reports.

[0011] Preferably, the morphological features include the tumor's aspect ratio, perimeter, area, roundness, eccentricity, boundary irregularity, number of depressions, and maximum diameter change rate; the acoustic features include the tumor's internal echo uniformity, posterior echo attenuation value, number of calcifications, average size of calcifications, and edge echo intensity.

[0012] Preferably, the tumor morphological change classification results include changes in the shape of the mass, changes in the direction of the mass, changes in the edge of the mass, changes in internal echoes, changes in posterior echoes, calcification, and the growth rate of the mass.

[0013] The present invention also provides a breast ultrasound image processing system, comprising: The data acquisition and processing module is used to acquire breast ultrasound images of the same patient on two different dates, and simultaneously acquire the patient's clinical data, molecular marker data, and lifestyle data; perform multi-level ROI division on the breast ultrasound images from different dates to obtain multi-level ROI images; and perform normalization processing on the clinical data, molecular marker data, and lifestyle data. The feature extraction module is used to extract multiple morphological features representing tumor structural attributes and multiple acoustic features representing tumor ultrasound signal attributes based on multi-level ROI images; the multi-level ROI images are input into a temporal twin network, and the two branches of the temporal twin network process multi-level ROI images from two different dates respectively, and output the depth visual features of the two different images. The feature fusion module is used to subtract the depth visual features of two different images element by element, and then perform L2 normalization on the result to obtain the differential features. The differential features are then weighted and multiple morphological features and multiple acoustic features are concatenated with the weighted differential features to form a preliminary multimodal feature set. The preliminary multimodal feature set is then concatenated with normalized clinical data, molecular marker data, and lifestyle data to form a temporal multimodal fusion feature vector. The classification module is used to input the temporal multimodal fusion feature vector into the pre-trained multi-task classification network and output the tumor morphological change classification result.

[0014] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of any one of the breast ultrasound image processing methods.

[0015] The present invention also provides a computer-readable storage medium storing a computer program that, when loaded by a processor, can execute any of the steps in the breast ultrasound image processing method.

[0016] The breast ultrasound image processing method provided by this invention has the following beneficial effects: This invention precisely addresses the problems of blurred ultrasound image boundaries, significant individual variability, and weak algorithm generalization ability through a temporal multimodal fusion strategy. First, temporal ultrasound images and multidimensional clinical data from the same patient are collected, combined with multi-level ROI segmentation. This focuses on key lesion areas while preserving hierarchical structural information, overcoming the shortcomings of single images in capturing blurred boundaries and microcalcifications. Next, a temporal twin network extracts depth visual features from images from two different time periods. These features are then subtracted and normalized using L2 normalization to obtain differential features, which are then weighted. Simultaneously, extracted morphological and acoustic features are fused. Further, normalized clinical, molecular marker, and lifestyle data are stitched together to form a temporal multimodal fusion feature vector. This effectively integrates temporal change information and individual variability-related data, improving the comprehensiveness and robustness of feature representation. Finally, the results are input into a pre-trained multi-task classification network to output tumor morphological changes. The temporal information enhances the sensitivity to changes in micro-lesions, and multimodal fusion reduces the impact of individual variability and image quality on the results, significantly improving the algorithm's generalization ability and diagnostic accuracy, providing reliable support for the accurate assessment of early-stage breast cancer. Attached Figure Description

[0017] To more clearly illustrate the embodiments and design schemes of the present invention, the accompanying drawings required for this embodiment will be briefly described below. The drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart of the breast ultrasound image processing method according to Embodiment 1 of the present invention; Figure 2 It is a temporal twin network learning model. Detailed Implementation

[0019] To enable those skilled in the art to better understand and implement the technical solutions of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and should not be construed as limiting the scope of protection of the present invention.

[0020] This invention addresses the challenges of blurred boundaries in breast ultrasound images, difficulty in identifying microcalcifications, and strong image heterogeneity due to differences in patient populations and procedures. It overcomes the limitations of traditional static single-modal analysis by constructing a comprehensive intelligent processing system encompassing time-series data acquisition, multi-dimensional preprocessing, cross-modal feature fusion, time-series difference learning, and multi-task accurate classification. Through dynamic comparison of time-series images, multi-modal information fusion, and an improved deep learning network design, it significantly improves the accuracy, generalization ability, and clinical applicability in distinguishing between benign and malignant breast tumors, providing efficient and reliable intelligent auxiliary support for early breast cancer diagnosis.

[0021] Example 1 First, this invention provides a model for processing breast ultrasound images and a method for training the model, comprising the following steps: Step 1. Data collection.

[0022] Since single-time-point images cannot capture the dynamic changes of tumors, a 1-6 month interval can capture subtle morphological changes in early malignant tumors, such as the transformation of edges from smooth to rough; while benign tumors remain relatively stable in morphology within this period. Therefore, this invention acquires breast ultrasound images from multiple patients at at least two time points, with an interval of 1-6 months, covering the early morphological change cycle of malignant tumors. The ultrasound images acquired in both acquisitions must be of uniform specifications, such as resolution and pixel spacing. The dynamic time window is adaptively adjusted according to the actual situation, shortening the interval to 1-3 months for patients with high-density glandular tissue and extending it to 3-6 months for patients with low-density glandular tissue, improving the targeted capture of subtle early changes.

[0023] In addition to ultrasound images, clinical, molecular marker, and lifestyle data were simultaneously collected from the patient. Clinical data included patient age, breast density grading (ACR grading), family history of breast cancer, and previous imaging reports. Molecular marker data included CA15-3 and HER2 expression levels, while lifestyle data included smoking history and exercise frequency. Combining clinical, molecular marker, and lifestyle data with imaging data reduces the bias of isolated image interpretations. For example, high-density breast tissue can easily obscure masses, requiring adjustments to the analysis weights based on density grading. This overcomes the information limitations of single images, improves processing accuracy, enhances generalization ability, and adapts to complex clinical scenarios.

[0024] Building upon this foundation, the coordinates of the lesion region, six tumor morphological change features (see Table 1), and biopsy labels (benign / malignant, with pathological examination results as the gold standard) on ultrasound images were manually annotated by two physicians. Dual physician annotation reduces subjective error and improves the reliability of the dataset annotation. Furthermore, dynamic change labels (such as the addition / disappearance of calcifications and changes in the number of edge depressions) are added to enrich the annotation dimensions.

[0025] This invention combines a tumor morphology feature change learning task as an auxiliary task with a tumor benign / malignant classification task. The descriptions of changes in ultrasound image features of breast tumors are shown in Table 1.

[0026] Table 1 Description of changes in ultrasound characteristics This invention expands the information dimensions of the dataset from a single image to time-series images, clinical data, molecular biomarker data, lifestyle data, and professional annotations, providing a foundation for subsequent multimodal fusion.

[0027] Step 2. Data preprocessing.

[0028] Image Standardization: Due to the large differences in ultrasound image resolution, with different devices outputting resolutions ranging from 256×256 to 1024×1024, this invention first performs image standardization: Bilinear interpolation is used to scale all images to 512×512 pixels to match the input requirements of ResNet50, achieving image size uniformity and avoiding inconsistencies in feature scale caused by size differences. Uniform size ensures consistent input feature scales for the network. The uniformly sized images are then processed using the contrast-limited adaptive histogram equalization (CLAHE) algorithm, increasing image contrast by 20%-30%. Simultaneously, the clipping threshold is dynamically adjusted from 1.5 to 2.5 based on the image grayscale distribution, avoiding detail loss or noise amplification caused by a fixed threshold. Since the original ultrasound image has low contrast, CLAHE can specifically enhance the contrast of local areas, such as the boundary between masses and adipose tissue.

[0029] Noise Removal and Registration: Single filtering can easily lead to blurred edges, while a combination of dual filters can preserve edge details while removing noise. Therefore, this invention first uses a 5×5 Gaussian filter to smooth the typical high-frequency speckle noise in the contrast-enhanced ultrasound image; then, a 3×3 median filter is used to process the salt-and-pepper noise that may be generated during the acquisition process, preserving high-frequency key features such as tumor edges and calcifications. Simultaneously, the ultrasound images from two examinations may experience feature shifts due to differences in equipment, doctor's operating angle, and patient position, necessitating alignment of the tumor region in the two denoised images. This invention uses a Gaussian filter + median filter combination for denoising, followed by a new deep learning registration network (based on the U-Net architecture). This network, combined with patient position parameters and probe operation records, achieves sub-pixel-level registration of the tumor region, with a registration error ≤0.8 pixels, correcting feature shifts caused by operational differences.

[0030] ROI Extraction: A pre-trained improved YOLOv8-nano model (introducing an attention mechanism to optimize feature extraction) is used to automatically locate the tumor region in the preprocessed image. The bounding box is expanded by 10% and then cropped. Through multi-level ROI partitioning, the ROI is divided into core region, edge region, and surrounding tissue region, resulting in multiple ROI images. Features of each region are then extracted separately, preserving region-specific information, improving the extraction accuracy to 93.2% and reducing the extraction time to 0.2 seconds per image. Compared to the inefficiency and subjectivity of manual ROI extraction, YOLOv8-nano achieves an average precision (mAP) of 89.7% on the breast tumor detection task, enabling efficient and accurate ROI localization and reducing the interference of background regions on subsequent feature extraction. The signal-to-noise ratio of the preprocessed image is improved by 45%, and the edge clarity of the tumor is improved by 38%. The ROI extraction time is reduced from 5-8 minutes per image manually to 0.3 seconds per image, with an extraction accuracy of 90.5%, making it possible to process large-scale datasets.

[0031] Data normalization and calibration: Clinical data and molecular biomarker data were normalized using Z-Score, and lifestyle data were normalized using Min-Max.

[0032] The preprocessed data forms a basic dataset integrating image, clinical, molecular, lifestyle, and annotation.

[0033] Step 3. Training set construction.

[0034] Dataset partitioning: The basic dataset is divided into training set, validation set and test set in a ratio of 7:2:1 (data from each patient at two time points appear in pairs).

[0035] Data augmentation: Generative adversarial networks (GANs) are used to generate simulated early images of malignant tumors (such as microcalcifications and lesions with blurred boundaries) to expand the scarce samples; dynamic resampling is performed on the training set, and the ratio of positive and negative samples is adjusted every 10 rounds according to the model loss to alleviate the imbalance of benign and malignant samples.

[0036] Step 4. Model building.

[0037] Constructing a temporal Siamese network: Using the Siamese Siamese network as a framework, it includes two structurally identical branches, M1, with parameters synchronized in real time. A pre-trained ResNet50 is used as the backbone of each branch. After the conv5_x layer of each ResNet50 branch, a multi-scale feature fusion module and a convolutional block attention module (CBAM) are sequentially inserted, forming an improved Siamese-ResNet50, i.e., a temporal Siamese network. The two structurally identical branches enable real-time parameter synchronization, ensuring consistent feature extraction standards.

[0038] Traditional pre-trained ResNet50, when used for medical images, often fails to capture minute lesions and fine structures due to its output of only the highest layer (conv5_x) features. Therefore, this invention retains the original ResNet50 backbone architecture, freezing only the first 10 layers (preserving general low-level features such as edge and texture extraction capabilities) and fine-tuning the last 10 layers (specifically for breast ultrasound images, learning breast tissue-specific features) to adapt to breast ultrasound images. This allows the network to adapt to the specific features of breast ultrasound images, such as grayscale distribution and tissue texture. A CBAM module (Convolutional Block Attention Module) is inserted after the conv5_x layer of the original ResNet50. The CBAM module includes channel attention (enhancing key channel features through a Squeeze-Excitation mechanism) and spatial attention (locating key spatial regions through pooling and convolution). Channel attention enhances key channel features through the Squeeze-Excitation mechanism, while spatial attention locates key spatial regions through pooling and convolution; the attention weights are dynamically adjusted. The CBAM attention module focuses on changes in key tumor regions (such as edges and calcifications), suppressing invalid differences in background tissues (such as fat and glands) (e.g., slight positional shifts due to respiratory movements), thus improving the effectiveness of differential features. CBAM enhances key channel features through a Squeeze-Excitation mechanism, then uses pooling and convolution to locate key spatial regions, dynamically adjusting attention weights to accurately capture core features such as microcalcifications and blurred boundaries. This compensates for the shortcomings of traditional ResNet50 in capturing subtle features, improving the network's sensitivity to tumor edge changes by 35%, and providing more discriminative deep visual features for subsequent multimodal feature fusion. ResNet50 is pre-trained on the ImageNet dataset and possesses powerful general image feature extraction capabilities. The pre-trained ResNet50 remains the core backbone network, extracting features from the ROI image layer by layer through convolutional layers (conv1-conv5_x) and residual connections. Instead, after ResNet50 outputs the features of each layer, a multi-scale feature fusion module is added to integrate the output features of conv3_x, conv4_x, and conv5_x through channel concatenation, adaptive weighting, and other methods. This solves the problem of loss of low-dimensional details caused by ResNet50 relying only on the high-dimensional features of conv5_x. For example, tiny calcifications with a diameter of <1mm are easily diluted by high-dimensional semantic features.

[0039] Traditional Siamese networks are suitable for comparing differences in data from the same source. The improved Siamese-ResNet50 proposed in this invention leverages the feature extraction capabilities of ResNet50 to accurately capture subtle differences in temporal images, such as the increase in minute depressions at the edge of a mass. Real-time synchronous parameter updates avoid inconsistencies in feature extraction between the two branches due to parameter deviations, ensuring the accuracy of difference calculations. After introducing CBAM, the network's sensitivity to identifying changes in the mass edge is improved by 35%, the signal-to-noise ratio of temporal feature differences is improved by 50%, and the interference rate of invalid differences (such as background tissue shift) is reduced by 28%.

[0040] Loss Function Design: Traditional multi-task loss does not consider task uncertainty, which can easily lead to tasks with more samples overdoing training. Homoscedastic uncertainty can automatically quantify task difficulty. For example, calcification change tasks have few samples, uneven label distribution, and high uncertainty. Dynamically adjusting weights ensures balanced learning across tasks. Therefore, the final loss function of the branch network in this invention... The loss functions for the seven tasks are calculated using formula (1):

[0041] (1) Here, represents the loss function for the i-th classification task, and represents the uniformity of the label value distribution in the i-th classification task. Specifically, The loss function for the BI-RADS label classification task. arrive These are the loss functions for the six tumor morphological feature classification tasks. This is related to the uncertainty of the task, and is measured by entropy. It can be calculated directly using formula (2):

[0042] (2) in Let be the number of label categories in the i-th classification task. This represents the proportion of the j-th class label in the learning sample set for this classification task. Alternatively, the entropy value can be used as the initial value and then further adjusted as a network parameter.

[0043] Loss weight adjustment: If a task's Increase, then The weight of this task in the total loss is automatically increased, ensuring that tasks with scarce samples are fully learned.

[0044] Cross-entropy loss is suitable for classification tasks and can accurately measure classification error. When combined with uncertainty parameters, the loss function becomes more robust. By designing the loss function, the classification accuracy of tasks with scarce samples (such as calcification deterioration, where the sample ratio is only 8%) is improved by 31%, avoiding underfitting caused by imbalanced samples; the total loss convergence speed is accelerated by 25%, and the loss value is reduced by 40% compared with the traditional average loss after 100 training rounds.

[0045] Step 4. Model training.

[0046] Based on the designed loss function, the constructed temporal Siamese network is trained using the training set.

[0047] Training setup: The optimizer is AdamW (initial learning rate 1e-4), employing a hybrid scheduling of cosine annealing and warmup (warmup for the first 10 rounds, cosine annealing for the next 90 rounds to decay). Training is stopped if there is no improvement after 10 consecutive rounds, based on validation set AUC, accuracy, and recall, to avoid overfitting. During training, the loss for each task is monitored in real-time, and network parameters are dynamically adjusted to ensure balanced learning across tasks.

[0048] Example 2 Based on the twin network trained in Example 1, breast ultrasound images are processed, such as... Figure 1 and Figure 2 As shown, it includes the following steps: Step 1. Data collection.

[0049] Breast ultrasound images of the same patient at at least two time points were acquired, with an interval of 1-6 months. Clinical data, molecular marker data, and lifestyle data of the patient were acquired simultaneously. Image specifications and data integrity were automatically verified, and prompts were given to supplement the acquisition if key data was missing. In this embodiment, time-series ultrasound images of the same patient were acquired at time T0 and T1.

[0050] Step 2. Data preprocessing.

[0051] The process follows an intelligent adaptive workflow during training: image standardization, deep learning registration, multi-level ROI extraction, cross-modal data calibration and normalization, outputting high-quality multi-level ROI images and standardized multimodal data. The entire preprocessing is automated and requires no manual intervention.

[0052] Step 3. Multimodal feature extraction and fusion.

[0053] Step 3.1. Extract 8 morphological features and 5 acoustic features based on the preprocessed ROI image.

[0054] Eight morphological features include the tumor's aspect ratio (major axis / minor axis), perimeter, area, roundness (4π × area / perimeter²), eccentricity (major axis / minor axis of the ellipse), boundary irregularity (actual perimeter / perimeter of the smallest circumscribed rectangle), number of depressions (count of boundary depression points), and rate of change of the maximum diameter (maximum diameter in the current image / maximum diameter in the earlier image).

[0055] Five acoustic features include the internal echo homogeneity of the tumor (standard deviation of gray level within the ROI / mean gray level), posterior echo attenuation (mean gray level of the area behind the mass / mean gray level of the mass area), number of calcifications (count of pixel clusters with gray level > 220, calcifications appear as high gray level on ultrasound), average size of calcifications (average area of ​​calcification pixel clusters), and edge echo intensity (mean gray level of the 10-pixel area at the edge of the mass / mean gray level of the surrounding normal tissue).

[0056] Morphological and acoustic features correspond to the structural and ultrasound signal attributes of tumors, respectively, and their combination covers the core imaging features of tumors. Furthermore, these features are designed based on clinical experience in ultrasound diagnosis; for example, an aspect ratio > 1 is a typical indicator of malignant tumors, directly capturing key information that doctors focus on during diagnosis and compensating for the lack of clinical prior knowledge in the deep learning black box. After fusing these features with subsequent deep features, the model's recognition rate for early microcalcifications (diameter < 1 mm) improved by 27%.

[0057] Z-Score normalization is used to normalize the above features, mapping all features to the [-1,1] interval to eliminate dimensional differences. Normalization avoids the network from overemphasizing features with large values ​​due to differences in feature dimensions.

[0058] Step 3.2. Input the preprocessed multi-level ROI images at time T0 and time T1 into the temporal Siamese network. One branch inputs the earlier image (time T0), and the other branch inputs the current image (time T1, T1-T0=1-6 months).

[0059] Specifically, the pre-trained ResNet50's conv3_x, conv4_x, and conv5_x layers output features at different scales. conv3_x outputs low-dimensional detail features (such as pixel-level texture of calcifications and minute edges), conv4_x outputs mid-dimensional structural features (such as local contours of the mass and regional echo distribution), and conv5_x outputs high-dimensional semantic features (such as overall mass morphology patterns and global features related to benignity / malignancy), providing raw feature sources for subsequent fusion. The multi-scale feature fusion module directly receives these three layers of features as input, integrating features at different scales into multi-scale integrated features of detail, structure, and semantics. Then, the CBAM module applies attention weights to this fused feature, enhancing key information such as minute calcifications and edge details, ultimately outputting optimized deep visual features. Traditional ResNet50 relies solely on the high-dimensional features of conv5_x, easily losing minute details; a single CBAM can only enhance key information of a single-scale feature, failing to compensate for the limitations caused by scale differences. In this invention, the multi-scale feature fusion module solves the problem of incomplete scale, and CBAM solves the problem of insufficient key information. The two work together to achieve multi-scale coverage and focus on key information, which ultimately greatly improves the sensitivity of micro-calcification (diameter <1mm) recognition.

[0060] Two ResNet50 branches output feature vectors F0 (2048-dimensional depth visual features) at time T0 and F1 (2048-dimensional depth visual features) at time T1, respectively. Element-wise subtraction and L2 normalization are used to calculate the difference features: ΔF = L2_Norm(F1 - F0), maintaining the difference feature vector dimension at 2048. Element-wise subtraction directly quantifies the magnitude of temporal feature changes; for example, if the edge irregularity channel value in F1 is higher than that in F0, the subtraction will result in a positive value, indicating edge deterioration. L2 normalization maps the difference features to a uniform scale, avoiding fluctuations in absolute differences caused by variations in image brightness and contrast.

[0061] CBAM attention weighting is applied to ΔF to obtain temporal difference features: ΔF_att = CBAM(ΔF). This strengthens the weights of key difference features, such as increasing the number of calcification points and improving edge irregularity. Attention weighting can further filter effective differences and reduce the interference of irrelevant differences on subsequent classification, such as feature fluctuations caused by device noise.

[0062] The discriminative power of the temporal difference feature ΔF_att in this invention is 42% higher than that of the unweighted ΔF. The discriminative power of the temporal difference feature ΔF_att is calculated using cosine similarity, with similarity >0.9 for benign tumors and <0.7 for malignant tumors. The model achieves an accuracy of 93.8% in judging the morphological stability of benign tumors, reducing the possibility of misclassification as malignant due to minor noise differences.

[0063] Step 4: Multi-task feature fusion.

[0064] Step 4.1: Concatenate the 8 morphological features and 5 acoustic features extracted in Step 3.1 with the weighted temporal difference feature ΔF_att output in Step 3.2 to form a preliminary multimodal feature set. This preliminary multimodal feature set covers the static basic attributes of the tumor (such as 3D volume and inherent texture features) and multimodal correlation information (such as the weight ratio of image features and molecular markers), addressing the limitations of single image features. The temporal difference feature ΔF_att reflects the dynamic changes of the tumor, while the preliminary multimodal feature set reflects the static attributes of the tumor, such as initial morphology and acoustic features. The fusion of these two features comprehensively covers the diagnostic basis of both static and dynamic changes, aligning with the diagnostic logic of clinicians who combine historical images to judge the trend of change. Verification shows that the feature discrimination of the proposed multimodal preliminary feature set is 32% higher than that of a single deep feature; it also accelerates the convergence speed of the model on small sample data (training set < 500 image pairs) by 40%, avoiding overfitting due to the scarcity of medical data.

[0065] Step 4.2: Concatenate the preliminary multimodal feature set with the preprocessed clinical data, molecular marker data and lifestyle data to form the temporal multimodal fusion feature vector F_final.

[0066] Batch normalization of F_final accelerates subsequent network training convergence and avoids gradient explosion. Batch normalization makes the distribution of fused features more stable, reducing training fluctuations caused by high feature dimensionality.

[0067] Through verification, the classification ability of the fusion feature F_final proposed in this invention is improved by 29% compared with the single temporal difference feature; the generalization accuracy of the model on ultrasound images acquired by different devices reaches 88.6%, reducing the diagnostic bias caused by device differences.

[0068] This invention also utilizes a cross-modal attention fusion network to dynamically adjust the weights of static multimodal features and temporal difference features (e.g., for patients with high-density glands, increasing the weights of dynamic texture features and ΔF_att), making it more adaptable to complex clinical scenarios than the fixed process of inputting the former into the latter.

[0069] Step 5. Output the tumor morphological change classification results.

[0070] The temporal multimodal fusion feature vector F_final is input into the trained multi-task classification network, which outputs the tumor morphological change classification results, specifically the predicted probabilities of seven changes in ultrasound image features of breast tumors. This meets the needs of real-time clinical diagnosis and provides accurate diagnostic support for primary healthcare institutions. The multi-task classification network consists of two fully connected layers (512 and 128 neurons respectively) and nine independent Softmax classification layers. The GELU activation function is used to mitigate gradient vanishing.

[0071] Specifically, the classification head structure takes the temporal multimodal fusion feature vector F_final as input and first performs feature dimensionality reduction through two fully connected layers. The first layer has 512 neurons, and the second layer has 128 neurons, with ReLU activation function. Independent classification layers ensure that each task learns independently, avoiding interference between tasks. For example, the correlation between calcification changes and benign / malignant labels differs, and independent layers can adjust weights separately. Dimensionality reduction through fully connected layers reduces feature dimensions and computational complexity, while the ReLU activation function introduces non-linearity, enhancing feature expressiveness.

[0072] Multi-task output: Seven independent Softmax classification layers are set up, corresponding to seven classification tasks, as follows: Task 1: Changes in the shape of a mass. Task: Predict the trend of changes in the shape of a mass (0 = more regular, 1 = no change, 2 = more irregular).

[0073] Task 2: Changes in the orientation of a mass. Task: Predict changes in the positional relationship between the mass and the skin (0 = parallel to the skin, 1 = no change, 2 = not parallel to the skin).

[0074] Task 3: Changes in Block Edges: Predict changes in the smoothness of block edges (0 = smoother, 1 = no change, 2 = rougher).

[0075] Task 4: Changes in Internal Echoes. Task: Predict changes in the internal echo intensity of the mass (0 = increased, 1 = no change, 2 = decreased).

[0076] Task 5: Posterior Echo Changes Task: Predict changes in the posterior echo state of the mass (0 = increased, 1 = no change, 2 = decreased).

[0077] Task 6: Changes in Calcification: Predict the changes in the severity of calcification points (0 = improvement, 1 = no change, 2 = worsening).

[0078] Task 7: Mass Growth Rate Task: Determine the growth rate based on the maximum diameter change rate (0=slow, 1=stable, 2=fast, based on the maximum diameter change rate: <5% / month is slow, 5%-10% / month is stable, >10% / month is fast).

[0079] The seven tasks cover the core indicators of clinical diagnosis. The first six tasks correspond to key morphological changes in ultrasound images, and the seventh task adds growth rate, forming a logical chain from sub-indicators to comprehensive judgment.

[0080] The average classification accuracy of the seven tasks in this invention reaches 91.5%, and the interference rate between tasks calculated by task loss correlation is less than 15%, thus avoiding the impact of overfitting of a single task on the overall performance.

[0081] Ultimately, doctors can provide risk grading and related solutions based on the patient's basic information, T0 / T1 time-time image comparison, and prediction results from seven tasks, and offer clinical decision-making suggestions.

[0082] This invention first provides high-quality input for subsequent processing through temporal data acquisition and preprocessing, then integrates clinical priors and deep features through multimodal feature extraction, captures dynamic changes through temporal difference learning, and outputs accurate image processing results through multi-task classification. It shifts from static diagnosis to dynamic monitoring, addressing the challenge of detecting early-stage malignant tumors in clinical practice; it moves from single-image processing to multimodal fusion, reducing biases caused by missing clinical data; and it provides strong evidence for doctors' diagnoses.

[0083] Based on the same inventive concept, this invention also proposes a breast ultrasound image processing system, including a data acquisition and processing module, a feature extraction module, a feature fusion module, and a classification module.

[0084] Specifically, the data acquisition and processing module is used to acquire breast ultrasound images of the same patient on two different dates, and simultaneously acquire the patient's clinical data, molecular marker data, and lifestyle data; perform multi-level ROI division on the breast ultrasound images from different dates to obtain multi-level ROI images; and perform normalization processing on the clinical data, molecular marker data, and lifestyle data.

[0085] The feature extraction module is used to extract multiple morphological features representing tumor structural attributes and multiple acoustic features representing tumor ultrasound signal attributes based on multi-level ROI images. The multi-level ROI images are input into a temporal twin network, and the two branches of the temporal twin network process multi-level ROI images from two different dates, respectively, and output the depth visual features of the two different images.

[0086] The feature fusion module is used to subtract the depth visual features of two different images element by element, and then perform L2 normalization on the result to obtain the differential features. The differential features are then weighted and multiple morphological features and multiple acoustic features are concatenated with the weighted differential features to form a preliminary multimodal feature set. The preliminary multimodal feature set is then concatenated with normalized clinical data, molecular marker data, and lifestyle data to form a temporal multimodal fusion feature vector.

[0087] The classification module is used to input the temporal multimodal fusion feature vector into the pre-trained multi-task classification network and output the tumor morphological change classification result.

[0088] Each module in the aforementioned breast ultrasound image processing system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0089] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps in the embodiments of the breast ultrasound image processing method. Specific implementation methods can be found in the method embodiments, and will not be repeated here.

[0090] Furthermore, the present invention also provides a non-transitory computer-readable storage medium containing instructions on which a computer program is stored. For example, a memory containing instructions that can be executed by a processor of a computer device to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc. When the computer program is executed by the processor, it can implement the steps in the embodiments of the breast ultrasound image processing method. Specific implementation methods can be found in the method embodiments, which will not be repeated here.

[0091] Those skilled in the art will understand that embodiments of the present invention can provide methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0092] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0093] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0094] These computer program instructions can also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0095] It should be noted that the specific embodiments described above enable those skilled in the art to more fully understand the present invention, but do not limit the present invention in any way. Therefore, although the present invention has been described in detail in this specification and embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the present invention; and all technical solutions and improvements that do not depart from the spirit and scope of the present invention are covered within the protection scope of the present invention patent. No reference numerals in the claims should be construed as limiting the scope of the claims. Any simple variations or equivalent substitutions of technical solutions that can be readily obtained by those skilled in the art within the scope of the technology disclosed in the present invention are within the protection scope of the present invention.

Claims

1. A method for processing breast ultrasound images, characterized in that, Includes the following steps: Breast ultrasound images of the same patient from two different dates were acquired, and the patient's clinical data, molecular marker data, and lifestyle data were acquired simultaneously. Multi-level ROI division was performed on the breast ultrasound images from different dates to obtain multi-level ROI images. The clinical data, molecular marker data, and lifestyle data were then normalized. Multiple morphological features characterizing tumor structural properties and multiple acoustic features characterizing tumor ultrasound signal properties are extracted from multi-level ROI images. The multi-level ROI images are input into a temporal Siamese network. The two branches of the temporal Siamese network process the multi-level ROI images from two different dates and output the depth visual features of the two different images. The depth visual features of two different images are subtracted element-wise, and the result is L2 normalized to obtain the difference features. The difference features are then weighted, and multiple morphological features and multiple acoustic features are concatenated with the weighted difference features to form a preliminary multimodal feature set. The preliminary multimodal feature set is then concatenated with normalized clinical data, molecular marker data, and lifestyle data to form a temporal multimodal fusion feature vector. The temporal multimodal fusion feature vector is input into a pre-trained multi-task classification network, which outputs the tumor morphological change classification results.

2. The breast ultrasound image processing method according to claim 1, characterized in that, The temporal Siamese network uses the Siamese Siamese network as its framework, comprising two branch networks M1 with identical structures and real-time synchronized parameters. A pre-trained ResNet50 is used as the backbone network for each branch. A multi-scale feature fusion module and a convolutional block attention module (CBAM) are sequentially inserted after the conv5_x layer of each ResNet50 branch, forming the temporal Siamese-ResNet50 network. The branch networks process multi-level ROI images to output deep visual features, including the following steps: The pre-trained ResNet50 layers conv3_x, conv4_x, and conv5_x output low-dimensional detail features, mid-dimensional structural features, and high-dimensional semantic features, respectively. The multi-scale feature fusion module fuses the low-dimensional detail features, mid-dimensional structural features, and high-dimensional semantic features into multi-scale integrated features. Then, the CBAM module applies attention weights to the multi-scale integrated features, and finally outputs deep visual features.

3. The breast ultrasound image processing method according to claim 1, characterized in that, Before processing multi-level ROI images through a temporal Siamese network, the temporal Siamese network is trained, and the training includes the following steps: Collect breast ultrasound images from multiple patients at at least two time points, with an interval of 1-6 months; simultaneously collect the patient's clinical data, molecular marker data, and lifestyle data; All breast ultrasound images were scaled to a uniform size of 512×512 pixels using bilinear interpolation. The contrast of these uniformly sized images was then enhanced using the CLAHE algorithm. After contrast enhancement, the images were first subjected to a 5×5 Gaussian filter, followed by a 3×3 median filter. The filtered images were then divided into multi-level regions of interest (ROIs) to obtain multi-level ROI images. Clinical and molecular biomarker data were normalized using Z-Score, while lifestyle data were normalized using Min-Max. A basic dataset was constructed using the normalized clinical, molecular biomarker, and lifestyle data, along with the multi-level ROI images. The constructed temporal Siamese network is trained using a base dataset.

4. The breast ultrasound image processing method according to claim 3, characterized in that, The loss function for training a temporal Siamese network is: ; in, Let represent the loss function for the i-th classification task. denoted as the uniformity of the label value distribution in the i-th classification task.

5. The breast ultrasound image processing method according to claim 1, characterized in that, The clinical data included patient age, breast density grading, family history of breast cancer, and previous imaging reports.

6. The breast ultrasound image processing method according to claim 1, characterized in that, The morphological features include the tumor's aspect ratio, perimeter, area, roundness, eccentricity, boundary irregularity, number of depressions, and maximum diameter variation rate; the acoustic features include the tumor's internal echo homogeneity, posterior echo attenuation value, number of calcifications, average size of calcifications, and edge echo intensity.

7. The breast ultrasound image processing method according to claim 1, characterized in that, The tumor morphological change classification results include changes in the shape of the mass, changes in the direction of the mass, changes in the edge of the mass, changes in internal echoes, changes in posterior echoes, calcification, and the growth rate of the mass.

8. A breast ultrasound image processing system, characterized in that, include: The data acquisition and processing module is used to acquire breast ultrasound images of the same patient on two different dates, and simultaneously acquire the patient's clinical data, molecular marker data, and lifestyle data; perform multi-level ROI division on the breast ultrasound images from different dates to obtain multi-level ROI images; and perform normalization processing on the clinical data, molecular marker data, and lifestyle data. The feature extraction module is used to extract multiple morphological features characterizing tumor structural properties and multiple acoustic features characterizing tumor ultrasound signal properties based on multi-level ROI images. The multi-level ROI images are input into a temporal Siamese network. The two branches of the temporal Siamese network process the multi-level ROI images from two different dates and output the depth visual features of the two different images. The feature fusion module is used to subtract the depth visual features of two different images element by element, and then perform L2 normalization on the result to obtain the difference features. The differential features are weighted and then concatenated with multiple morphological and acoustic features to form a preliminary multimodal feature set. The preliminary multimodal feature set is then concatenated with normalized clinical data, molecular marker data, and lifestyle data to form a temporal multimodal fusion feature vector. The classification module is used to input the temporal multimodal fusion feature vector into the pre-trained multi-task classification network and output the tumor morphological change classification result.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is loaded by a processor, it is able to perform the steps of the method according to any one of claims 1 to 7.

Images