Multi-modal mammary gland image classification method and terminal

Abstract

The invention discloses a multi-modal breast image classification method and a terminal, and the method comprises the steps: cutting an obtained breast cancer ultrasonic image through employing a trained YOLOv8 target detection model, training a ResNet-18-based multi-modal classification model through employing an obtained breast cancer molybdenum target image and the cut breast cancer ultrasonic image, and carrying out the classification of the breast cancer. By using the YOLOv8 target detection model, the interference of background noise and artifacts in the ultrasonic image can be avoided, and the foreground and the background in the ultrasonic image can be effectively separated through a fine feature extraction and classification mechanism. And breast cancer molybdenum target images and breast cancer ultrasonic images of different modals can be well adapted and processed by utilizing a ResNet-18-based multi-modal classification model, so that image information can be propagated and fused in a network in a deeper level, the advantages of the breast cancer molybdenum target images and the breast cancer ultrasonic images are fused, and the method has the advantages of being high in accuracy and high in robustness. And the trained multi-modal classification model is utilized, so that the accuracy of breast image recognition and classification is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a multimodal breast image classification method and a terminal. Background Art

[0002] Breast cancer is one of the most common malignancies in women, and it mainly occurs in women, especially those over the age of 50. According to the 2022 Global Cancer Statistics, breast cancer has become one of the top five most diagnosed cancers in women worldwide, with 2.3089 million confirmed cases and 665,700 deaths. In many high-income countries with high breast cancer incidence, the mortality rate has declined since around the early 1990s, mainly due to numerous treatment breakthroughs and progress made in early detection through screening. Since there are few identified modifiable risk factors for the disease, the focus of breast cancer control is to increase the opportunity for early diagnosis or screening and timely and comprehensive cancer treatment. In order to detect and diagnose breast cancer early, screening is usually required through various imaging methods. Commonly used imaging methods in breast cancer screening include breast ultrasound (US), mammography (DM), breast magnetic resonance imaging (MRI) and digital breast tomosynthesis (DBT), among which mammography and breast ultrasound are the most commonly used. Radiologists need to observe the density, structure and morphology of breast tissue in the images, look for potential lumps or abnormal shadows, and judge the benign or malignant nature of the lumps by their morphology, edges and size. This places high demands on the radiologists’ professional knowledge and clinical experience. Specifically, there are many factors that affect the diagnosis of breast cancer, including overlaps and artifacts in the images, the visual perception ability of radiologists, lack of clarity, poor contrast and lack of experience. In addition, breast cancer screening covers a wide range of people and has a large population base, and full coverage of screening is a very important link. In this link, it is often necessary to observe thousands of medical images, which is not only time-consuming but also laborious. The introduction of artificial intelligence technology has brought significant improvements to breast cancer screening.

[0003] In recent years, the rapid development of artificial intelligence has provided new possibilities for medical image analysis and diagnosis, and has also played a positive role in promoting the early diagnosis and screening of breast cancer. Deep learning models can automatically learn features from large-scale medical images without manually designing complex rules. This highly automated feature makes the breast cancer screening process more efficient and reduces the burden on doctors. Compared with radiologists, deep learning models can learn complex and abstract image features, including tiny, imperceptible lesions, which helps to improve the sensitivity of early breast cancer, enabling the model to detect potential abnormalities, thereby assisting clinicians in diagnosis and reducing unnecessary biopsies. Convolutional neural networks (CNNs) in deep learning models are one of the most widely used methods in medical image processing tasks. They have the ability to process high-dimensional data and are therefore often used in breast cancer-related research. In existing studies on early detection of breast cancer based on deep learning, the imaging methods studied are mainly concentrated on mammography and ultrasound images.

[0004] There are certain limitations and deficiencies in the existing studies based on mammography and ultrasound. These studies only used a single medical image for analysis. However, the information provided by a single medical image is limited and may not fully reflect the structural information or local details of breast tissue, especially for hidden or deep lesions. In contrast, multimodal medical images can fuse different types of information, and each imaging technology provides a unique information perspective, thereby presenting more comprehensive and multidimensional information. For example, color ultrasound can detect coarse calcifications ≥0.5 mm (generally benign calcifications), but for microcalcifications <0.5 mm, color ultrasound is often insensitive. This requires further mammography. For areas where mammography is difficult to examine, such as the edge of the gland, breast color ultrasound can be used as a compensatory examination. Color ultrasound has a high resolution and can detect tiny nodules with a diameter of several millimeters. Especially when the mammography indicates a dense breast and the glands overlap, it can clearly show the contour and morphology of the lesion when the mass or nodule is difficult to distinguish. Therefore, the combination of mammography and color ultrasound can greatly improve the detection rate of breast cancer.

[0005] In the prior art, there is a multimodal classification model based on ultrasound and mammography for predicting the malignancy of patients with BI-RADS US 4A (Breast Imaging Reporting and Data System Classification 4A) lesions. Through experimental comparison, the authors demonstrated that the model has higher accuracy in predicting the malignancy of patients with BI-RADS US 4A lesions. However, the generalizability of the model is limited because the research subjects are limited to patients with BI-RADS US 4A lesions.

[0006] Therefore, how to build a multimodal breast image classification model with wide applicability to reduce the burden on doctors and improve the accuracy of breast image recognition and classification has become a problem that needs to be studied at present. Summary of the invention

[0007] The technical problem to be solved by the present invention is to provide a multimodal breast image classification method and terminal, which can improve the accuracy of breast image recognition and classification.

[0008] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0009] A multimodal breast image classification method comprises the steps of:

[0010] Acquire breast cancer mammography images and corresponding breast cancer ultrasound images;

[0011] Using the trained YOLOv8 target detection model to crop the breast cancer ultrasound image to obtain a cropped breast cancer ultrasound image;

[0012] Constructing a multimodal classification model based on ResNet-18, and training the multimodal classification model based on ResNet-18 using the breast cancer mammography image and the cropped breast cancer ultrasound image to obtain a trained multimodal classification model;

[0013] Obtain breast cancer mammography images and breast cancer ultrasound images to be classified, and use the trained multimodal classification model to identify and classify the breast cancer mammography images and breast cancer ultrasound images to obtain classification results.

[0014] In order to solve the above technical problems, another technical solution adopted by the present invention is:

[0015] A multimodal breast image classification terminal comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0016] Acquire breast cancer mammography images and corresponding breast cancer ultrasound images;

[0017] Using the trained YOLOv8 target detection model to crop the breast cancer ultrasound image to obtain a cropped breast cancer ultrasound image;

[0018] Constructing a multimodal classification model based on ResNet-18, and training the multimodal classification model based on ResNet-18 using the breast cancer mammography image and the cropped breast cancer ultrasound image to obtain a trained multimodal classification model;

[0019] Obtain breast cancer mammography images and breast cancer ultrasound images to be classified, and use the trained multimodal classification model to identify and classify the breast cancer mammography images and breast cancer ultrasound images to obtain classification results.

[0020] The beneficial effects of the present invention are as follows: using a trained YOLOv8 target detection model to crop an acquired breast cancer ultrasound image, using the acquired breast cancer mammography image and the cropped breast cancer ultrasound image to train a multimodal classification model based on ResNet-18 to obtain a trained multimodal classification model, using the YOLOv8 target detection model can be free from the interference of background noise and artifacts in the ultrasound image, and effectively separate the foreground and background in the ultrasound image through a fine feature extraction and classification mechanism, and using the multimodal classification model based on ResNet-18 can well adapt to and process breast cancer mammography images and breast cancer ultrasound images of different modalities, so that image information can be propagated and integrated at a deeper level in the network, effectively processing different types of medical images, combining breast cancer mammography images and breast cancer ultrasound images, integrating the advantages of the two images, and using the trained multimodal classification model to perform recognition and classification of breast images, thereby improving the accuracy of recognition and classification of breast images. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A flowchart of a multimodal breast image classification method according to an embodiment of the present invention;

[0022] Figure 2 A schematic diagram of the structure of a multimodal breast image classification terminal according to an embodiment of the present invention;

[0023] Figure 3 A schematic diagram of tumor region cropping of breast cancer ultrasound images in a multimodal breast image classification method according to an embodiment of the present invention;

[0024] Figure 4 It is a schematic diagram of prediction results of the YOLOv8 target detection model in the multimodal breast image classification method according to an embodiment of the present invention on different test set data;

[0025] Figure 5 A schematic diagram of multimodal classification model training in a multimodal breast image classification method according to an embodiment of the present invention;

[0026] Figure 6 A multimodal classification model framework diagram in a multimodal breast image classification method according to an embodiment of the present invention;

[0027] Figure 7 Schematic diagram of ROC curves of three models in the multimodal breast image classification method according to an embodiment of the present invention;

[0028] Figure 8 A schematic diagram of a confusion matrix of a single-modality classification model using only molybdenum target images in a multi-modality breast image classification method according to an embodiment of the present invention;

[0029] Fig. 9 A schematic diagram of a confusion matrix of a single-modality classification model using only ultrasound images in a multi-modality breast image classification method according to an embodiment of the present invention;

[0030] Fig.10 Schematic diagram of a confusion matrix of a multimodal classification model in a multimodal breast image classification method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0031] In order to explain the technical content, achieved objectives and effects of the present invention in detail, the following is an explanation in combination with the implementation modes and the accompanying drawings.

[0032] Please refer to Figure 1 , a multimodal breast image classification method, comprising the steps of:

[0033] Acquire breast cancer mammography images and corresponding breast cancer ultrasound images;

[0034] Using the trained YOLOv8 target detection model to crop the breast cancer ultrasound image to obtain a cropped breast cancer ultrasound image;

[0035] Constructing a multimodal classification model based on ResNet-18, and training the multimodal classification model based on ResNet-18 using the breast cancer mammography image and the cropped breast cancer ultrasound image to obtain a trained multimodal classification model;

[0036] Obtain breast cancer mammography images and breast cancer ultrasound images to be classified, and use the trained multimodal classification model to identify and classify the breast cancer mammography images and breast cancer ultrasound images to obtain classification results.

[0037] From the above description, it can be seen that the beneficial effects of the present invention are: using the trained YOLOv8 target detection model to crop the acquired breast cancer ultrasound image, using the acquired breast cancer mammography image and the cropped breast cancer ultrasound image to train the multimodal classification model based on ResNet-18, and obtaining the trained multimodal classification model, using the YOLOv8 target detection model can be free from the interference of background noise and artifacts in the ultrasound image, and effectively separate the foreground and background in the ultrasound image through a fine feature extraction and classification mechanism, and using the multimodal classification model based on ResNet-18 can well adapt to and process breast cancer mammography images and breast cancer ultrasound images of different modalities, so that image information can be more deeply propagated and integrated in the network, effectively processing different types of medical images, combining breast cancer mammography images and breast cancer ultrasound images, integrating the advantages of the two images, and using the trained multimodal classification model to perform recognition and classification of breast images, thereby improving the accuracy of breast image recognition and classification.

[0038] Furthermore, before acquiring the breast cancer mammographic target image and the corresponding breast cancer ultrasound image, the method further includes:

[0039] Build the YOLOv8 target detection model;

[0040] Acquire a first data set, wherein the first data set includes a first training set;

[0041] Expanding the first training set using a data enhancement method to obtain an expanded first training set;

[0042] The YOLOv8 target detection model is trained using the expanded first training set to obtain a trained YOLOv8 target detection model.

[0043] From the above description, it can be seen that using the data enhancement method to expand the first training set and using the expanded first training set to train the YOLOv8 target detection model can ensure that the model is fully trained and ensure that the YOLOv8 target detection model has a better target detection effect.

[0044] Furthermore, the method of using the trained YOLOv8 target detection model to crop the breast cancer ultrasound image to obtain the cropped breast cancer ultrasound image includes:

[0045] The trained YOLOv8 target detection model is used to crop the tumor area in the breast cancer ultrasound image to obtain a cropped breast cancer ultrasound image.

[0046] From the above description, we can see that the boundary between tumors and normal tissues in ultrasound images is sometimes not obvious, and YOLOv8's multi-scale feature detection capability enables it to extract features at different feature levels, thereby identifying tumors of different sizes and capturing subtle differences in breast cancer ultrasound images. In addition, in response to the interference of background noise and artifacts in ultrasound images, YOLOv8 can effectively separate the foreground (tumor area) and background through its sophisticated feature extraction and classification mechanism, thereby ensuring the accuracy of subsequent recognition and classification.

[0047] Furthermore, after the breast cancer ultrasound image is cropped using the trained YOLOv8 target detection model to obtain the cropped breast cancer ultrasound image, the method further includes:

[0048] Adjusting the sizes of the breast cancer mammography image and the cropped breast cancer ultrasound image to a preset size to obtain an adjusted breast cancer mammography image and an adjusted breast cancer ultrasound image;

[0049] The adjusted breast cancer mammography image and the adjusted breast cancer ultrasound image are grayscale normalized using a min-max normalization method to obtain a normalized breast cancer mammography image and a normalized breast cancer ultrasound image.

[0050] From the above description, it can be seen that the acquired breast cancer medical images often have inconsistent grayscale distribution due to the acquisition and imaging factors of different devices. The inconsistent grayscale distribution of these images will affect the training effect of the deep learning model. Therefore, the min-max normalization method is used to normalize the grayscale of the adjusted breast cancer mammography image and the adjusted breast cancer ultrasound image, so that the images have consistency in grayscale, reduce the grayscale changes between images, and facilitate the subsequent breast image classification.

[0051] Furthermore, the step of using the min-max normalization method to perform grayscale normalization on the adjusted breast cancer mammography image and the adjusted breast cancer ultrasound image to obtain the normalized breast cancer mammography image and the normalized breast cancer ultrasound image includes:

[0052]

[0053] In the formula, x norm1 represents the normalized breast cancer mammography image, x norm2 represents the normalized breast cancer ultrasound image, x1 represents the adjusted breast cancer mammography image, x2 represents the adjusted breast cancer ultrasound image, x min1 represents the minimum grayscale value in the adjusted breast cancer mammography image, x max1 represents the maximum grayscale value in the adjusted breast cancer mammography image, xmax2 represents the maximum grayscale value in the adjusted breast cancer ultrasound image, x min2 It represents the minimum grayscale value in the adjusted breast cancer ultrasound image.

[0054] From the above description, it can be seen that using the min-max normalization method for grayscale normalization is simpler and more effective.

[0055] Furthermore, after the adjusted mammographic target image of breast cancer and the adjusted ultrasound image of breast cancer are gray-scale normalized using the min-max normalization method to obtain the normalized mammographic target image of breast cancer and the normalized ultrasound image of breast cancer, the method further includes:

[0056] Performing data enhancement on the normalized breast cancer mammography image and the normalized breast cancer ultrasound image to obtain an enhanced breast cancer mammography image and an enhanced breast cancer ultrasound image;

[0057] The multimodal classification model based on ResNet-18 is trained using the breast cancer mammography image and the cropped breast cancer ultrasound image to obtain the trained multimodal classification model, which includes:

[0058] The enhanced breast cancer mammography image and the enhanced breast cancer ultrasound image are used to train the ResNet-18-based multimodal classification model to obtain a trained multimodal classification model.

[0059] From the above description, it can be seen that by performing data enhancement on the normalized mammography images of breast cancer and the normalized ultrasound images of breast cancer, the problems of data imbalance and insufficient data in mammography and ultrasound images can be well solved, thereby increasing the diversity and quantity of data, and further improving the robustness and performance of the trained model.

[0060] Furthermore, the construction of a multimodal classification model based on ResNet-18 includes:

[0061] A multimodal classification model based on ResNet-18 is constructed using a late feature fusion strategy.

[0062] From the above description, we can know that late feature fusion refers to training multiple models independently, extracting features from each modality through multiple models, and then fusing the obtained features, and finally connecting the classifiers for classification. Using the late feature fusion strategy to build a multimodal classification model based on ResNet-18 can be better applicable to the two modalities of mammography and ultrasound images with significant differences.

[0063] Furthermore, the trained multimodal classification model is used to identify and classify the breast cancer mammography images and breast cancer ultrasound images to be classified, and the classification results obtained include:

[0064] Preprocessing the breast cancer mammography image and the breast cancer ultrasound image to be classified to obtain a preprocessed breast cancer mammography image and a preprocessed breast cancer ultrasound image;

[0065] Inputting the preprocessed breast cancer mammographic target image into the first channel of the trained multimodal classification model for feature extraction, and outputting a first feature map after an average pooling operation;

[0066] Inputting the preprocessed breast cancer ultrasound image into the second channel of the trained multimodal classification model for feature extraction, and outputting a second feature map after an average pooling operation;

[0067] Adding and fusing the first feature map and the second feature map to obtain a fused feature map;

[0068] Inputting the fused feature map into the fully connected layer and softmax layer of the trained multimodal classification model in sequence for recognition and classification, and outputting the classification result;

[0069] The first channel and the second channel respectively adopt the ResNet-18 network structure.

[0070] From the above description, it can be seen that the trained multimodal classification model can be used to identify and classify breast cancer mammography images and breast cancer ultrasound images to be classified, which can accurately classify the benign and malignant tumors, reduce unnecessary biopsies for patients, and has a wider applicability and is suitable for all breast cancer patient groups.

[0071] Furthermore, it also includes:

[0072] Calculating the sensitivity index, specificity index, accuracy index and precision index of the trained multimodal classification model;

[0073] Using the sensitivity index, the specificity index, the accuracy index and the precision index as model performance evaluation indicators of the trained multimodal classification model;

[0074] The trained multimodal classification model is optimized based on the model performance evaluation index.

[0075] From the above description, it can be seen that by calculating the sensitivity index, specificity index, accuracy index and precision index of the trained multimodal classification model, these indicators can reliably reflect the model performance of the trained multimodal classification model. Using these indicators can help to subsequently optimize the model and ensure the reliability of the multimodal classification model.

[0076] Please refer to Figure 2 Another embodiment of the present invention provides a multimodal breast image classification terminal, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements each step of the above-mentioned multimodal breast image classification method when executing the computer program.

[0077] The multimodal breast image classification method and terminal of the present invention can be applied to the scene where breast images need to be classified, and are described below through specific implementation methods:

[0078] Please refer to Figure 1 , Figure 3-Figure 10 , Embodiment 1 of the present invention is:

[0079] A multimodal breast image classification method comprises the steps of:

[0080] S1. Build a YOLOv8 target detection model.

[0081] S2. Obtain a first data set, where the first data set includes a first training set.

[0082] S3. Expand the first training set using a data enhancement method to obtain an expanded first training set.

[0083] S4. Use the expanded first training set to train the YOLOv8 target detection model to obtain a trained YOLOv8 target detection model.

[0084] Wherein, the first data set also includes a first test set. In an optional implementation, the number of the first data set is 1348, the number of the first training set is 1098, the number of the first test set is 250, the expanded first training set is 4392, and in the training, the initial learning rate is set to 0.01, the pre-training model is set to yolov8n.pt, the maximum number of iterations is set to 300, the image resize is set to 640, the batch size is set to 8, the confidence threshold is set to 0.25, and the IOU threshold (a threshold for judging the degree of overlap between the predicted box and the true box) is set to 0.7.

[0085] In an optional implementation, the method further includes: using the first test set to test the trained YOLOv8 target detection model; if the test passes, the training is completed; if the test fails, the trained YOLOv8 target detection model is retrained.

[0086] The YOLOv8 target detection model trained by the present invention shows strong performance on the test set, such as Figure 4 As shown, the accuracy reached 0.913, the sensitivity reached 0.887, and the mAP50 (mean average precision when the IoU threshold is 0.5) reached 0.938.

[0087] There are several reasons why YOLOv8 can effectively detect tumor areas in breast cancer ultrasound images: the boundary between tumors and normal tissues in ultrasound images is sometimes not obvious, and YOLOv8's multi-scale feature detection capability enables it to extract features at different feature levels, thereby identifying tumors of different sizes and capturing subtle differences in ultrasound images. In addition, in response to the interference of background noise and artifacts in ultrasound images, YOLOv8 can effectively separate the foreground (tumor area) and background through its sophisticated feature extraction and classification mechanism.

[0088] S5. Acquire a mammographic target image of breast cancer and a corresponding ultrasound image of breast cancer.

[0089] For example, there are 2235 mammography images of breast cancer, of which 1832 are benign and 403 are malignant. There are 1348 breast cancer ultrasound images corresponding to the mammography images of breast cancer (i.e., from the same breast cancer patient), of which 1139 are benign and 209 are malignant.

[0090] S6. Use the trained YOLOv8 target detection model to crop the breast cancer ultrasound image to obtain a cropped breast cancer ultrasound image, such as Figure 3 shown.

[0091] Specifically, the trained YOLOv8 target detection model is used to crop the tumor area in the breast cancer ultrasound image to obtain a cropped breast cancer ultrasound image.

[0092] In breast cancer ultrasound images, the information that affects the judgment of benign and malignant tumors mainly exists in the tumor area, while the regional information outside the tumor area is irrelevant interference information, including text information, color spots, etc. Therefore, tumor area cropping is performed on breast cancer ultrasound images, which can reduce a large amount of interference information and obtain effective tumor area feature information. At the same time, this also helps to reduce the computational complexity of the deep learning model.

[0093] S7, adjusting the size of the breast cancer mammography image and the cropped breast cancer ultrasound image to a preset size, to obtain an adjusted breast cancer mammography image and an adjusted breast cancer ultrasound image, such as Figure 5 shown.

[0094] Specifically, the size of the breast cancer mammography image is adjusted to a first preset size, and the size of the cropped breast cancer ultrasound image is adjusted to a second preset size to obtain an adjusted breast cancer mammography image and an adjusted breast cancer ultrasound image.

[0095] In an optional implementation, the first preset size is 1024×1024, and the second preset size is 512×512.

[0096] S8, using a min-max normalization method to perform grayscale normalization on the adjusted breast cancer mammography image and the adjusted breast cancer ultrasound image to obtain a normalized breast cancer mammography image and a normalized breast cancer ultrasound image, such as Figure 5 As shown, specifically:

[0097]

[0098] In the formula, x norm1 represents the normalized breast cancer mammography image, x norm2 represents the normalized breast cancer ultrasound image, x1 represents the adjusted breast cancer mammography image, x2 represents the adjusted breast cancer ultrasound image, x min1 represents the minimum grayscale value in the adjusted breast cancer mammography image, x max1 represents the maximum grayscale value in the adjusted breast cancer mammography image, x max2 represents the maximum grayscale value in the adjusted breast cancer ultrasound image, x min2 It represents the minimum grayscale value in the adjusted breast cancer ultrasound image.

[0099] Since breast images obtained from hospitals often have inconsistent grayscale distribution due to factors such as acquisition and imaging by different devices, the inconsistent grayscale distribution of these images will affect the training effect of the deep learning model. Therefore, it is necessary to perform the above-mentioned grayscale normalization operation on breast cancer mammography images and breast cancer ultrasound images to make the images consistent in grayscale and reduce the grayscale changes between images.

[0100] S9, performing data enhancement on the normalized breast cancer mammography image and the normalized breast cancer ultrasound image to obtain an enhanced breast cancer mammography image and an enhanced breast cancer ultrasound image, such as Figure 5 shown.

[0101] In an optional implementation, the data enhancement includes random horizontal flipping, random vertical flipping, elastic deformation, and random horizontal and vertical flipping.

[0102] In order to solve the problems of data imbalance and insufficient data in mammography and ultrasound images, the above-mentioned data enhancement method is used to expand the number of malignant samples, thereby increasing the diversity and quantity of data by transforming and expanding existing samples, and improving the robustness and performance of the model.

[0103] S7, constructing a multimodal classification model based on ResNet-18, and using the breast cancer mammography image and the cropped breast cancer ultrasound image to train the multimodal classification model based on ResNet-18 to obtain a trained multimodal classification model, specifically including S71-S72:

[0104] S71. Use the late feature fusion strategy to build a multimodal classification model based on ResNet-18.

[0105] Due to the different data sources of multiple modalities, early feature fusion strategies will face the problem of difficult splicing, and directly splicing the original data may lead to larger feature dimensions and high requirements for data preprocessing. Late fusion refers to the independent training of multiple models, and each modality is subjected to feature extraction by multiple models respectively. The obtained features are then fused and finally connected to the classifier for classification. The independent networks of each modality are allowed to extract specific features first without interfering with each other's feature extraction process. In this way, the network of each modality can focus on its unique features, thereby maximizing the use of their respective advantages. It is more suitable for two modalities with significant differences (i.e., mammography images of breast cancer and ultrasound images of breast cancer).

[0106] S72, using the enhanced breast cancer mammography image and the enhanced breast cancer ultrasound image to train the ResNet-18 based multimodal classification model to obtain a trained multimodal classification model, such as Figure 5 shown.

[0107] In an optional implementation, during the training, the Adam optimizer is used to update the model parameters, the initial learning rate is set to 0.0001, the batch size is set to 16, and the maximum iteration step is set to 200.

[0108] S8, obtaining breast cancer mammography images and breast cancer ultrasound images to be classified, and using the trained multimodal classification model to identify and classify the breast cancer mammography images and breast cancer ultrasound images to be classified to obtain classification results, such as Figure 6 As shown, specifically including S81-S86:

[0109] S81. Obtain a breast cancer mammography image and a breast cancer ultrasound image to be classified.

[0110] S82: preprocess the breast cancer mammography image and breast cancer ultrasound image to be classified to obtain a preprocessed breast cancer mammography image and a preprocessed breast cancer ultrasound image.

[0111] The preprocessing includes cropping, resizing and grayscale normalization of the breast cancer ultrasound image. The specific processing method is the same as the above and will not be repeated here.

[0112] S83, inputting the preprocessed breast cancer mammography image into the first channel of the trained multimodal classification model for feature extraction, and outputting a first feature map after an average pooling operation.

[0113] S84, inputting the preprocessed breast cancer ultrasound image into the second channel of the trained multimodal classification model for feature extraction, and outputting a second feature map after an average pooling operation.

[0114] Among them, S83 and S84 are executed synchronously, and the first channel and the second channel respectively adopt the ResNet-18 network structure.

[0115] S85. Add and fuse the first feature map and the second feature map to obtain a fused feature map.

[0116] S86. Input the fused feature map into the fully connected layer and softmax layer of the trained multimodal classification model in sequence for recognition and classification, and output the classification result.

[0117] The classification result is to identify whether each input image is benign or malignant.

[0118] In an optional implementation, it also includes:

[0119] S9. Calculate the sensitivity index, specificity index, accuracy index and precision index of the trained multimodal classification model.

[0120] Specifically, the number of samples TP predicted by the trained multimodal classification model as malignant but actually being malignant, the number of samples FP predicted as malignant but actually being benign, the number of samples TN predicted as benign but actually being benign, and the number FN predicted as benign but actually being malignant are obtained; based on the number of samples that are malignant but actually being malignant, the number of samples predicted as malignant but actually being benign, the number of samples predicted as benign but actually being benign, and the number of samples predicted as benign but actually being malignant, the sensitivity index, specificity index, accuracy index, and precision index of the trained multimodal classification model are calculated.

[0121] Wherein, the sensitivity index sensitivity is:

[0122]

[0123] The specificity index is:

[0124]

[0125] The accuracy index accuracy is:

[0126]

[0127] The precision index precision is:

[0128]

[0129] S10. Using the sensitivity index, the specificity index, the accuracy index and the precision index as model performance evaluation indicators of the trained multimodal classification model.

[0130] S11. Optimizing the trained multimodal classification model based on the model performance evaluation index.

[0131] The present invention experimentally compares six common classification networks, ResNet-18, ResNet-50, ResNext-50, Inception v3, VGG16, and GoogleNet, and then chooses ResNet-18 to build a multimodal classification model, as follows:

[0132] For the benign and malignant classification of mammography images, the experiment used a total of 2235 mammography datasets. The training set was data enhanced to make the benign and malignant ratio in the training set as balanced as possible. The enhanced training set was 2738 (1446 benign and 1292 malignant), and the test set was 446 (336 benign and 80 malignant). Each model was pre-trained using the ImageNet natural image dataset before training with the mammography dataset. The experimental results are shown in Table 1. It can be seen that ResNet-18 showed the highest sensitivity (92.50%), accuracy (91.03%) and AUC (0.954), while GoolgeNet achieved the highest specificity (94.81%) and precision (73.24%). From the results, ResNet-18 performed better than other models on the mammography image dataset. This also shows that the scale and complexity of the mammography image dataset do not require a particularly deep network structure to learn features, and the relatively shallow network structure of ResNet-18 is more suitable for mammography images, avoiding the risk of overfitting. In addition, the resolution and details of mammography images determine the distribution of features, and the shallower network ResNet-18 is more suitable for such image features and can better capture key features.

[0133] Table 1 Performance of different models on mammography images

[0134]

[0135] For the benign and malignant classification of ultrasound images, the experiment used a total of 1,348 ultrasound datasets. The malignant images in the training set were also enhanced to balance the benign and malignant ratios in the training set. The enhanced training set consisted of 1,584 images (912 benign and 672 malignant) and the test set consisted of 268 images (227 benign and 41 malignant). Similarly, each model was pre-trained using the ImageNet natural image dataset before being trained using the ultrasound dataset. The experimental results are shown in Table 2. ResNet-18 achieved the highest sensitivity (92.68%), specificity (92.07%), accuracy (92.16%), and AUC (0.956), while GoogleNet achieved the highest precision (68.47%). From the results, the performance of the ResNet-18 network on the ultrasound image dataset is also better than that of other networks. Ultrasound images usually contain a lot of noise and artifacts, which reduces the image quality. The presence of noise will have an adverse effect on deep and complex models and may cause model overfitting. ResNet-18 is less complex and it is easier to avoid overfitting when training on small-scale datasets.

[0136] Table 2 Performance of different models on ultrasound images

[0137]

[0138] In order to analyze the performance of the multimodal classification model, the multimodal classification model was compared with the single-modal classification model (using only ultrasound images and only mammography images). The experiment used the data of 205 patients in the test cohort as a unified test set. The ROC curves and confusion matrices of the three models were plotted, as shown in Figure 2. Figure 7 , Figure 8 , Fig. 9 and Fig.10 As shown in Table 3, the test results show that although the sensitivity of the multimodal classification model on the test set is lower than that of the single-modal classification model, its AUC, specificity, precision and accuracy on the test set are higher than those of the single-modal classification model, among which AUC reaches 0.969, accuracy is 93.78%, specificity is 96.91%, and accuracy is 83.56%. The high AUC and high accuracy of the multimodal classification model indicate that the multimodal classification model is better than the single-modal classification model in the classification of benign and malignant tumors. The high specificity of the multimodal classification model indicates that the multimodal classification model is better than the single-modal classification model in the recognition of benign patients during the screening process, and the high accuracy of the multimodal classification model indicates that the false positive situation of the multimodal classification model during the screening process is much lower than that of the single-modal classification model, which can reduce unnecessary biopsies for patients. The multimodal classification model of the present invention fully utilizes the information characteristics of the two modalities of molybdenum target images and ultrasound images, improves the classification performance of the model, and the model is suitable for all breast cancer patient groups.

[0139] Table 3 Performance of three models on the test set

[0140]

[0141] Please refer to Figure 2 , Embodiment 2 of the present invention is:

[0142] A multimodal breast image classification terminal comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, each step of the multimodal breast image classification method in the first embodiment is implemented.

[0143] In summary, the present invention provides a multimodal breast image classification method and terminal, which uses a trained YOLOv8 target detection model to crop the acquired breast cancer ultrasound image, and uses the acquired breast cancer mammography image and the cropped breast cancer ultrasound image to train a multimodal classification model based on ResNet-18 to obtain a trained multimodal classification model. The YOLOv8 target detection model can be free from the interference of background noise and artifacts in the ultrasound image, and can effectively separate the foreground and background in the ultrasound image through a fine feature extraction and classification mechanism. The multimodal classification model based on ResNet-18 can well adapt to and process breast cancer mammography images and breast cancer ultrasound images of different modalities, so that image information can be used on the Internet. The proposed method can effectively process different types of medical images by deeper propagation and fusion in the network, combining mammary cancer mammography images and mammary cancer ultrasound images, integrating the advantages of the two images, and using the trained multimodal classification model to identify and classify breast images, thereby improving the accuracy of breast image recognition and classification; in addition, the min-max normalization method is used to normalize the grayscale of the adjusted mammary cancer mammography images and the adjusted mammary cancer ultrasound images, so that the images have consistency in grayscale, reducing the grayscale changes between images, which is conducive to the subsequent breast image classification; at the same time, the multimodal classification model based on ResNet-18 constructed using the late feature fusion strategy can be better applicable to the two modalities of mammary cancer images and ultrasound images with significant differences.

[0144] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent transformations made using the contents of the present invention's specification and drawings, or directly or indirectly applied in related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A multimodal breast image classification method, characterized in that: Includes steps: Acquire breast cancer mammography images and corresponding breast cancer ultrasound images; Using the trained YOLOv8 target detection model to crop the breast cancer ultrasound image to obtain a cropped breast cancer ultrasound image; Constructing a multimodal classification model based on ResNet-18, and training the multimodal classification model based on ResNet-18 using the breast cancer mammography image and the cropped breast cancer ultrasound image to obtain a trained multimodal classification model; Obtain breast cancer mammography images and breast cancer ultrasound images to be classified, and use the trained multimodal classification model to identify and classify the breast cancer mammography images and breast cancer ultrasound images to obtain classification results.

2. A multimodal breast image classification method according to claim 1, characterized in that: Before acquiring the breast cancer mammographic target image and the corresponding breast cancer ultrasonic image, the method further includes: Build the YOLOv8 target detection model; Acquire a first data set, wherein the first data set includes a first training set; Expanding the first training set using a data enhancement method to obtain an expanded first training set; The YOLOv8 target detection model is trained using the expanded first training set to obtain a trained YOLOv8 target detection model.

3. A multimodal breast image classification method according to claim 1, characterized in that: The step of using the trained YOLOv8 target detection model to crop the breast cancer ultrasound image to obtain the cropped breast cancer ultrasound image comprises: The trained YOLOv8 target detection model is used to crop the tumor area in the breast cancer ultrasound image to obtain a cropped breast cancer ultrasound image.

4. A multimodal breast image classification method according to claim 1, characterized in that: After the trained YOLOv8 target detection model is used to crop the breast cancer ultrasound image to obtain the cropped breast cancer ultrasound image, the method further includes: Adjusting the sizes of the breast cancer mammography image and the cropped breast cancer ultrasound image to a preset size to obtain an adjusted breast cancer mammography image and an adjusted breast cancer ultrasound image; The adjusted breast cancer mammography image and the adjusted breast cancer ultrasound image are grayscale normalized using a min-max normalization method to obtain a normalized breast cancer mammography image and a normalized breast cancer ultrasound image.

5. A multimodal breast image classification method according to claim 4, characterized in that: The method of using the min-max normalization method to perform grayscale normalization on the adjusted breast cancer mammography image and the adjusted breast cancer ultrasound image to obtain the normalized breast cancer mammography image and the normalized breast cancer ultrasound image includes: In the formula, x norm1 represents the normalized breast cancer mammography image, x norm2 represents the normalized breast cancer ultrasound image, x1 represents the adjusted breast cancer mammography image, x2 represents the adjusted breast cancer ultrasound image, x min1 represents the minimum grayscale value in the adjusted breast cancer mammography image, x max1 represents the maximum grayscale value in the adjusted breast cancer mammography image, x max2 represents the maximum grayscale value in the adjusted breast cancer ultrasound image, x min2 It represents the minimum grayscale value in the adjusted breast cancer ultrasound image.

6. A multimodal breast image classification method according to claim 4, characterized in that: After the adjusted mammographic target image of breast cancer and the adjusted ultrasound image of breast cancer are gray-scale normalized by using the min-max normalization method to obtain the normalized mammographic target image of breast cancer and the normalized ultrasound image of breast cancer, the method further includes: Performing data enhancement on the normalized breast cancer mammography image and the normalized breast cancer ultrasound image to obtain an enhanced breast cancer mammography image and an enhanced breast cancer ultrasound image; The multimodal classification model based on ResNet-18 is trained using the breast cancer mammography image and the cropped breast cancer ultrasound image to obtain the trained multimodal classification model, which includes: The enhanced breast cancer mammography image and the enhanced breast cancer ultrasound image are used to train the ResNet-18-based multimodal classification model to obtain a trained multimodal classification model.

7. A multimodal breast image classification method according to claim 1, characterized in that: The construction of a multimodal classification model based on ResNet-18 includes: A multimodal classification model based on ResNet-18 is constructed using a late feature fusion strategy.

8. A multimodal breast image classification method according to claim 1, characterized in that: The trained multimodal classification model is used to identify and classify the breast cancer mammography images and breast cancer ultrasound images to be classified, and the classification results obtained include: Preprocessing the breast cancer mammography image and the breast cancer ultrasound image to be classified to obtain a preprocessed breast cancer mammography image and a preprocessed breast cancer ultrasound image; Inputting the preprocessed breast cancer mammographic target image into the first channel of the trained multimodal classification model for feature extraction, and outputting a first feature map after an average pooling operation; Inputting the preprocessed breast cancer ultrasound image into the second channel of the trained multimodal classification model for feature extraction, and outputting a second feature map after an average pooling operation; Adding and fusing the first feature map and the second feature map to obtain a fused feature map; Inputting the fused feature map into the fully connected layer and softmax layer of the trained multimodal classification model in sequence for recognition and classification, and outputting the classification result; The first channel and the second channel respectively adopt the ResNet-18 network structure.

9. A multimodal breast image classification method according to claim 1, characterized in that: Also includes: Calculating the sensitivity index, specificity index, accuracy index and precision index of the trained multimodal classification model; Using the sensitivity index, the specificity index, the accuracy index and the precision index as model performance evaluation indicators of the trained multimodal classification model; The trained multimodal classification model is optimized based on the model performance evaluation index.

10. A multimodal breast image classification terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, each step in the multimodal breast image classification method described in any one of claims 1 to 9 is implemented.

Images