Breast cancer ultrasound image classification method based on multimodal deep learning radiomics

Abstract

The present invention provides a breast cancer ultrasound image classification method based on multimodal deep learning radiomics, involving deep neural networks, multimodal deep learning, data analysis, and artificial intelligence technologies. Through deep learning and radiomics technologies, a large number of features are extracted from multimodal data such as images, videos, and text, and a nomogram is constructed based on this multimodal information, thereby realizing the function of identifying and classifying ultrasound images of benign and malignant breast tumors. The present invention combines multimodal images with clinical features of deep learning radiomics to construct a nomogram model that intuitively identifies and classifies images. This model supports visualization of the model's prediction process and results, provides visual and easily interpretable prediction results, and addresses problems existing in the prior art. It improves the accuracy of identifying and classifying ultrasound images of benign and malignant breast tumors, intuitively reflects the nature of the images, and is of great significance and value for taking personalized measures.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning technology, and specifically relates to a breast cancer ultrasound image classification method based on multimodal deep learning imaging omics. Background Art

[0002] Traditional breast examinations are performed using ultrasound. However, due to the low resolution and signal-to-noise ratio of ultrasound images, and the reliance on the expertise of radiologists, accurate image recognition of the progression of diseased areas is difficult. Currently, the core approach to precision medicine lies in comprehensive and complete analysis of a patient's genomic information. However, in clinical practice, precision treatment for breast cancer remains difficult to implement on a large scale due to numerous challenges, including technical limitations, data privacy, and cost. In medicine, radiomics is an advanced technique for extracting medical image features. It automatically extracts features (such as first-order statistical features, second-order texture features, or wavelet features) from regions of interest (ROIs) in ultrasound images, then selects quantitative features to identify and classify cancer images. While these features can quantify information difficult to perceive visually, such as internal texture patterns or tissue distribution, radiomics relies on tumor boundaries marked by radiologists, making it insufficient for accurately identifying and classifying breast tumors as benign or malignant in ultrasound images.

[0003] In recent years, with the advancement of artificial intelligence, deep learning algorithms have demonstrated outstanding performance in image, video, and text recognition tasks, garnering significant attention. For example, deep convolutional neural networks (CNNs), by extracting hierarchical features from data through stacked sampling layers and applying backpropagation to update model parameters, can automatically learn richer representations from images. However, the generalization ability of AI-based classification models alone remains questionable. This is primarily due to the fact that during the manual classification of tumor ultrasound images, physicians not only observe various ultrasound images and videos but also refer to various clinical data, such as tumor morphology and size, patient age, and various pathological parameters. Therefore, combining features extracted from radiomics and deep learning techniques with clinical data for multimodal feature analysis and inference has become a new technology in medical radiomics combined with AI analysis. Currently, this technology can improve intelligence, feature extraction results, and the accuracy of predictive models while maintaining a constant sample size. Unfortunately, while each feature has various dependencies with the data, the sheer number of features is so abstract that neither researchers nor physicians can fully grasp their meaning.

[0004] As a visually intuitive model, the nomogram is widely used in medical image analysis. For biochemical researchers and clinical diagnosticians, the nomogram converts the probability of each modality's prediction into a score, allowing researchers to intuitively see the contribution of the artificial intelligence (AI) black box to the final classification results. Therefore, for more accurate predictions and more convenient real-world clinical applications, a multimodal nomogram classification model is needed to provide decision support and assistance. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a breast cancer ultrasound image classification method based on multimodal deep learning imaging omics, which is used to identify and classify breast tumor ultrasound images according to benign and malignant.

[0006] The technical solution adopted by the present invention to solve the above technical problems is: a breast cancer ultrasound image classification method based on multimodal deep learning imaging omics, comprising the following steps:

[0007] S0: Establish a breast cancer ultrasound image classification system based on multimodal deep learning radiomics, including a deep learning module, a radiomics module, a clinical data module, and a nomogram construction module;

[0008] The deep learning module includes a dual-input deep learning binary classification network, which is used to simultaneously extract, fuse, and abstract the deep learning features of grayscale ultrasound images and color ultrasound images to obtain deep learning scores;

[0009] The radiomics module is used to extract radiomics features from the region of interest (ROI) of grayscale ultrasound images, filter features through dimensionality reduction, and calculate the radiomics score using the logistic regression formula;

[0010] The clinical data module is used to analyze patients' clinical data, build multivariate logistic regression models, and screen for significant clinical features;

[0011] The nomogram construction module is used to normalize the outputs of the above three modules, screen out the predictive factors related to the classification results of benign and malignant tumor ultrasound images, and construct a nomogram;

[0012] S1: The deep learning module inputs the grayscale ultrasound image of the tumor into the grayscale branch of the dual-input deep learning binary classification network, and inputs the color ultrasound image of the tumor into the color ultrasound branch. After extracting and fusing features, it obtains deep learning features.

[0013] S2: The radiomics module extracts ROI imaging features from grayscale ultrasound images and performs feature dimensionality reduction to obtain radiomics features;

[0014] S3: The clinical data module analyzes and reduces the dimensionality of the patient's clinical data features, screening out several clinical features that are highly correlated with the benign or malignant nature of the tumor;

[0015] S4: The nomogram construction module uses the obtained multimodal features to perform factor fusion and construct a nomogram. Finally, the nomogram is used to obtain the scores and corresponding recognition probabilities of the benign and malignant tumor ultrasound image classification.

[0016] According to the above scheme, in step S1, the specific steps are:

[0017] S11: Apply the channel attention mechanism to the color ultrasound branch to enhance the color information in color ultrasound;

[0018] S12: Use convolutional neural networks to extract image features; use the attention mechanism of modality fusion to fuse the features of grayscale ultrasound images into color ultrasound images as the features of color ultrasound images;

[0019] S13: Use a feature fuser to fuse and classify the features of grayscale ultrasound images and color ultrasound images; use the patient's clinical data to build a deep learning model and evaluate the performance to obtain a deep learning score.

[0020] Furthermore, in step S1, the convolutional neural network used is a deep learning classification network architecture including RestNet50 or EfficientNet; the feature fuser used includes LSTM or Transformer.

[0021] According to the above scheme, in step S1, the probability value of deep learning or the unactivated output value of the last layer of the deep learning model is converted into a deep learning score, which is used to abstract multiple features into one feature value; or the deep learning score is obtained from the features of the last layer of the deep learning model through a logistic regression machine learning formula; the range of the deep learning score is [-10, 10].

[0022] According to the above scheme, in step S2, the specific steps are:

[0023] S21: Manually outline the ROI region of the grayscale ultrasound image based on the tumor boundary provided by the physician, and use radiomics tools to extract the imaging features of the ROI region;

[0024] S22: The extracted imaging features were screened using the maximum correlation minimum redundancy (mRMR) and Lasso algorithms in turn;

[0025] S23: Use the logistic regression formula to calculate the imaging features, let X i is the radiomics feature, W iis the feature weight coefficient, b is the offset coefficient, and n is the number of features after screening. The radiomics score is:

[0026]

[0027] Furthermore, in step S2, the radiomics score is the result of the regression formula, or the result of the regression formula after the activation function; the range of the radiomics score is [-1, 2].

[0028] Furthermore, in step S2, the activation function includes Sigmoid, Tanh or Relu.

[0029] According to the above scheme, in step S3, the specific steps are:

[0030] S31: Analyze various clinical indicators of patients, including age, tumor size, and pathological data;

[0031] S32: There are no significant differences between the screened cohorts, but there are significant differences between the benign and malignant groups. This makes the screened features highly correlated with the benign and malignant nature of the tumor.

[0032] Furthermore, in step S3, the pathological data includes estrogen receptor and Ki-67.

[0033] According to the above scheme, in step S4, the specific steps are:

[0034] S41: All modal data were normalized, and features with P < 0.05 were first selected by univariate logistic regression screening;

[0035] S42: Use these features through multivariable logistic regression;

[0036] S43: Select significant features with P < 0.05 as predictors for identifying and classifying benign and malignant tumor ultrasound images to construct a nomogram and obtain the total label score and the corresponding recognition probability.

[0037] The beneficial effects of the present invention are:

[0038] 1. The present invention's breast cancer ultrasound image classification method based on multimodal deep learning radiomics involves deep neural networks, multimodal deep learning, data analysis, and artificial intelligence technologies. Through deep learning and radiomics technologies, a large number of features are extracted from multimodal data such as images, videos, and text, and a nomogram is constructed based on this multimodal information, realizing the function of identifying and classifying ultrasound images of benign and malignant breast tumors.

[0039] 2. The present invention constructs a nomogram by combining multimodal images and clinical information to provide visual and easily interpretable prediction results. It aims to improve the accuracy of breast tumor ultrasound image recognition and classification, intuitively reflect the benign or malignant nature of the patient, and has great significance and value for providing individualized prognostic assessment.

[0040] 3. This invention combines deep learning imaging genomics clinical characteristics and other multimodal data to construct a nomogram model for intuitively identifying and classifying breast tumor ultrasound images, supports the visualization of the model prediction process and results, and solves the problems existing in the existing technology.

[0041] 4. The present invention performed correlation analysis on all features and screened a large number of features. For deep learning networks, the model probability was considered to represent the prediction results of the multimodal model. The ability of the proposed model to identify and classify breast tumor ultrasound images was evaluated by using values ​​such as the area under the receiver operating characteristic curve (AUC) and accuracy (ACC). BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 is a flow chart of an embodiment of the present invention.

[0043] Figure 2 2 is a schematic diagram of a dual-input deep learning model according to an embodiment of the present invention.

[0044] Figure 3 It is a typical multimodal deep learning imaging nomogram of an embodiment of the present invention. DETAILED DESCRIPTION

[0045] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0046] See also Figure 1 , the embodiment of the present invention takes breast cancer ultrasound images as the object and builds a model including a deep learning module, an imaging omics module, a clinical data module and a nomogram construction module;

[0047] The deep learning module designs a dual-input deep learning binary classification network to simultaneously extract deep learning features from breast grayscale ultrasound images and breast color ultrasound images. After feature fusion, the resulting features are abstracted into a single feature value, defined as the "deep learning score."

[0048] The radiomics module is used to extract radiomics features from the region of interest (ROI) of breast grayscale ultrasound images, then filter the features using multiple dimensionality reduction methods and calculate the "radiomic score" using a logistic regression formula;

[0049] The clinical data module is used to analyze various clinical data of patients (including age, breast tumor size, pathological data, etc.) and construct a multivariate logistic regression model to screen for significant "clinical characteristics";

[0050] The nomogram construction module is used to normalize the outputs of the above three modules, screen out the predictive factors related to the benign and malignant classification results, and construct a nomogram.

[0051] The breast cancer ultrasound image classification method based on multimodal deep learning radiomics according to an embodiment of the present invention includes the following steps:

[0052] First, design a dual-input end-to-end deep learning classification model, see Figure 2 , using two ultrasound image modalities as input. A channel-wise attention mechanism is applied to the color ultrasound branch to enhance color information in the color ultrasound image. A convolutional neural network (such as RestNet50 or EfficientNet) is then used to extract features from the images. Furthermore, to better mimic the clinical scenario of physicians interpreting color ultrasound images based on corresponding grayscale ultrasound images, a modality fusion attention mechanism is employed to fuse grayscale ultrasound features into color ultrasound images as color ultrasound image features. Finally, the grayscale and color ultrasound image features are combined using a feature fusion model (such as an LSTM or Transformer) for classification. Modeling and performance evaluation are performed using clinical data, and the best performing models and their weights are used for subsequent predictions. To abstract a large number of features into a single feature value, the "deep learning score" can be derived from deep learning probability values ​​or the unactivated output value of the last layer of the deep learning model. Alternatively, it can be obtained from the features of the last layer of the model using a machine learning formula such as logistic regression. This "deep learning score," ranging from -10 to 10, will be used as a predictor of the nomogram in subsequent research.

[0053] Second, the ROI region of interest (ROI) of the ultrasound image was manually delineated based on the physician's provided breast tumor boundaries. Radiomics tools were then used to automatically extract imaging features from the ROI region. The extracted features were then filtered using the maximum relevance minimum redundancy (mRMR) algorithm and the Lasso algorithm. Finally, the imaging features were calculated using a logistic regression formula. The calculation formula is as follows:

[0054]

[0055] Among them, X i is the radiomics feature, W iis the feature weight coefficient, b is the offset coefficient, and n is the number of features after filtering. The "radiomics score" can be the result of a regression formula or the result of a regression formula after an activation function (such as Sigmoid, Tanh, ReLU, etc.), and the score ranges from -1 to 2.

[0056] Third, we analyzed various clinical indicators, primarily age, breast tumor size, and pathological data (e.g., estrogen receptor, Ki-67). We screened for features that were not significantly different between cohorts but were significantly different between the benign and malignant groups. These features were highly correlated with benign and malignant breast cancer and were therefore more conducive to subsequent research.

[0057] Finally, the obtained multimodal features were used to perform factor fusion and construct a nomogram. All modal data were normalized, and features with a P < 0.05 were first selected through univariate logistic regression. These features were then used in multivariate logistic regression to select significant features with a P < 0.05 as predictors for identifying and classifying benign and malignant breast ultrasound images to construct a nomogram.

[0058] The final effect of the nomogram is as follows Figure 3 As shown in the figure, the nomogram shows the scores of each characteristic factor, ultimately resulting in the patient's total label score and the corresponding recognition probability of the breast tumor ultrasound image for the benign or malignant risk. The recognition probability (the risk in the last row of the figure) is calculated based on the nomogram. If the recognition probability is greater than or equal to 0.5, the input image is classified as malignant; if the recognition probability is less than 0.5, the input image is classified as benign.

[0059] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0060] The above embodiments are intended only to illustrate the design concepts and features of the present invention. Their purpose is to enable those skilled in the art to understand the contents of the present invention and implement them accordingly. The scope of protection of the present invention is not limited to the above embodiments. Therefore, any equivalent changes or modifications made based on the principles and design concepts disclosed in the present invention are within the scope of protection of the present invention.

Claims

1. A breast cancer ultrasound image classification method based on multimodal deep learning radiomics, characterized by: The following steps are involved: S1: The deep learning module inputs the grayscale ultrasound image of the tumor into the grayscale branch of the dual-input deep learning binary classification network, and inputs the color ultrasound image of the tumor into the color ultrasound branch. After extracting and fusing features, it obtains deep learning features. The specific steps are: S11: Apply the channel attention mechanism to the color ultrasound branch to enhance the color information in color ultrasound; S12: Use convolutional neural networks to extract image features; use the attention mechanism of modality fusion to fuse the features of grayscale ultrasound images into color ultrasound images as the features of color ultrasound images; S13: Use a feature fuser to fuse and classify features from grayscale ultrasound images and color ultrasound images; use the patient's clinical data to build a deep learning model and evaluate its performance to obtain a deep learning score; S2: The radiomics module extracts the ROI imaging features of the grayscale ultrasound image and performs feature dimensionality reduction to obtain radiomics features. The specific steps are as follows: S21: Manually outline the ROI region of the grayscale ultrasound image based on the tumor boundary provided by the physician, and use radiomics tools to extract the imaging features of the ROI region; S22: The extracted imaging features were screened using the maximum correlation minimum redundancy (mRMR) and Lasso algorithms in turn; S23: Use the logistic regression formula to calculate the imaging features. X i It is the radiomics feature, W i is the feature weight coefficient, b is the offset coefficient, n is the number of features after screening, and the radiomics score is obtained score for: ； S3: The clinical data module analyzes and reduces the dimensionality of the patient's clinical data features, screening out several clinical features that are highly correlated with the benign or malignant nature of the tumor. The specific steps are as follows: S31: Analyze various clinical indicators of patients, including age, tumor size, and pathological data; S32: Screen out features that have no significant differences between cohorts but are significantly different between the benign and malignant groups, making the screened features highly correlated with the benign and malignant nature of the tumor; S4: The nomogram construction module uses the obtained multimodal features to perform factor fusion and construct a nomogram. Finally, the nomogram is used to obtain the scores and corresponding recognition probabilities of benign and malignant tumor ultrasound image classification. The specific steps are as follows: S41: All modal data were normalized, and features with P < 0.05 were first selected by univariate logistic regression screening; S42: Use these features through multivariable logistic regression; S43: Select significant features with P < 0.05 as predictors for identifying and classifying benign and malignant tumor ultrasound images, construct a nomogram, and obtain the total label score and the corresponding recognition probability.

2. The breast cancer ultrasound image classification method based on multimodal deep learning radiomics according to claim 1, characterized in that: In the step S1, The convolutional neural network used is a deep learning classification network architecture including RestNet50 or EfficientNet; The feature fusers used include LSTM or Transformer.

3. The breast cancer ultrasound image classification method based on multimodal deep learning radiomics according to claim 1, characterized in that: In step S1, the probability value of deep learning or the unactivated output value of the last layer of the deep learning model is converted into a deep learning score for abstracting multiple features into one feature value; or the deep learning score is obtained from the features of the last layer of the deep learning model through a logistic regression machine learning formula; the range of the deep learning score is [-10, 10].

4. The breast cancer ultrasound image classification method based on multimodal deep learning radiomics according to claim 1, characterized in that: In step S2, the radiomics score is the result of the regression formula, or the result of the regression formula after the activation function; the range of the radiomics score is [-1, 2].

5. The breast cancer ultrasound image classification method based on multimodal deep learning radiomics according to claim 4, characterized in that: In step S2, the activation function includes Sigmoid, Tanh or Relu.

6. The breast cancer ultrasound image classification method based on multimodal deep learning radiomics according to claim 1, characterized in that: In step S3, the pathological data includes estrogen receptor and Ki-67.

Images