Breast cancer detection method based on multi-modal fusion and three-stage prediction

Abstract

The invention relates to the field of medical image analysis and computer-aided diagnosis, in particular to a breast cancer detection method based on multi-modal fusion and three-stage prediction, which comprises the following steps: acquiring breast medical image data to be processed and structured clinical data of a patient; processing the to-be-processed breast medical image data and the structured clinical data of the patient by adopting a pre-trained breast cancer detection model based on multi-modal fusion and three-stage prediction to obtain a breast cancer detection result; according to the method, a multivariate diagnosis strategy is adopted to process the visual features and the structured clinical data of the breast medical image, the ResNet-34 network based on adaptive optimization is adopted to evaluate the image shielding degree, and the efficient and light-weight OfficientV2 network is introduced to perform feature extraction, so that the problems of high calculation cost and single diagnosis strategy in the existing breast cancer detection method are solved, and the detection accuracy of the breast cancer is improved. And the method provided by the invention is not easily influenced by tissue shielding.

Description

Technical Field

[0001] The present invention relates to the fields of medical image analysis and computer-aided diagnosis, and in particular to a breast cancer detection method based on multimodal fusion and three-stage prediction. Background Art

[0002] Breast cancer screening is a method of detecting breast cancer early in asymptomatic individuals through medical examinations, aiming to reduce mortality and improve cure rates. Breast cancer screening primarily relies on mammogram analysis, combined with artificial intelligence to improve the accuracy of early detection.

[0003] Among existing technologies, computer-aided diagnosis (CAD) is widely used in breast cancer diagnosis. This method, based on image processing, pattern recognition, and machine learning, first preprocesses the input medical images, including image enhancement and noise reduction, to improve image quality. Feature extraction algorithms are then used to extract key features from the images, such as the shape, size, and density of the lesion. Finally, a classifier or decision model is used to compare and analyze the extracted features with existing medical knowledge and case data, thereby determining the nature and extent of the lesion and providing diagnostic advice to the physician. However, CAD-assisted diagnosis suffers from a single strategy, limitations, high computational cost, and susceptibility to tissue obstruction. Furthermore, it cannot integrate actual images with clinical data for comprehensive evaluation of breast cancer screening.

[0004] Therefore, there is an urgent need for a breast cancer detection method with lower computational cost, more diverse diagnostic strategies, and less susceptible to tissue shielding. Summary of the Invention

[0005] In view of this, the present invention discloses a breast cancer detection method based on multimodal fusion and three-stage prediction to solve the above problems;

[0006] A breast cancer detection method based on multimodal fusion and three-stage prediction includes: obtaining breast medical image data to be processed and structured clinical data of a patient; using a pre-trained breast cancer detection model based on multimodal fusion and three-stage prediction to process the breast medical image data to be processed and the structured clinical data of the patient to obtain a breast cancer detection result;

[0007] Furthermore, the breast cancer detection model based on multimodal fusion and three-stage prediction processes data in the following ways:

[0008] S1. Preprocessing the breast medical image data to be processed and the patient's structured clinical data to obtain enhanced breast images and structured clinical data vectors;

[0009] S2. Use the improved EfficientV2 neural network model to perform symptom recognition on enhanced breast images, obtain the probability density heat map of the lesion area, and calculate the tumor risk score based on the probability density heat map;

[0010] S3, using the ResNet-34 network based on adaptive optimization to predict the occlusion of the enhanced breast image and obtain the occlusion degree score;

[0011] S4. Using a time-series multimodal deep prediction model, we concatenate the tumor risk score, the masking score, and the structured clinical data vector to obtain historical screening records. We then model and predict these historical screening records based on the Transformer architecture to obtain the predicted risk probability.

[0012] S5. Generate breast cancer detection results based on the predicted risk probability.

[0013] The beneficial effects of the present invention include:

[0014] By adopting a multi-dimensional diagnostic strategy to process the visual features of breast medical images and structured clinical data, the limitations of a single data source can be overcome.

[0015] The adaptively optimized ResNet-34 network is used to evaluate the degree of image obscuration. By performing obscuration prediction, robustness is improved and the impact of interference factors on diagnosis is reduced, making the method proposed in this invention less susceptible to tissue obscuration.

[0016] By introducing the efficient and lightweight EfficientV2 network for feature extraction and adaptively optimizing the ResNet-34 network to achieve occlusion degree assessment, combined with the spatial attention mechanism and local density area perception module, efficient feature modeling and resource optimization are achieved in the feature extraction and occlusion prediction process, reducing computational costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 Schematic diagram of the process of data processing by the breast cancer detection model based on multimodal fusion and three-stage prediction in the present invention;

[0018] Figure 2 Schematic diagram of iterative pre-training in an embodiment of the present invention;

[0019] Figure 3 Schematic diagram of the process of processing the CBIS dataset by the breast cancer detection method based on multimodal fusion and three-stage prediction in an embodiment of the present invention;

[0020] Figure 4 This is an ROC curve graph obtained by testing a breast cancer detection model based on multimodal fusion and three-stage prediction in an embodiment of the present invention;

[0021] Figure 5 This is a graph showing the detection rate results obtained by testing clinical data using a breast cancer detection model based on multimodal fusion and three-stage prediction in an embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to make the objectives, technical solutions, features and advantages of the present invention more clearly understood, the present invention is further described below with reference to the accompanying drawings and embodiments.

[0023] This embodiment includes a breast cancer detection method based on multimodal fusion and three-stage prediction, comprising: obtaining breast medical image data to be processed and structured clinical data of a patient; using a pre-trained breast cancer detection model based on multimodal fusion and three-stage prediction to process the breast medical image data to be processed and the structured clinical data of the patient to obtain a breast cancer detection result.

[0024] Specifically, if Figure 1 As shown in the figure, the breast cancer detection model based on multimodal fusion and three-stage prediction processes the data in the following ways:

[0025] S1. Preprocess the breast medical image data to be processed and the patient's structured clinical data to obtain enhanced breast images and structured clinical data vectors.

[0026] Specifically, preprocessing includes: unifying the size and grayscale normalization. Structured clinical data includes: patient age, family medical history, hormone levels, and breast density grade.

[0027] Assume that the breast medical image is Where H and W are the height and width of the image respectively. The image is uniformly adjusted to 1024×1024 size and grayscale normalized. The formula is:

[0028]

[0029] in, represents the enhanced breast image, μ I , σ I Represent the mean and standard deviation of image pixels respectively. Grayscale normalization is used to enhance image contrast and reduce the impact of illumination changes.

[0030] Structured clinical data vector Each element c i Each represents a standardized clinical variable, such as age, breast density grade, and estrogen level. Categorical variables are one-hot encoded, and numerical variables are normalized to the interval [0,1].

[0031] S2. Use the improved EfficientV2 neural network model to perform symptom recognition on enhanced breast images, obtain the probability density heat map of the lesion area, and calculate the tumor risk score based on the probability density heat map.

[0032] The method designed in the present invention introduces an attention mechanism into the traditional EfficientV2 neural network to improve sensitivity to the mass area and adopts a lightweight design to maintain computational efficiency.

[0033] Specifically, generating a tumor risk score includes:

[0034] Step 1: Use the standard EfficientV2 neural network to extract preliminary features from the enhanced breast image. The formula is:

[0035]

[0036] Among them, F s Represents preliminary features, and EfficientV2() represents the standard EfficientV2 neural network.

[0037] Step 2: Use the SE attention mechanism and recalibrate the preliminary features using channel weighting to obtain the attention recalibrated features. The formula is:

[0038] s c =σ(W2·ReLU(W1·GAP(F s )))

[0039]

[0040] Among them, s c represents SE attention weight, σ represents Sigmoid activation function, ReLU() represents ReLU activation function, W2 and W1 represent two different weight matrices, GAP() represents average pooling, represents the attention recalibration feature.

[0041] Step 3: Use two-dimensional convolution to calculate the probability density heat map based on the attention recalibration features, and calculate the tumor risk score based on the probability density heat map. The formula is:

[0042]

[0043] Among them, S tumor represents the tumor risk score, ranging from [0, 1], H represents the probability density heat map of the lesion area, and H i,j represents the i,jth element in H, w and h represent the number of columns and rows in H respectively.

[0044] S3. Use the ResNet-34 network based on adaptive optimization to predict the occlusion of the enhanced breast image and obtain the occlusion degree score.

[0045] Specifically, since the masking risk is dominated by breast density, in order to simulate the degree of masking in the image area, the present invention adopts a ResNet-34 network based on adaptive optimization. The input of the ResNet-34 network based on adaptive optimization is a standard breast image. The network structure introduces a spatial attention mechanism and a local density region perception mechanism to determine the potential masking area in the image caused by breast density. The output is a quantitative score of the masking degree, which is used as the input for subsequent multimodal risk prediction. The masking degree score is obtained by:

[0046] Step 1: Use the standard ResNet-34 network to extract basic features from the enhanced breast image. The formula is:

[0047]

[0048] Among them, F m Represents basic features, and ResNet34() represents the standard ResNet-34 network.

[0049] Step 2: Calculate the spatial attention mask based on the basic features, associate the spatial attention mask with the basic features, and obtain the spatial attention feature. The formula is:

[0050]

[0051] Among them, M s represents the spatial attention mask, Conv 7×7 () represents a 7×7 convolutional layer, AvgPool() represents average pooling, and MaxPool() represents maximum pooling. Represents spatial attention features.

[0052] Step 3: Flatten the spatial attention features and process them with a fully connected layer to obtain the occlusion score:

[0053]

[0054] Among them, S mask Represents the masking degree score. The higher the score, the greater the masking risk of the lesion area. Flatten() represents the flattening operation. W m represents the weight matrix of the fully connected layer, b m Represents the bias vector of the fully connected layer.

[0055] S4. Use a time-series multimodal deep prediction model to concatenate the tumor risk score, masking degree score, and structured clinical data vector to obtain historical screening records. Model and predict the historical screening records based on the Transformer architecture to obtain the predicted risk probability.

[0056] Specifically, the temporal multimodal deep prediction model uses a Transformer architecture to capture the temporal association between historical screening records and the tumor risk score and masking score of the current screening. The temporal multimodal deep prediction model can adaptively weight features and generate a final risk probability as the basis for screening decisions. The predicted risk probability includes:

[0057] Step 1: Combine the tumor risk score, masking score, and structured clinical data to construct a historical screening record. The formula for the historical screening record is:

[0058] X=[S tumor ,S mask ,c1,...,c k ]

[0059] Among them, X represents the historical screening record, S tumor represents the tumor risk score, S mask represents the shading degree score, c1,...,c k Represents structured clinical data.

[0060] Step 2: Use Transformer to encode the historical screening records to obtain a time series input sequence. The formula is:

[0061] Z t =TransformerEncoder(X)

[0062]

[0063] Among them, Z t Represents the tth time unit in the temporal input sequence, TransformerEncoder() represents the Transformer encoder, Z represents the temporal input sequence, and T represents the number of time units in the temporal input sequence.

[0064] Step 3: Calculate the cancer risk probability based on the time series input sequence. The formula is:

[0065] P cancer =σ(W r ·Z+b r )

[0066] Among them, P cancer represents the risk of cancer, W rrepresents the weight matrix, b r Represents the bias vector.

[0067] S5. Generate breast cancer detection results based on the predicted risk probability.

[0068] Specifically, a cancer probability threshold is set. When the cancer risk probability is greater than or equal to the threshold, the test result is considered positive for breast cancer. When the cancer risk probability is less than the threshold, the test result is considered negative for breast cancer. The cancer probability threshold is preferably 0.5.

[0069] Furthermore, pre-training utilizes a multi-task joint optimization strategy, including lesion identification loss (cross entropy), occlusion prediction loss (mean squared error), and risk prediction loss (binary cross entropy). Multi-scale image enhancement, dropout, and L2 regularization are introduced during pre-training to prevent overfitting, and the AdamW optimizer and cosine annealing strategy are used for optimization.

[0070] Specifically, the total loss function used in pre-training is The formula is:

[0071]

[0072] in, They represent lesion identification loss, occlusion prediction loss, and risk prediction loss respectively. λ1, λ2, and λ3 represent the weight coefficients of the corresponding loss functions, and their preferred values are 1, 0.5, and 1, respectively.

[0073] Furthermore, the lesion recognition loss function The formula is:

[0074]

[0075] Here, y represents the true lesion label of the current sample, y = 1 represents the actual presence of a breast lesion, and y = 0 represents the actual absence of a breast lesion, that is, a normal state.

[0076] Furthermore, the occlusion prediction loss function The formula is:

[0077]

[0078] Among them, m * The true value of the degree of obscuration comes from the breast density grade marked by the doctor.

[0079] Furthermore, the risk prediction loss function The formula is:

[0080]

[0081] like Figure 2 As shown in the figure, the breast cancer detection model designed by the present invention based on multimodal fusion and three-stage prediction adopts the loss function Perform iterative pre-training to obtain the optimal parameters. After the pre-training is completed, the optimal parameters are set as the initial parameters of the model to complete the pre-training.

[0082] like Figure 3 As shown, the experimental data of this embodiment selected the public breast X-ray image datasets INbreast and CBIS-DDS, and combined them with the real screening clinical dataset (including review records and clinical labels), divided the samples into training set and test set in an 8:2 ratio, and used AUC, accuracy (ACC), sensitivity (SEN) and specificity (SPE) as evaluation indicators.

[0083] Specifically, if Figure 4 The figure shows the receiver operating characteristic (ROC) curve of the breast cancer detection model based on multimodal fusion and three-stage prediction. The ROC curve is the receiver operating characteristic (ROC) curve, which is used to evaluate the performance of binary classification models. As can be seen from the figure, the breast cancer detection model based on multimodal fusion and three-stage prediction proposed in this invention, corresponding to the purple curve, has an accuracy area under the curve (AUC) of 0.72, performing the best among all compared models. The overall ROC curve is significantly better than the traditional breast density assessment method.

[0084] Figure 4 The comparison methods include: a prediction model based on percent breast density (AUC = 0.51), an age-adjusted percent density prediction model (AUC = 0.60), a prediction model based on dense area (AUC = 0.53), and an age-adjusted dense area prediction model (AUC = 0.61). It can be seen that the AUCs of traditional breast density-related models are all lower than 0.62. This is because traditional methods rely solely on breast density features and are unable to effectively identify potential high-risk individuals. The detection model proposed in this paper, by integrating breast imaging features, masking assessment scores, and structured clinical data, can maintain a low false positive rate (FPR < 0.2) while still maintaining high sensitivity, showing significantly better screening performance than traditional methods. The above experimental results further verify the effectiveness of the present invention based on multimodal fusion and three-stage prediction strategy, which can improve the detection rate of early breast cancer screening, reduce the risk of missed detection, and help optimize the personalized screening process.

[0085] like Figure 5 As shown, the breast cancer detection method proposed in this paper based on multimodal fusion and three-stage prediction can significantly improve the detection rate of breast cancer in clinical application. Specifically, among 559 subjects who underwent supplemental MRI screening, a total of 36 cases of breast cancer were detected, with an overall cancer detection rate of 64.4 cases per 1000 MRI examinations, with a 95% confidence interval (CI) of 46.8–88.1, significantly outperforming traditional screening methods. Further combining the detection results of different breast density levels of BI-RADS 3–5, in the BI-RADS 5 group (highly suspected malignancy), 12 breast cancers were detected in 14 examinations, with a cancer detection rate of 85.7%, and positive predictive values PPV1 and PPV3 were both 85.7% (95% CI: 53.5–96.9); in the BI-RADS 4 group (suspicious malignancy), 17 breast cancers were detected in 27 examinations, with a detection rate of 63.0% (95% CI: 42.8–79.4), PPV1 of 63.0%, and PPV3 of 65.4% (95% CI: 44.7–81.5); and in the BI-RADS 3 group (possibly benign), 7 cancers were detected in 54 examinations, with a detection rate of 13.0% (95% CI: 42.8–79.4). CI: 6.2–25.1), PPV1 was 13.0%, and PPV3 was 22.6% (95% CI: 10.8–41.2). At the same time, among the 464 cases in which breast cancer was not detected, only 35 cases underwent benign biopsy, and 24 cases could not be finally confirmed due to incomplete MRI follow-up, indicating that the present invention effectively controls the benign biopsy rate while improving the detection rate, avoiding excessive screening and excessive intervention. Overall, the present invention accurately screens out truly high-risk individuals through intelligent multimodal fusion analysis, guides supplementary MRI screening decisions, and greatly improves the early detection rate of breast cancer, especially in the BI-RADS 4–5 high-risk group. The effect is particularly prominent, reflecting the significant advantages of the present invention in optimizing screening paths, reducing missed detection rates, and improving screening efficiency, and has important clinical application value and promotion prospects.

[0086] Finally, it should be noted that the above only describes some embodiments of the present invention. For those skilled in the art, it is conceivable that various changes, modifications, substitutions and deformations can be made to these embodiments without departing from the principles and spirit of the present invention. The scope of protection of the present invention is defined by the attached claims and their equivalents, and the above-mentioned actions should be covered within the scope of protection of the present invention.

Claims

1. A breast cancer detection method based on multimodal fusion and three-stage prediction, characterized in that: include: Acquire breast medical image data to be processed and structured clinical data of the patient; A pre-trained breast cancer detection model based on multimodal fusion and three-stage prediction is used to process the pending breast medical image data and the patient's structured clinical data to obtain breast cancer detection results. The breast cancer detection model based on multimodal fusion and three-stage prediction processes data in the following ways: S1. Preprocessing the breast medical image data to be processed and the patient's structured clinical data to obtain enhanced breast images and structured clinical data vectors; S2. Use the improved EfficientV2 neural network model to perform symptom recognition on enhanced breast images, obtain the probability density heat map of the lesion area, and calculate the tumor risk score based on the probability density heat map; S3, using the ResNet-34 network based on adaptive optimization to predict the occlusion of the enhanced breast image and obtain the occlusion degree score; S4. Using a time-series multimodal deep prediction model, we concatenate the tumor risk score, the masking score, and the structured clinical data vector to obtain historical screening records. We then model and predict these historical screening records based on the Transformer architecture to obtain the predicted risk probability. S5. Generate breast cancer detection results based on the predicted risk probability.

2. The breast cancer detection method based on multimodal fusion and three-stage prediction according to claim 1, characterized in that: The preprocessing of the breast medical image data to be processed includes: adjusting the size of the breast medical image to 1024×1024, and performing grayscale normalization processing to obtain an enhanced breast image.

3. The breast cancer detection method based on multimodal fusion and three-stage prediction according to claim 1, characterized in that: Obtaining a probability density heat map of the lesion area and calculating the tumor risk score based on the probability density heat map includes: Step 1: Use the standard EfficientV2 neural network to extract preliminary features from the enhanced breast image; Step 2: Use the SE attention mechanism and channel weighting to recalibrate the preliminary features to obtain the attention recalibrated features; Step 3: Use two-dimensional convolution to calculate the probability density heat map based on the attention recalibration features, and calculate the tumor risk score based on the probability density heat map.

4. The breast cancer detection method based on multimodal fusion and three-stage prediction according to claim 3, characterized in that: The formula used to recalibrate the preliminary features is: s c =σ(W2·ReLU(W1·GAP(F s ))) Among them, s c represents SE attention weight, σ represents Sigmoid activation function, ReLU() represents ReLU activation function, W2 and W1 represent two different weight matrices, GAP() represents average pooling, represents the attention recalibration feature.

5. The breast cancer detection method based on multimodal fusion and three-stage prediction according to claim 3, characterized in that: The formula used to calculate the tumor risk score is: Among them, H represents the probability density heat map, H i,j Represents the i,jth element in H, Conv2D 1×1 Indicates a two-dimensional convolution with a convolution kernel of 1×1. represents the attention recalibration feature, S tumor represents the tumor risk score, w and h represent the number of columns and rows of H, respectively.

6. The breast cancer detection method based on multimodal fusion and three-stage prediction according to claim 1, characterized in that: Obscuration scores include: Step 1: Use the standard ResNet-34 network to extract basic features from the enhanced breast image; Step 2: Calculate the spatial attention mask based on the basic features, associate the spatial attention mask with the basic features to obtain the spatial attention features; Step 3: Flatten the spatial attention features and process them with a fully connected layer to obtain an occlusion score.

7. The breast cancer detection method based on multimodal fusion and three-stage prediction according to claim 6, characterized in that: The formula used to obtain the spatial attention feature is: Among them, M s represents the spatial attention mask, σ() represents the Sigmoid activation function, F m Represents basic features, Conv 7×7 () represents a 7×7 convolutional layer, AvgPool() represents average pooling, and MaxPool() represents maximum pooling. Represents spatial attention features.

8. The breast cancer detection method based on multimodal fusion and three-stage prediction according to claim 1, characterized in that: The predicted risk probability includes: Step 1: Combine the tumor risk score, masking score, and structured clinical data to construct a historical screening record. Step 2: Use Transformer to encode the historical screening records to obtain a time series input sequence; Step 3: Calculate the cancer risk probability based on the time series input sequence.

9. The breast cancer detection method based on multimodal fusion and three-stage prediction according to claim 1, characterized in that: Pre-training adopts a multi-task joint optimization strategy, which includes: lesion recognition loss, occlusion prediction loss and risk prediction loss. The loss function formula used is: in, represents the loss function, represents the lesion recognition loss function, represents the occlusion prediction loss function, represents the risk prediction loss function, λ1, λ2, λ3 represent the weight coefficients of lesion recognition loss, masking prediction loss, and risk prediction loss, with values of 1, 0.5, and 1, respectively. S tumor represents the tumor risk score, S mask Indicates the degree of obstruction score, P cancer represents the predicted risk probability, y represents the actual lesion label of the current sample, y = 1 represents the actual presence of breast lesions, y = 0 represents the actual absence of breast lesions, m * The true value representing the degree of occlusion.

Images