Multi-modal deep learning fusion model construction method for breast cancer
By constructing a multimodal deep learning fusion model that combines WSI and CN features, the problems of high cost and low accuracy in existing technologies are solved, achieving low-cost and high-precision breast cancer risk stratification, which is suitable for primary hospitals.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CANCER INST & HOSPITAL CHINESE ACADEMY OF MEDICAL SCI
- Filing Date
- 2025-10-17
- Publication Date
- 2026-07-07
AI Technical Summary
Existing deep learning models based on pathological images suffer from limited feature diversity and crude fusion strategies in breast cancer diagnosis, resulting in high costs and low accuracy, making them difficult to popularize in resource-scarce areas.
A multimodal deep learning fusion model is constructed, combining WSI and CN features. It employs a direct concatenation network, a single-branch attention fusion network, and a multi-branch customized fusion network to dynamically optimize feature contribution weights and fuse WSI global features with CN features.
It achieves low-cost, high-precision breast cancer risk stratification, reduces testing costs, is suitable for primary hospitals, and improves the accuracy and reliability of breast cancer diagnosis.
Smart Images
Figure CN121439245B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of medical technology, and in particular relates to a method for constructing a multimodal deep learning fusion model for breast cancer. Background Technology
[0002] Lumened breast cancer accounts for 60%-70% of breast cancer cases, and the recurrence rate of early-stage patients is as high as 10%-15% 5 years after surgery. Although the EndoPredict gene test can accurately stratify (97% 10-year disease-free survival rate in the low-risk group vs. 76% in the high-risk group), it relies on RNA extraction and qPCR technology, and the cost of a single test exceeds $3,000, making it difficult to popularize in resource-scarce areas.
[0003] Existing deep learning models based on pathological images (such as CLAM and ResNet) have two major drawbacks:
[0004] Feature singularity: It only utilizes global features of WSI, ignoring the complementary value of nuclear morphology (such as eccentricity and tumor cell ratio) and clinical parameters.
[0005] The fusion strategy is crude: directly splicing multimodal features leads to dimensional imbalance and redundant information that overwhelms key discrimination signals, resulting in low accuracy. Summary of the Invention
[0006] In view of the problems existing in the prior art, the present invention provides a method for constructing a multimodal deep learning fusion model for breast cancer, which at least partially solves the problems of high cost and low accuracy in the prior art.
[0007] This disclosure provides a method for constructing a multimodal deep learning fusion model for breast cancer, including:
[0008] Obtain the dataset;
[0009] Preprocess the data in the dataset;
[0010] Constructing a multimodal deep learning fusion model, the construction of the multimodal deep learning fusion model includes direct splicing network structure, single-branch attention fusion network structure and multi-branch customized fusion network structure;
[0011] The constructed multimodal deep learning fusion model is trained based on the preprocessed dataset to obtain the trained multimodal deep learning fusion model.
[0012] Optionally, the direct splicing network structure is used to directly splice WSI features and CN features;
[0013] The single-branch attention fusion network structure is used to introduce a dynamic weighting attention mechanism after splicing;
[0014] The multi-branch customized fusion network structure is used to independently design attention branches for each category and then splice them together after copying CN features.
[0015] Optionally, the direct splicing network structure, used to directly splice WSI features and CN features, includes:
[0016] WSI feature extraction uses a pre-trained ResNet50 as the backbone network, freezes the first 4 layers of ResNet50, and replaces the last layer of ResNet50 with a 512-dimensional fully connected layer.
[0017] Output image patch features, and aggregate the extracted WSI features into WSI global features through attention pooling;
[0018] Feature concatenation involves concatenating the WSI global features with the CN features along the feature dimension to obtain the fused features.
[0019] The classification layer fuses features through a fully connected layer, and outputs the risk probability via Softmax.
[0020] Optionally, the single-branch attention fusion network structure is used to introduce a dynamic weighting attention mechanism after splicing, including:
[0021] Feature concatenation yields fused features;
[0022] The first fully connected layer is used to activate the attention mechanism, and the second fully connected layer is used to output the attention weights. After the weights are normalized, they are weighted and summed with the fused features to obtain the weighted features.
[0023] After the weighted features are classified by the third fully connected layer, the risk probability is output through Softmax.
[0024] Optionally, the multi-branch customized fusion network structure, used to independently design attention branches for each category and concatenate them after copying CN features, includes:
[0025] The CN feature is copied into two branches, each branch is concatenated with the category-specific WSI feature. The first branch is the high-risk weight WSI feature and the first CN feature; the second branch is the low-risk weight WSI feature and the second CN feature. Each branch uses a gated attention network independently.
[0026] Optionally, the gated attention network structure includes element-wise multiplication to obtain gate weights and weighted aggregation to form branch features.
[0027] Optionally, the preprocessing of the data in the dataset includes:
[0028] WSI preprocessing, nuclear feature extraction, and CN feature optimization.
[0029] Optionally, the extraction of cell nuclear features includes:
[0030] Output cell kernel masks and type labels using a pre-trained HoverNet model;
[0031] Fifteen morphological features and one type feature were extracted from each cell nucleus, and then aggregated into 16-dimensional cell nucleus features through sample-level statistical aggregation.
[0032] Optionally, the step of using the pre-trained HoverNet model to output cell nucleus masks and type labels includes: using the NP branch of the pre-trained HoverNet model to identify cell nucleus pixels, using the N branch of the pre-trained HoverNet model to classify cell types, using the HoVer branch of the pre-trained HoverNet model to predict centroid distance, and separating overlapping cell nuclei.
[0033] Optionally, after the step of training the constructed multimodal deep learning fusion model based on the preprocessed dataset to obtain the trained multimodal deep learning fusion model, the method further includes:
[0034] The trained multimodal deep learning fusion model is evaluated, and the evaluation metrics include the area under the ROC curve and confidence interval, accuracy, and calibration curve.
[0035] The present invention provides a method for constructing a multimodal deep learning fusion model for breast cancer. By designing a direct splicing network structure, computational efficiency is improved. A single-branch attention fusion network structure improves risk discrimination ability, and a multi-branch customized fusion network structure improves classification accuracy. The fusion of the single-branch attention fusion network structure and the multi-branch customized fusion network structure can dynamically optimize feature contribution weights. The combination of the three strategies of direct splicing network structure, single-branch attention fusion network structure and multi-branch customized fusion network structure achieves risk classification. Compared with existing RNA extraction and qPCR technologies, it reduces costs, thereby achieving the goal of low cost and high accuracy. Attached Figure Description
[0036] The above and other objects, features and advantages of this disclosure will become more apparent from the accompanying drawings, in which like reference numerals generally denote like parts.
[0037] Figure 1 A flowchart illustrating the method for constructing a multimodal deep learning fusion model for breast cancer provided in this embodiment of the disclosure;
[0038] Figure 2The ROC curve of MCLAM_sb with AUC=0.8181 provided for embodiments of this disclosure;
[0039] Figure 3 Survival curves of the DFS difference validation cohort for HR+ / HER patients provided in embodiments of this disclosure.
[0040] Figure 4 This is a schematic block diagram of an electronic device provided in an embodiment of the present disclosure. Detailed Implementation
[0041] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.
[0042] It should be understood that the following specific examples illustrate the implementation of this disclosure, and those skilled in the art can easily understand other advantages and effects of this disclosure from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. This disclosure can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this disclosure. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.
[0043] It should be noted that various aspects of embodiments within the scope of the appended claims are described below. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this disclosure, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using structures and / or functionalities other than one or more of the aspects set forth herein.
[0044] It should also be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this disclosure. The illustrations only show the components related to this disclosure and are not drawn according to the number, shape and size of the components in actual implementation. In actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.
[0045] Furthermore, specific details are provided in the following description to facilitate a thorough understanding of the examples. However, those skilled in the art will understand that the described aspects can be practiced without these specific details.
[0046] This embodiment discloses a method for constructing a multimodal deep learning fusion model for breast cancer, including:
[0047] Obtain the dataset;
[0048] Preprocess the data in the dataset;
[0049] Constructing a multimodal deep learning fusion model, the construction of the multimodal deep learning fusion model includes direct splicing network structure, single-branch attention fusion network structure and multi-branch customized fusion network structure;
[0050] The constructed multimodal deep learning fusion model is trained based on the preprocessed dataset to obtain the trained multimodal deep learning fusion model.
[0051] Optionally, the direct splicing network structure is used to directly splice WSI features and CN features;
[0052] The single-branch attention fusion network structure is used to introduce a dynamic weighting attention mechanism after splicing;
[0053] The multi-branch customized fusion network structure is used to independently design attention branches for each category and then splice them together after copying CN features.
[0054] Optionally, the direct splicing network structure, used to directly splice WSI features and CN features, includes:
[0055] WSI feature extraction uses a pre-trained ResNet50 as the backbone network, freezes the first 4 layers of ResNet50, and replaces the last layer of ResNet50 with a 512-dimensional fully connected layer.
[0056] Output image patch features, and aggregate the extracted WSI features into WSI global features through attention pooling;
[0057] Feature concatenation involves concatenating the WSI global features with the CN features along the feature dimension to obtain the fused features.
[0058] The classification layer fuses features through a fully connected layer, and outputs the risk probability via Softmax.
[0059] Optionally, the single-branch attention fusion network structure is used to introduce a dynamic weighting attention mechanism after splicing, including:
[0060] Feature concatenation yields fused features;
[0061] The first fully connected layer is used to activate the attention mechanism, and the second fully connected layer is used to output the attention weights. After the weights are normalized, they are weighted and summed with the fused features to obtain the weighted features.
[0062] After the weighted features are classified by the third fully connected layer, the risk probability is output through Softmax.
[0063] Optionally, the multi-branch customized fusion network structure, used to independently design attention branches for each category and concatenate them after copying CN features, includes:
[0064] The CN feature is copied into two branches, each branch is concatenated with the category-specific WSI feature. The first branch is the high-risk weight WSI feature and the first CN feature; the second branch is the low-risk weight WSI feature and the second CN feature. Each branch uses a gated attention network independently.
[0065] Optionally, the gated attention network structure includes element-wise multiplication to obtain gate weights and weighted aggregation to form branch features.
[0066] Optionally, the preprocessing of the data in the dataset includes:
[0067] WSI preprocessing, nuclear feature extraction, and CN feature optimization.
[0068] Optionally, the extraction of cell nuclear features includes:
[0069] Use a pre-trained HoverNet model to output cell kernel masks and type labels;
[0070] Fifteen morphological features and one type feature were extracted from each cell nucleus, and then aggregated into 16-dimensional cell nucleus features through sample-level statistical aggregation.
[0071] Optionally, the step of using the pre-trained HoverNet model to output cell nucleus masks and type labels includes: using the NP branch of the pre-trained HoverNet model to identify cell nucleus pixels, using the N branch of the pre-trained HoverNet model to classify cell types, using the HoVer branch of the pre-trained HoverNet model to predict centroid distance, and separating overlapping cell nuclei.
[0072] Optionally, after the step of training the constructed multimodal deep learning fusion model based on the preprocessed dataset to obtain the trained multimodal deep learning fusion model, the method further includes:
[0073] The trained multimodal deep learning fusion model is evaluated, and the evaluation metrics include the area under the ROC curve and confidence interval, accuracy, and calibration curve.
[0074] In a specific application scenario, such as Figure 1 As shown, the specific method for constructing a multimodal deep learning fusion model for breast cancer is as follows:
[0075] I. Data Collection:
[0076] We collected data from 254 patients with early-stage HR+ / HER2 breast cancer, including clinical information (age, ER / PR expression, tumor size, lymph node status, etc.), HE-stained sections, and EndoPredict results (EP score and risk grouping).
[0077] To validate the model, two independent samples were collected: (1) 65 HR+ / HER2- patients who did not receive chemotherapy after surgery; and (2) 273 patients with low HER2 expression, both of whom had DFS (disease-free survival) and dDFS (distant disease-free survival) follow-up data.
[0078] II. Data Preprocessing:
[0079] (1) WSI preprocessing:
[0080] Color normalization: The Macenko algorithm is used to eliminate batch differences in staining. The specific steps are as follows: ① Convert the RGB image to optical density (OD) space (OD = -log10(I), where I is the pixel value); ② Estimate the H&E staining matrix through singular value decomposition (SVD) and extract the vectors of hematoxylin (blue) and eosin (red); ③ Project the OD values onto the staining matrix and standardize the concentration to a preset reference range (mean 0.1, standard deviation 0.05); ④ Convert back to RGB space and output the normalized image.
[0081] Tissue region localization: The OTSU thresholding method is used to segment the foreground (tissue) and background, remove blank areas, and retain effective tissue regions.
[0082] Image patch extraction: The WSI is cut into non-overlapping image patches of 256×256 pixels with a step size of 256 pixels, and blank patches with pixel values <5% are filtered out.
[0083] (2) Extraction of cell nuclear features
[0084] Segmentation and classification: A pre-trained HoverNet model (trained on the BCaPanNuke dataset) is used to output cell nucleus masks and type labels (tumor cells, inflammatory cells, etc.). The specific process is as follows: ① NP branch identifies cell nucleus pixels; ② NC branch classifies cell types; ③ HoVer branch predicts centroid distance and separates overlapping cell nuclei.
[0085] Feature calculation: 15 morphological features (eccentricity, equivalent diameter, Euler number, etc.) and 1 type feature (tumor cell proportion = number of tumor cells / total number of cells) are extracted from each cell nucleus. These features are then aggregated into 16-dimensional cell nucleus features through sample-level statistics (mean, standard deviation) to calculate morphological parameters (eccentricity, equivalent diameter) and the proportion of tumor cells.
[0086] (3) CN feature optimization:
[0087] Clinical feature coding: Categorical variables (such as HER2 status) are one-hot coded (0→[1,0,0], 1+→[0,1,0], 2+FISH-→[0,0,1]), and continuous variables (such as age and tumor size) are min-max normalized (range [0,1]), finally resulting in 7-dimensional clinical features.
[0088] Feature dimensionality reduction: Pearson and Spearman correlation coefficients of clinical and nuclear features were calculated. For highly correlated features (such as tumor size and area) with P>0.8, PCA dimensionality reduction was performed: ① The features were standardized to mean 0 and variance 1; ② The covariance matrix was calculated and eigenvalue decomposition was performed; ③ The first two principal components (cumulative variance contribution rate 95%) were selected and combined with the low-correlation features (P≤0.8) to form 13-dimensional CN features.
[0089] Feature enhancement: Constructing a cascaded transformation architecture: ① Fully connected layer (nn.Linear(13,256)) increases the dimension to 256; ② ReLU activation function (ReLU(x)=max(0,x)) introduces non-linearity; ③ Dropout layer (rate 0.5) suppresses overfitting; Output the enhanced 256-dimensional CN features, which are matched with the dimensions of WSI features (512 dimensions). Through cascaded fully connected layers, ReLU activation, and Dropout regularization, the 13-dimensional features are mapped to a 256-dimensional balanced space, solving the dimensionality imbalance problem with WSI features.
[0090] III. Model Construction:
[0091] This embodiment is implemented based on the PyTorch framework, and the network structures for the three fusion strategies are as follows:
[0092] (1) MCLAM_CN (direct concatenation):
[0093] WSI Feature Extraction: A pre-trained ResNet50 was used as the backbone network. The first four layers were frozen, and the last layer was replaced with a 512-dimensional fully connected layer.
[0094] Output image patch features; aggregated into WSI global features (512 dimensions) through attention pooling (Attn_Net).
[0095] Feature concatenation: The WSI features (512 dimensions) and CN features (256 dimensions) are concatenated along the feature dimensions to obtain a 768-dimensional fused feature.
[0096] Classification layer: The fused features are passed through a fully connected layer (256→128→2), and the risk probability (low risk / high risk) is output through Softmax.
[0097] (2) MCLAM_sb (single-branch attention fusion)
[0098] Feature concatenation: Same as MCLAM_CN, resulting in 768-dimensional fused features.
[0099] Attention mechanism: A non-gated attention network (Attn_Net) is adopted, with the following structure: ① Fully connected layer 1 (768→128) + Tanh activation; ② Fully connected layer 2 (128→1) outputs attention weights; ③ Weights are normalized (Softmax) and then weighted and summed with the fused features to obtain weighted features.
[0100] Classification layer: Weighted features pass through a fully connected layer (128→2), and Softmax outputs the risk probability.
[0101] (3) MCLAM_mb (Multi-branch customized fusion)
[0102] Branch construction: Copy the CN feature into 2 branches (corresponding to high and low risk), and concatenate each branch with the class-specific WSI feature: ① Class 1
[0103] Branch 1: WSI feature (high risk weight) + CN feature 1; ② Category 2 Branch 2: WSI feature (low risk weight) + CN feature 2.
[0104] Attention calculation: Each branch uses a gated attention network (Attn_Net_Gated) independently, with the following structure: ① Fully connected layer 1 (512→64) + Tanh; ② Fully connected layer 2 (512→64) + Sigmoid; ③ Element-wise multiplication to obtain the gate weights; ④ Weighted aggregation to form branch features.
[0105] Classification layer: Each branch feature passes through a fully connected layer (64→1), outputting two class probabilities, which are then fused into the final result through Softmax.
[0106] IV. Model Training and Evaluation Module:
[0107] (1) Training settings:
[0108] Dataset partitioning: Patient-level 10-fold cross-validation was used, and the training set, validation set, and test set were split in an 8:1:1 ratio to maintain the same high-risk and low-risk ratio as the original dataset (43% high-risk and 57% low-risk).
[0109] Hyperparameter configuration: Optimizer is Adam, learning rate is 1e-4, weight decay is 1e-5, batch size is 32, iterations are 300 rounds, and an early stopping strategy is adopted (if the validation set AUC does not improve for 10 consecutive rounds, the process stops).
[0110] Loss function: Cross-entropy loss is used, and clustering loss is added to MCLAM_mb.
[0111] (2) Evaluation indicators:
[0112] Classification performance: ① AUC (Area Under the ROC Curve) and 95% confidence interval (calculated using the bootstrap method, 1000 samples); ② Acc (Accuracy = (TP + TN) / Total number of samples); ③ Calibration curve (Brier score = Σ(p _i -y _i ()² / N, assess probability calibration.
[0113] Survival analysis: ① Kaplan-Meier curve: DFS curves for high- and low-risk groups were plotted, and the p-value was calculated using the Log-rank test; ② Hazard ratio (HR) and 95% confidence interval: Estimated using the Cox proportional hazards model.
[0114] (3) Verification results:
[0115] Internal validation: MCLAM_sb showed the best performance, with a validation set AUC of 0.8250 (95% CI 0.7812-0.8688). Figure 2 As shown, Test_AUC=0.8181 (95% CI 0.7625-0.8737); MCLAM_mb validation set Acc=0.7480, Test_Acc=0.7160.
[0116] External validation: such as Figure 3As shown, a total of 75 patients with early-stage HR-positive, HER2-negative cancer who did not receive adjuvant therapy after surgery were included in the study for the first time. Using the MCLAM_sb_CN model, 48 patients were classified as low-risk and 17 as high-risk. The 5-year DFS rate was 100% in the low-risk group and 82.35% in the high-risk group (P=0.003), while the 10-year DFS rates were 100% and 76.47%, respectively (P<0.001). Similarly, the DDFS in the low-risk group was also significantly better: Figure 3 As shown, a) is the 5-year DDFS rate of 100% versus 88.24% (P=0.016), and b) is the 10-year DDFS rate of 100% versus 82.35% (P=0.002).
[0117] The prognostic value of MCLAM-based risk stratification for HER2-low breast cancer:
[0118] The results showed that the MCLAM model outperformed other models in predicting prognosis, and its effectiveness has been validated in real-world settings. Given that the population consisted of HER2-negative patients, the aim was to evaluate the performance of the MCLAM model in the context of low HER2 expression. Data were collected from 273 patients with early postoperative HER2-low expression cancer, including 75 low-risk and 198 high-risk patients. Significant differences in disease-free survival (DFS) were observed between the two risk groups. The 5-year DFS rate was 90.67% in the low-risk group and 74.24% in the high-risk group (P=0.003), while the 10-year DFS rates were 89.33% and 70.2%, respectively (P=0.08×10⁻⁶). -4 The heart rate was 3.30 (95% confidence interval: 1.58–6.92). Consistent with these findings, the low-risk group also had significantly better DDFS. The 5-year DDFS rate was 93.33%, compared to 76.26% in the high-risk group (P=0.001; HR=4.05, 95% CI: 1.61–10.19), and the 10-year DDFS rate was 92.00%, compared to 74.24% in the high-risk group (P=0.001, HR=3.75, 95% CI: 1.61–8.74).
[0119] V. Model Application:
[0120] Input: HE slides and clinical data (age, ER / PR, etc.) of the patient to be predicted;
[0121] Output: Stratification results of high and low recurrence risk (probability value + risk level) and visualization report (attention heatmap, contribution of key features);
[0122] Application scenarios: Assisting doctors in developing treatment plans (such as exempting low-risk patients from chemotherapy and intensifying treatment for high-risk patients).
[0123] The proposed Multimodal Feature Structured Fusion Framework (MCLAM) in this implementation achieves low-cost, high-precision risk stratification through three innovative strategies:
[0124] Data layer: integrates WSI, clinical data (age / ER / PR / Ki-67), and cell nuclear characteristics (16-dimensional morphology + 1-dimensional type ratio).
[0125] Algorithm layer: Design attention-weighted fusion (MCLAM_sb) and multi-branch customized fusion (MCLAM_mb) to dynamically optimize feature contribution weights.
[0126] Application layer: Outputs binary risk labels (high / low risk) to directly guide clinical chemotherapy decisions.
[0127] In this embodiment, in a cohort of 254 patients, the AUC of MCLAM_sb on the test set reached 0.8181, which was significantly better than ResNet50 (AUC 0.61) and DTFD_MIL (AUC 0.78).
[0128] This embodiment requires only routine HE staining sections and clinical data, and costs less than 10% of EndoPredict, making it suitable for primary hospitals.
[0129] In this embodiment, the 10-year disease-free survival rate of the high-risk group stratified by MCLAM was 76.47% lower than that of EndoPredict (P<0.001), confirming its clinical equivalence.
[0130] The electronic device of this embodiment includes a memory and a processor. The memory is used to store non-transitory computer-readable instructions. Specifically, the memory may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may, for example, include random access memory (RAM) and / or cache memory. The non-volatile memory may, for example, include read-only memory (ROM), hard disk, flash memory, etc.
[0131] The processor may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions. In one embodiment of this disclosure, the processor is used to execute computer-readable instructions stored in the memory, causing the electronic device to perform all or part of the steps of the foregoing embodiments of the present disclosure for constructing a multimodal deep learning fusion model for breast cancer.
[0132] Those skilled in the art will understand that, in order to solve the technical problem of how to achieve a good user experience, this embodiment may also include well-known structures such as communication buses and interfaces, and these well-known structures should also be included within the protection scope of this disclosure.
[0133] like Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. It illustrates a structural schematic diagram suitable for implementing the electronic device in the embodiment of the present disclosure. Figure 4 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.
[0134] like Figure 4 As shown, an electronic device may include a processing unit (such as a central processing unit, graphics processing unit, etc.) that can perform various appropriate actions and processes based on a program stored in read-only memory (ROM) or a program loaded from a storage device into random access memory (RAM). The RAM also stores various programs and data required for the operation of the electronic device. The processing unit, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also connected to the bus.
[0135] Typically, the following devices can be connected to the I / O interface: input devices, such as sensors or visual information acquisition devices; output devices, such as displays; storage devices, such as magnetic tapes or hard drives; and communication devices. Communication devices allow electronic devices to exchange data wirelessly or via wired communication with other devices, such as edge computing devices. Although Figure 4 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.
[0136] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device, or installed from a ROM. When the computer program is executed by a processing device, all or part of the steps of the method for constructing a multimodal deep learning fusion model for breast cancer according to embodiments of this disclosure are performed.
[0137] For a detailed description of this embodiment, please refer to the corresponding descriptions in the foregoing embodiments, which will not be repeated here.
[0138] A computer-readable storage medium according to embodiments of the present disclosure stores non-transitory computer-readable instructions. When these non-transitory computer-readable instructions are executed by a processor, all or part of the steps of the foregoing embodiments of the present disclosure for constructing a multimodal deep learning fusion model for breast cancer are performed.
[0139] The aforementioned computer-readable storage media include, but are not limited to: optical storage media (e.g., CD-ROM and DVD), magneto-optical storage media (e.g., MO), magnetic storage media (e.g., magnetic tape or portable hard drive), media with built-in rewritable non-volatile memory (e.g., memory card), and media with built-in ROM (e.g., ROM cartridge).
[0140] For a detailed description of this embodiment, please refer to the corresponding descriptions in the foregoing embodiments, which will not be repeated here.
[0141] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.
[0142] In this disclosure, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The block diagrams of devices, apparatuses, devices, and systems involved in this disclosure are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as "comprising," "including," "having," etc., are open-ended terms meaning "including but not limited to," and are used interchangeably with them. The terms "or" and "and" as used herein refer to the terms "and / or," and are used interchangeably with them unless the context clearly indicates otherwise. The term "such as" as used herein refers to the phrase "such as but not limited to," and is used interchangeably with it.
[0143] Additionally, as used herein, the "or" used in a list of items beginning with "at least one" indicates a separate list, such that a list of, for example, "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the word "exemplary" does not imply that the described example is preferred or better than other examples.
[0144] It should also be noted that in the systems and methods of this disclosure, the components or steps can be decomposed and / or recombined. These decompositions and / or recombinations should be considered as equivalent solutions to this disclosure.
[0145] Various changes, substitutions, and modifications can be made to the technology described herein without departing from the teachings defined by the appended claims. Furthermore, the scope of the claims of this disclosure is not limited to the specific aspects of the processes, machines, manufactures, events, means, methods, and actions described above. Currently existing or later-developed processes, machines, manufactures, events, means, methods, or actions that perform substantially the same function or achieve substantially the same result as the corresponding aspects described herein can be utilized. Therefore, the appended claims include such processes, machines, manufactures, events, means, methods, or actions within their scope.
[0146] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.
[0147] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.
Claims
1. A method for constructing a multimodal deep learning fusion model for breast cancer, characterized in that, include: Obtain the dataset; Preprocess the data in the dataset; Constructing a multimodal deep learning fusion model, the construction of the multimodal deep learning fusion model includes direct splicing network structure, single-branch attention fusion network structure and multi-branch customized fusion network structure; The multimodal deep learning fusion model is trained based on the preprocessed dataset to obtain the trained multimodal deep learning fusion model. The direct splicing network structure is used to directly splice WSI features and CN features. CN features are features obtained after dimensionality reduction and enhancement of clinical features and cell nuclear features. The single-branch attention fusion network structure is used to introduce a dynamic weighting attention mechanism after splicing; The multi-branch customized fusion network structure is used to independently design attention branches for each category and then splice them together after copying CN features; The direct splicing network structure is used to directly splice WSI features and CN features, including: WSI feature extraction uses a pre-trained ResNet50 as the backbone network, freezes the first 4 layers of ResNet50, and replaces the last layer of ResNet50 with a 512-dimensional fully connected layer. Output image patch features, and aggregate the extracted WSI features into WSI global features through attention pooling; Feature concatenation involves concatenating the WSI global features with the CN features along the feature dimension to obtain the fused features. The classification layer fuses features through a fully connected layer, and outputs the risk probability via Softmax. The single-branch attention fusion network structure is used to introduce a dynamic weighting attention mechanism after splicing, including: Feature concatenation yields fused features; The first fully connected layer is used to activate the attention mechanism, and the second fully connected layer is used to output the attention weights. After the weights are normalized, they are weighted and summed with the fused features to obtain the weighted features. After the weighted features are classified by the third fully connected layer, the risk probability is output through Softmax. The multi-branch customized fusion network structure is used to independently design attention branches for each category, and then concatenate them after copying CN features, including: The CN feature is copied into two branches, each branch is concatenated with the category-specific WSI feature. The first branch is the high-risk weight WSI feature and the first CN feature; the second branch is the low-risk weight WSI feature and the second CN feature. Each branch uses a gated attention network independently. The category-specific WSI features include the high-risk weight WSI feature and the low-risk weight WSI feature.
2. The method for constructing a multimodal deep learning fusion model for breast cancer according to claim 1, characterized in that, The gated attention network structure includes element-wise multiplication to obtain gate weights and weighted aggregation to form branch features.
3. The method for constructing a multimodal deep learning fusion model for breast cancer according to claim 1, characterized in that, The preprocessing of the data in the dataset includes: WSI preprocessing, nuclear feature extraction, and CN feature optimization.
4. The method for constructing a multimodal deep learning fusion model for breast cancer according to claim 3, characterized in that, The extraction of cell nuclear features includes: Output cell kernel masks and type labels using a pre-trained HoverNet model; Fifteen morphological features and one type feature were extracted from each cell nucleus, and then aggregated into 16-dimensional cell nucleus features through sample-level statistical aggregation.
5. The method for constructing a multimodal deep learning fusion model for breast cancer according to claim 4, characterized in that, The step of using a pre-trained HoverNet model to output cell nucleus masks and type labels includes: using the NP branch of the pre-trained HoverNet model to identify cell nucleus pixels, using the N branch of the pre-trained HoverNet model to classify cell types, using the HoVer branch of the pre-trained HoverNet model to predict centroid distance, and separating overlapping cell nuclei.
6. The method for constructing a multimodal deep learning fusion model for breast cancer according to claim 1, characterized in that, After the step of training the constructed multimodal deep learning fusion model based on the preprocessed dataset to obtain the trained multimodal deep learning fusion model, the method further includes: The trained multimodal deep learning fusion model is evaluated, and the evaluation metrics include the area under the ROC curve and confidence interval, accuracy, and calibration curve.