Breast cancer pcr prediction method and system based on multi-modal mri fusion

Abstract

The application provides a breast cancer pCR prediction method and system based on multi-modal MRI fusion, belongs to the technical field of medical artificial intelligence and medical image processing, and aims to solve the technical problems of insufficient multi-modal fusion, weak feature discriminability, poor cross-center generalization ability and lack of longitudinal change description of curative effect in the existing prediction method. Through parallel processing of dynamic contrast enhancement (DCE) and diffusion weighted imaging (DWI) sequences by a double-tower attention network, a feature quality enhancement module is introduced to impose intra-class compactness, inter-class separability and consistency constraints to improve feature robustness. At the same time, pre-treatment and post-treatment radiomic features are extracted, and difference radiomic features are calculated to reflect the dynamic change of curative effect. After hierarchical feature screening, the deep features, radiomic features, difference radiomic features and clinical pathological variables are input into a logistic regression model, and the pCR prediction result is output, so that stable and accurate preoperative pCR prediction can be realized under the condition of multi-center heterogeneous data.

Description

Technical Field

[0001] This invention relates to the fields of medical artificial intelligence and medical image processing technology, and more specifically, to a method and system for predicting pCR in breast cancer using multimodal MRI fusion. Background Technology

[0002] Breast cancer is one of the most common malignant tumors in women and a leading cause of cancer-related deaths worldwide. Neoadjuvant chemotherapy (NAC) has become the standard treatment for locally advanced breast cancer, as well as triple-negative and HER2-positive breast cancer subtypes. Whether a patient achieves pathological complete response (pCR) after neoadjuvant chemotherapy—defined as the absence of residual invasive cancer in the surgical specimen—is a crucial indicator for assessing efficacy and guiding subsequent treatment and surgical strategies. Studies have shown that pCR is closely associated with improved disease-free survival and overall survival. However, pCR currently relies primarily on postoperative pathological examination for confirmation. This invasive procedure is time-consuming and cannot comprehensively assess tumor heterogeneity, making it difficult to provide timely decision support for treatment adjustments and surgical de-escalation strategies preoperatively.

[0003] Multiparametric magnetic resonance imaging (mpMRI), combined with dynamic contrast-enhanced (DCE) and diffusion-weighted imaging (DWI), has been widely used to assess treatment response after neoadjuvant chemotherapy. DCE sequences can capture contrast agent kinetics related to vascular permeability, reflecting tumor perfusion characteristics; DWI sequences, on the other hand, reflect microstructural properties related to cell density through apparent diffusion coefficients. Therefore, mpMRI can reflect the tumor's response to neoadjuvant chemotherapy from multiple dimensions, including perfusion and cell density, and is widely recognized as an important imaging tool for efficacy assessment.

[0004] In recent years, with the rapid development of artificial intelligence technology, deep learning-based medical image analysis has provided a new technical approach for preoperative prediction of pCR (prophylactic cerebral refractory). However, existing image- or clinical-based pCR prediction methods still have the following shortcomings: First, multimodal fusion often uses simple stitching, making it difficult to fully model the complementary relationship between DCE (discretionary echocardiography) and DWI (discretionary image tracing); second, deep learning features are usually only driven by classification objectives, resulting in insufficient feature discriminative power and susceptibility to differences in scanning protocols, equipment, and populations across different centers, leading to decreased cross-institutional generalization performance; third, some methods ignore information on changes in images before and after treatment, lacking a characterization of dynamic changes in therapeutic efficacy. Therefore, a stable and accurate preoperative pCR prediction technology solution is needed that can maintain stability under multi-center heterogeneous data conditions. Summary of the Invention

[0005] The present invention aims to solve the technical problems mentioned in the background art and provide a method and system for predicting pCR of breast cancer by multimodal MRI fusion.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a method for predicting pCR in breast cancer using multimodal MRI fusion, comprising the following steps: S1: Collect multi-parameter magnetic resonance (MRI) imaging data and clinicopathological data of breast cancer patients after neoadjuvant chemotherapy. The multi-parameter magnetic resonance imaging data includes at least dynamic contrast enhancement (DCE) sequences and diffusion-weighted (DWI) sequences. S2: Preprocess the multi-parameter magnetic resonance imaging data, including format normalization, spatial alignment / registration, resampling, and intensity normalization; S3: Input the preprocessed dynamic contrast enhancement sequence and diffusion weighted sequence into the deep feature extraction network to extract dynamic contrast enhancement deep features and diffusion weighted deep features, respectively. S4: Multimodal fusion is performed on the dynamic contrast-enhanced deep features and diffusion-weighted deep features, and a feature quality enhancement mechanism is introduced during the training process to make the feature distribution of similar samples more compact and the feature interval of dissimilar samples larger, so as to obtain quality-enhanced multimodal deep features. S5: Extract radiomics features from the multi-parameter magnetic resonance imaging data, including shape descriptors, first-order statistics, and texture features; S6: Given pre-treatment and post-treatment imaging data, calculate the difference radiomics feature, which is the difference between the post-treatment radiomics feature and the pre-treatment radiomics feature; S7: Perform structured coding on the clinicopathological data to generate clinical feature vectors; S8: The enhanced multimodal deep features, radiomics features, differential radiomics features, and clinical feature vectors are standardized, redundantly removed, and screened and fused to obtain the final multimodal feature set; S9: Input the final multimodal feature set into the trained classification model and output the prediction result of whether the breast cancer patient has achieved pathological complete remission (pCR) after neoadjuvant chemotherapy.

[0007] Furthermore, in step S4, the feature quality enhancement mechanism is implemented through the following three constraints: Intra-class compactness constraint is used to make the features of samples of the same class move closer to the class center of that class; Inter-class separability constraints are used to maximize the Euclidean distance between class centers of different classes. Consistency constraints are used to maintain the consistency of feature representations of similar samples before and after data augmentation.

[0008] Furthermore, in step S4, the multimodal fusion employs a dual-tower attention network, where the dynamic contrast enhancement sequence and the diffusion weighted sequence are encoded by two parallel three-dimensional ResNet branches, and the nonlinear complementary information between the two sequences is captured through a multimodal attention block.

[0009] Further, in step S8, the screening includes: performing variance threshold filtering on the features, univariate feature selection based on the F statistic, and correlation filtering to remove redundant features; When the number of candidate features exceeds a preset threshold, an integrated voting mechanism is used for final feature selection.

[0010] Furthermore, in step S9, the classification model is an L2-regularized logistic regression model.

[0011] Furthermore, in step S1, the clinicopathological data includes T stage, estrogen receptor status, progesterone receptor status, HER2 status, Ki-67 index, and neoadjuvant chemotherapy regimen.

[0012] Furthermore, in step S5, the texture features include gray-level co-occurrence matrix features and gray-level run-length matrix features.

[0013] A multimodal MRI fusion-based pCR prediction system for breast cancer includes: The data acquisition module is used to acquire multi-parameter magnetic resonance imaging data and clinicopathological data of breast cancer patients after neoadjuvant chemotherapy. The multi-parameter magnetic resonance imaging data includes at least dynamic contrast-enhanced sequences and diffusion-weighted sequences. The preprocessing module is used to perform format standardization, spatial alignment / registration, resampling, and intensity normalization on the multi-parameter magnetic resonance imaging data. The deep feature extraction module is used to extract dynamic contrast-enhanced deep features and diffusion-weighted deep features from the preprocessed dynamic contrast-enhanced sequence and diffusion-weighted sequence, respectively. The multimodal fusion and quality enhancement module is used to perform multimodal fusion on the dynamic contrast-enhanced depth features and diffusion-weighted depth features, and to make the feature distribution of similar samples more compact and the feature interval of dissimilar samples larger through the feature quality enhancement mechanism, and output the quality-enhanced multimodal depth features. A radiomics feature extraction module is used to extract radiomics features from the multi-parameter magnetic resonance imaging data; The differential radiomics calculation module is used to calculate the difference between the radiomics characteristics after treatment and the radiomics characteristics before treatment, given pre-treatment imaging data, as the differential radiomics feature. The clinical feature encoding module is used to perform structured encoding on the clinicopathological data and generate clinical feature vectors. The feature filtering and fusion module is used to filter and fuse the quality-enhanced multimodal deep features, radiomics features, differential radiomics features, and clinical feature vectors to obtain the final multimodal feature set. The prediction output module is used to input the final multimodal feature set into the trained classification model and output the prediction result of whether the breast cancer patient has achieved pathological complete remission after neoadjuvant chemotherapy.

[0014] Furthermore, the multimodal fusion and quality enhancement module includes a dual-tower attention network and a feature quality enhancement submodule. The dual-tower attention network uses two parallel 3D ResNet branches to encode the dynamic contrast enhancement sequence and the diffusion weighted sequence respectively, and performs feature fusion through multimodal attention blocks. The feature quality enhancement submodule is used to apply intra-class compactness constraints, inter-class separability constraints, and consistency constraints.

[0015] Furthermore, the classification model is an L2-regularized logistic regression model.

[0016] The beneficial effects of this invention are as follows: Addressing existing problems such as insufficient multimodal fusion, weak discriminative power of deep features, poor cross-center generalization ability, and lack of longitudinal variation information, this invention proposes a multimodal fusion prediction concept with enhanced feature quality. Deep features are extracted from DCE and DWI at the image end, and multimodal feature fusion is performed. Simultaneously, a feature quality enhancement mechanism is introduced to make features of similar samples more compact and features of dissimilar samples more separable, thereby improving feature stability and resistance to domain shift. Radiomic features are extracted at the image phenotypic end, and the difference in radiomic features before and after treatment is further calculated to reflect changes in efficacy. Clinicopathological variables are also introduced to form an interpretable feature set. Subsequently, through feature selection and fusion, logistic regression modeling is used to output pCR prediction results. The beneficial effects of this concept are: improving the accuracy and robustness of prediction under multi-center data conditions, achieving reliable preoperative assessment of pCR, and providing auxiliary basis for individualized treatment and surgical decisions. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings in the following description are only drawings of some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings. In addition, the drawings in the following description can be regarded as schematic diagrams and are not intended to limit the actual size of the product, the actual flow of the method, the actual timing of the signals, etc. involved in the embodiments of the present invention.

[0018] Figure 1 This is a schematic diagram of the method steps of the present invention.

[0019] Figure 2 This is a schematic diagram of the multicenter breast cancer patient data inclusion process of the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] The serial numbers assigned to components in this document, such as "first" and "second," are used only to distinguish the described objects and have no sequential or technical meaning. The terms "connection" and "linkage" used in this application, unless otherwise specified, include both direct and indirect connections (linkages). In the description of this invention, it should be understood that the terms "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," and "counterclockwise," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention.

[0022] In this invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can mean that the first feature is in direct contact with the second feature, or that the first feature is in indirect contact with the second feature through an intermediate medium. Furthermore, "above," "over," and "on top" of the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply indicates that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply indicates that the first feature is at a lower horizontal level than the second feature.

[0023] like Figure 1 As shown in the figure and below, the various markings are explained as follows: DCE: Dynamic Contrast-Enhanced MRI DWI: Diffusion-Weighted Imaging ROI: Region of Interest pCR: Pathological Complete Response (after neoadjuvant chemotherapy) NAC: Neoadjuvant Chemotherapy FQE: Feature Quality Enhancer MAB: Multimodal Attention Block The multimodal MRI fusion-based method for predicting pCR in breast cancer provided in this invention includes the following steps: S1: Data Acquisition; Acquire multiparametric magnetic resonance imaging (MRI) data and clinicopathological data of breast cancer patients after neoadjuvant chemotherapy. The multiparametric MRI data should include at least dynamic contrast-enhanced (DCE) sequences and diffusion-weighted (DWI) sequences. The clinicopathological data should include T stage, estrogen receptor (ER) status, progesterone receptor (PR) status, HER2 status, Ki-67 index, and neoadjuvant chemotherapy regimen. If pre-treatment imaging is available, pre-treatment MRI images should also be acquired for differential radiomics calculations. S2: MRI Image Preprocessing and ROI Segmentation; Unified preprocessing of acquired DCE and DWI images, specifically including: format standardization (converting DICOM data to NIfTI format); coordinate redirection to a standard coordinate system; spatial registration (spatial alignment of images before and after treatment); resampling to a fixed voxel spacing; 3D Region of Interest (ROI) segmentation—semi-automatically delineating 3D ROIs covering the entire tumor volume on DCE and DWI images, with verification by a senior physician; uniformly cropping / scaling the ROI volume to a preset 3D network input size (e.g., 96×96×32 voxels); intensity normalization of non-zero voxels within the ROI to eliminate grayscale differences between different scanners. During the training phase, data augmentation operations involving random flipping, random rotation, and random intensity shifting are applied to the preprocessed images to expand sample diversity and enhance model robustness. S3: DCE image depth feature extraction; The preprocessed DCE image is input into a three-dimensional depth feature extraction network to obtain a depth feature representation that reflects tumor perfusion and enhancement dynamics information; S4: DWI image depth feature extraction; The preprocessed DWI image is input into a three-dimensional depth feature extraction network to obtain a depth feature representation that reflects tumor spread limitation and microstructure information; S5: Multimodal Feature Fusion and Feature Quality Enhancement; A dual-tower attention network is used to fuse the deep features of DCE and DWI to construct a multimodal joint representation. The dual-tower attention network uses two parallel 3D ResNet branches to encode the DCE and DWI sequences respectively, and captures the nonlinear complementary information between the two sequences through a multimodal attention block (MAB). Simultaneously, a feature quality enhancement mechanism (FQE) is introduced to make the feature distribution of samples of the same class (pCR or non-pCR) more compact, the feature interval between different classes larger, and maintain the consistency of feature representation before and after data enhancement for samples of the same class, thereby improving cross-center robustness and discriminative ability. Specifically, the feature quality enhancement mechanism is implemented through the following three constraints: Intra-class compactness constraint: Makes the features of samples of the same class move closer to the class center of that class; Inter-class separability constraint: Maximize the Euclidean distance between the class centers of different classes; Consistency constraint: Maintain the consistency of feature representations for similar samples before and after data augmentation; The total loss function of the dual-tower attention network is: Ltotal=LCE+λintraLintra+λinterLinter+λconsLcons Where LCE is the cross-entropy classification loss, Lintra is the intra-class compactness loss, Linter is the inter-class separability loss, and Lcons is the consistency loss; λintra, λinter, and λcons are the corresponding weight hyperparameters. S6: Radiomics Feature Extraction; Based on the tumor region of interest (ROI), handcrafted radiomics features are extracted from DCE and DWI images, including shape descriptors, first-order statistics, and texture features (such as gray-level co-occurrence matrix (GLCM) and gray-level run-length matrix (GLRLM) features) to supplement the interpretable image phenotypic information of depth features. S7: Calculation of differential radiomics features; When pre- and post-treatment images are available, calculate differential radiomics features (post-treatment radiomics features minus pre-treatment radiomics features) to characterize longitudinal changes caused by therapeutic effects and improve sensitivity to treatment response. S8: Clinical information extraction; clinical variables (T stage, ER / PR / HER2 status, Ki-67 index, neoadjuvant chemotherapy regimen, etc.) are structured and encoded to form clinical feature vectors, which are then used in fusion and modeling together with imaging-related features. S9: Feature Selection and Fusion; Standardization, redundancy removal, and selection and fusion of quality-enhanced multimodal deep features, radiomics features, differential radiomics features, and clinical features; Specifically, this includes: variance threshold filtering (removing features with variances below the 75th percentile of the total variance), univariate feature selection based on the F-statistic, and correlation filtering (removing redundant feature pairs with lower F-statistics among those with Pearson correlation coefficients greater than 0.85); When the number of candidate features exceeds a preset threshold (e.g., 60), an integrated voting mechanism is used for final feature selection—features are independently ranked by importance using two algorithms, CatBoost and ANOVA F-test, and the final comprehensive importance score of each feature is calculated by Borda counting voting aggregation, retaining the feature with the highest score to obtain the final multimodal feature set used for prediction; S10: Logistic Regression Modeling and Prediction Output; The final features obtained in step S9 are input into the L2 regularized logistic regression model for training or inference, and the predicted result of breast cancer patients achieving pCR (pCR prediction output) is output, providing an auxiliary reference for preoperative efficacy assessment.

[0024] A multimodal MRI fusion-based pCR prediction system for breast cancer includes: The data acquisition module is used to collect multi-parameter magnetic resonance imaging data and clinicopathological data of breast cancer patients after neoadjuvant chemotherapy; The preprocessing module is used to perform format standardization, coordinate redirection, spatial registration, resampling to a fixed voxel spacing, 3D region of interest segmentation, cropping to a preset network input size, and intensity normalization on multi-parameter magnetic resonance image data. During the training phase, it is also used to apply data augmentation operations such as random flipping, random rotation, and random intensity shift to the preprocessed images. The deep feature extraction module is used to extract DCE deep features and DWI deep features from the preprocessed DCE sequence and DWI sequence, respectively. The multimodal fusion and quality enhancement module is used to perform multimodal fusion of DCE deep features and DWI deep features through a dual-tower attention network, and to make the feature distribution of similar samples more compact and the feature interval of dissimilar samples larger through a feature quality enhancement mechanism, so as to output high-quality multimodal deep features. The radiomics feature extraction module is used to extract radiomics features from multi-parameter magnetic resonance imaging data; The differential radiomics calculation module is used to calculate the difference between the radiomics characteristics after treatment and the radiomics characteristics before treatment, given pre-treatment imaging data, as the differential radiomics feature. The clinical feature coding module is used to perform structured coding on clinicopathological data and generate clinical feature vectors. The feature filtering and fusion module is used to filter and fuse quality-enhanced multimodal deep features, radiomics features, differential radiomics features, and clinical feature vectors to obtain the final multimodal feature set. The prediction output module is used to input the final multimodal feature set into the trained classification model and output the prediction result of whether breast cancer patients have achieved pathological complete remission after neoadjuvant chemotherapy.

[0025] like Figure 2 As shown, this embodiment establishes a development queue and an external validation queue based on multicenter breast cancer patient data.

[0026] (1) Data source: The data comes from the main center and three external validation centers. The main center includes the SYF center (n=426) and the ZY center (n=337); the first external validation center is the YiZhong center (n=203); the second external validation center is the ShanZhong center (n=332); and the third external validation center is the Xian center (n=195).

[0027] (2) Exclusion criteria: Data that do not meet the research criteria will be excluded, including: those who have not undergone surgery, those with bilateral breast cancer, those with poor MRI image quality (such as severe motion artifacts or geometric distortions that prevent radiomics or deep learning analysis), those with incomplete clinical data, those with multifocal or occult breast cancer that cannot be reliably quantified as single lesions, and those with concurrent malignant tumors (the specific exclusion numbers for each center are as follows). Figure 2 (As shown).

[0028] (3) Queue division: After the exclusion is completed, the main center forms a development queue (n=684), which is formed by merging the data of the SYF center and the ZY center. The external verification centers form verification queue 1 (n=195), verification queue 2 (n=239), and verification queue 3 (n=181) respectively, which are used to evaluate the cross-center generalization ability and stability of the method of the present invention.

[0029] All patients were pathologically confirmed to have invasive breast cancer (T1-T4, N0-N3, M0 stages) and received standard neoadjuvant chemotherapy regimens based on anthracyclines or taxanes. HER2-positive patients also received targeted therapy with trastuzumab and / or pertuzumab and underwent radical surgery and postoperative pathological evaluation after completing neoadjuvant chemotherapy.

[0030] like Figure 1 As shown, the pCR prediction method in this embodiment includes the following steps: Step S1: Data Acquisition (MRI Images and Clinical Data) MRI imaging data of breast cancer patients after neoadjuvant chemotherapy were acquired, including dynamic contrast-enhanced (DCE) and diffusion-weighted (DWI) sequences. Clinical and pathological information, including age, T stage, ER / PR / HER2 status, Ki-67 index, and neoadjuvant chemotherapy regimen, were also collected. ER / PR positivity was defined as ≥1% nuclear staining, and HER2 positivity was defined as amplification confirmed by immunohistochemistry 3, or immunohistochemistry 2 with fluorescence in situ hybridization (FISH). Pre-treatment MRI was also acquired when available for differential radiomics calculations.

[0031] Step S2: Image and Data Preprocessing MRI images were preprocessed using standardized methods. MRI examinations were performed using 1.5T or 3.0T scanners from multiple vendors (including GE, Siemens, and Philips), and standardized protocols included T1-weighted dynamic contrast-enhanced imaging and diffusion-weighted imaging.

[0032] Preprocessing specifically includes the following sub-steps: (1) Format conversion and redirection: Convert DICOM data to NIfTI format and redirect it to the standard coordinate system; (2) Spatial registration and resampling: Spatial alignment of images before and after treatment, and resampling to a fixed voxel spacing; (3) Region of Interest (ROI) segmentation: On DCE and DWI images, a three-dimensional region of interest (3D ROI) covering the entire tumor volume is delineated in a semi-automatic manner. The delineation is performed by a radiologist with five years of experience and reviewed and confirmed by a senior radiologist with more than ten years of experience. (4) Size normalization: The ROI volume is uniformly cropped / scaled to the preset 3D network input size (96×96×32 voxels). (5) Intensity normalization: Intensity normalization is performed on non-zero voxels within the ROI to eliminate grayscale differences between different scanners; (6) Data augmentation (training phase only): Apply random flipping, random rotation and random intensity shift to the preprocessed images to increase sample diversity and enhance model robustness; deterministic preprocessing is used in the validation and inference phases to ensure reproducibility.

[0033] Step S3: DCE Image Depth Feature Extraction The preprocessed DCE image (X DCE ∈R C×D×H×WInput 3D depth feature extraction network f DCE (⋅), extracting deep feature representations that reflect tumor perfusion and enhancement dynamics: E DCE =f DCE (X DCE ) Step S4: DWI image depth feature extraction The preprocessed DWI image (X DWI ∈R C×D×H×W Input 3D depth feature extraction network f DWI (⋅), extracting deep feature representations that reflect tumor spread restriction and microstructural information: E DWI =f DWI (X DWI ) Step S5: Multimodal Feature Fusion and Feature Quality Enhancement A dual-tower attention network is used to fuse the deep features of DCE and DWI. The dual-tower attention network is built on a 3D ResNet-18 backbone network, and DCE and DWI are encoded by two parallel 3D ResNet-18 branches respectively.

[0034] Multimodal attention blocks (MABs) are used to capture nonlinear complementary information between the DCE and DWI. During the training phase, a feature quality enhancer (FQE) module is introduced to impose structural constraints on the latent representation.

[0035] Specifically, let the fusion feature of the i-th sample be hi, and its corresponding class label be yi∈{0,1} (1 represents pCR, 0 represents non-pCR). Let the class center of class k be μk, and define the following three loss functions: (1) Intra-class compactness loss (Lintra): Makes the features of samples of the same class move closer to the class center of that class: Lintra=Σ{k∈{0,1}}[(1 / Nk)×Σ‖gdisc(hi)-μk‖2] (2) Inter-class separability loss (Linter): Maximizes the Euclidean distance between the class centers of pCR classes and non-pCR classes: Linter = -||μ0-μ1|2 (3) Consistency Loss (Lcons): Ensures that the feature representations of similar samples maintain structural consistency before and after data augmentation. Lcons=Σ{k∈{0,1}}[(1 / Nk)×Σ‖gcons(hi)-ck‖2] Where ck is the consistency center of category k.

[0036] The total loss function of the dual-tower attention network is: Ltotal=LCE+λintraLintra+λinterLinter+λconsLcons Where LCE is the cross-entropy classification loss; the weight hyperparameters are set as follows: λintra=0.1, λinter=0.05, λcons=0.05.

[0037] During the inference phase, the FQE module is removed, and a 512-dimensional feature vector is extracted from the penultimate layer of the network as a deep learning representation for each patient.

[0038] Step S6: Radiomics Feature Extraction Based on the tumor region of interest (ROI), hand-designed radiomics features were extracted from DCE and DWI images using the open-source PyRadiomics package. The extracted features included three categories: (1) Shape descriptor: describes the three-dimensional geometric shape of the tumor; (2) First-order statistics: describe the distribution characteristics of voxel intensity; (3) Texture features: including gray co-occurrence matrix (GLCM) features and gray run-length matrix (GLRLM) features, used to characterize the heterogeneity patterns inside the tumor.

[0039] Feature extraction is standardized through voxel resampling and intensity normalization to reduce variability between different scanners.

[0040] Step S7: Calculation of differential radiomics characteristics With pre- and post-treatment images available, differential radiomics features (Δ = post-treatment features - pre-treatment features) are calculated to characterize longitudinal changes in tumor phenotype induced by neoadjuvant chemotherapy and improve the sensitivity to treatment response.

[0041] Step S8: Extraction of Clinical Information Clinical variables (age, T stage, ER / PR / HER2 status, Ki-67 index, neoadjuvant chemotherapy regimen, etc.) are structured and encoded to form clinical feature vectors, which are then integrated with imaging-related features for modeling.

[0042] Step S9: Feature Filtering and Fusion To prevent information leakage, all preprocessing and feature selection steps are strictly confined to the development queue. Z-score normalization parameters (mean and standard deviation) are calculated only using the development queue and then applied to the external validation queue.

[0043] The standardization, redundancy removal, and screening fusion of deep features, radiomics features, differential radiomics features, and clinical features include the following sub-steps: (1) Variance threshold filtering: Features with variances below the 75th percentile of the total variance are removed to eliminate non-informative predictors; (2) ANOVA F test: Calculate the F statistic of the remaining features to measure the linear dependence between the features and the target variable, and select the most significant feature; (3) Correlation filtering: Calculate the Pearson correlation coefficient between feature pairs. For feature pairs with |r|>0.85, remove those with lower F-statistics to minimize multicollinearity; (4) Integrated voting mechanism (triggered when the number of candidate features exceeds 60): The importance of features is independently ranked using two algorithms: CatBoost (gradient boosting based on oblivious decision trees) and ANOVA F test. The final comprehensive importance score of each feature is calculated by Borda counting voting aggregation. Stotal(fj)=RCatBoost(fj)+RANOVA(fj) Where R(fj) represents the reverse ranking position of feature fj in the corresponding algorithm. The feature with the highest score is retained.

[0044] Step S10: Logistic Regression Modeling and Output Prediction The final features obtained in step S9 are Z-score standardized and then input into an L2-regularized logistic regression classifier. The objective function optimized during training is: Minimize: Σlog(1+exp[-y i ⋅(w T ⋅x i +b)])+(1 / 2C)⋅‖w‖ 2 2 Where w is the coefficient vector, b is the intercept, and x... i Let be the multimodal feature vector of patient i, and C be the reciprocal of the regularization strength. Optimization is achieved through grid search (C∈{0.1,1.0,10.0}), while class weight balancing is combined to address slight label imbalance.

[0045] The model outputs the predicted probability of breast cancer patients achieving pCR, providing an auxiliary reference for preoperative efficacy assessment.

[0046] Example 3: Prediction System A multimodal MRI fusion-based system for predicting pathological complete remission after neoadjuvant chemotherapy in breast cancer includes: Data acquisition module: used to acquire multi-parameter magnetic resonance imaging data (including at least DCE and DWI sequences) and clinicopathological data (including T stage, ER / PR / HER2 status, Ki-67 index and neoadjuvant chemotherapy regimen) of breast cancer patients after neoadjuvant chemotherapy.

[0047] The preprocessing module is used to perform format standardization, coordinate redirection, spatial registration, resampling to a fixed voxel spacing, 3D region of interest segmentation, cropping to a preset network input size, and intensity normalization on multi-parameter magnetic resonance image data. During the training phase, it is also used to apply data augmentation operations such as random flipping, random rotation, and random intensity shift to the preprocessed images.

[0048] The depth feature extraction module is used to extract DCE depth features and DWI depth features from the preprocessed DCE and DWI sequences, respectively. This module is built on a 3D ResNet-18 backbone network.

[0049] The multimodal fusion and quality enhancement module is used to perform multimodal fusion of DCE and DWI deep features through a dual-tower attention network. It employs a feature quality enhancement mechanism to make the feature distribution of similar samples more compact and the feature spacing of dissimilar samples larger, outputting high-quality multimodal deep features. This module includes a dual-tower attention network and a feature quality enhancement submodule. The dual-tower attention network uses two parallel 3D ResNet branches to encode the DCE and DWI sequences respectively, and performs feature fusion through multimodal attention blocks. The feature quality enhancement submodule applies intra-class compactness constraints, inter-class separability constraints, and consistency constraints.

[0050] Radiomics Feature Extraction Module: Used to extract radiomics features from multi-parameter magnetic resonance imaging data, including shape descriptors, first-order statistics, and texture features (such as GLCM and GLRLM features).

[0051] The differential radiomics calculation module is used to calculate the difference between the radiomics features after treatment and the radiomics features before treatment, given pre-treatment imaging data, as the differential radiomics feature.

[0052] Clinical feature coding module: used to perform structured coding on clinical pathological data and generate clinical feature vectors.

[0053] Feature filtering and fusion module: Used to filter and fuse quality-enhanced multimodal deep features, radiomics features, differential radiomics features, and clinical feature vectors using variance threshold filtering, ANOVA F test, correlation filtering, and integrated voting mechanism to obtain the final multimodal feature set.

[0054] Prediction output module: This module takes the final multimodal feature set as input to the trained L2 regularized logistic regression model and outputs a prediction of whether breast cancer patients have achieved pathological complete remission after neoadjuvant chemotherapy.

[0055] Example 4: Model Evaluation and Validation To verify the effectiveness of the method of the present invention, a comprehensive performance evaluation was conducted on the development queue and three independent external verification queues.

[0056] Evaluation metrics: The model's discriminative ability was assessed using the area under the receiver operating characteristic (AUC), and the 95% confidence interval was estimated using the DeLong method. Paired DeLong tests were performed with reference to the model of this invention. To control for the phylogenetic error rate in six planned comparisons for each cohort, all p-values ​​were corrected using the Bonferroni-Holm procedure. Model calibration was evaluated by plotting the relationship between predicted pCR probabilities and actual pCR probabilities in the development cohort and each external validation cohort. Quantitative metrics included the Brier score, calibration slope, and calibration intercept, with the latter two being logistic regression estimates of the binary results using the logarithmically transformed predicted probabilities. A slope of 1.0 and an intercept of 0 indicate ideal calibration.

[0057] Model interpretability: Model interpretability analysis was conducted at three levels: (1) Grad-based class activation mapping (Grad-CAM) was used to locate the spatial attention distribution within the twin-tower attention network; (2) Feature attribution based on the absolute standardized coefficients of the final logistic regression model; (3) Based on point bicollinearity analysis, the association between image-derived features and pCR was evaluated, supplemented by nonparametric comparisons of key clinical biomarkers (such as Ki-67 and HER2) between response groups.

[0058] Evaluation Results: In the development cohort, the AUC of the proposed method (AIFPS) reached 0.882 (95% CI: 0.855–0.905), which was statistically significantly improved compared to all unimodal and bimodal models (corrected P<0.05). In external validation, the model maintained strong discriminative ability, achieving AUCs of 0.853 (95% CI: 0.778–0.915), 0.888 (95% CI: 0.846–0.926), and 0.895 (95% CI: 0.846–0.936) in validation cohorts 1, 2, and 3, respectively. Calibration evaluation showed that Brier scores were low and consistent (0.131, 0.141, 0.134 and 0.135 for the development cohort and the three validation cohorts, respectively), with calibration slopes close to 1 (range 0.897-1.197), indicating that it maintained stable probabilistic accuracy and risk ranking under cross-institutional distributed offset conditions.

[0059] Ablation experiments: To clearly evaluate the independent contribution of each network component to the overall prediction performance, ablation studies were conducted on three independent external validation cohorts. The baseline dual-tower ResNet-18 model (excluding FQE and MAB) had an AUC ranging from 0.794 to 0.825. Independent integration of MAB effectively captured the nonlinear complementary information between DCE and DWI sequences, resulting in a consistent AUC improvement. Independent application of the FQE module successfully regularized the latent representation geometry, increasing the AUC to 0.857 in validation cohort 2. The synergistic combination of the two modules achieved the highest performance across all external validation cohorts (AUCs of 0.853, 0.888, and 0.895, respectively), indicating that the simultaneous application of multimodal attention and feature quality constraints significantly enhances the discriminative power and cross-institutional robustness of deep learning representations.

[0060] Feature Importance Analysis: The distribution of feature importance weights for each modality in the final fusion model was analyzed (using the Kruskal-Wallis test). The results showed that radiomics, deep learning, and clinical data streams all made substantial contributions to the final prediction, with no single modality statistically overwhelming the others (Kruskal-Wallis H=1.28, p=0.529). This balanced contribution supports the rationale for integrating all available modalities rather than selecting a single dominant data source. The highest-ranking feature was the difference DCE texture metric (importance value 4.13), followed by another difference DCE gray-level co-occurrence matrix feature (importance 2.92) and post-treatment DCE wavelet texture feature (importance 2.84). Deep learning features were also consistently among the top contributors. Ki-67 was one of the most influential clinical predictors (importance 1.60), consistent with known clinicopathological factors related to treatment response.

[0061] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for predicting pCR in breast cancer using multimodal MRI fusion, characterized in that, Includes the following steps: S1: Collect multi-parameter magnetic resonance imaging data and clinicopathological data of breast cancer patients after neoadjuvant chemotherapy. The multi-parameter magnetic resonance imaging data includes at least dynamic contrast-enhanced sequences and diffusion-weighted sequences. S2: Preprocess the multi-parameter magnetic resonance imaging data, including format normalization, spatial alignment / registration, resampling, and intensity normalization; S3: Input the preprocessed dynamic contrast enhancement sequence and diffusion weighted sequence into the deep feature extraction network to extract dynamic contrast enhancement deep features and diffusion weighted deep features, respectively. S4: Multimodal fusion is performed on the dynamic contrast-enhanced deep features and diffusion-weighted deep features, and a feature quality enhancement mechanism is introduced during the training process to make the feature distribution of similar samples more compact and the feature interval of dissimilar samples larger, so as to obtain quality-enhanced multimodal deep features. S5: Extract radiomics features from the multi-parameter magnetic resonance imaging data, including shape descriptors, first-order statistics, and texture features; S6: Given pre-treatment and post-treatment imaging data, calculate the difference radiomics feature, which is the difference between the post-treatment radiomics feature and the pre-treatment radiomics feature; S7: Perform structured coding on the clinicopathological data to generate clinical feature vectors; S8: The enhanced multimodal deep features, radiomics features, differential radiomics features, and clinical feature vectors are standardized, redundantly removed, and screened and fused to obtain the final multimodal feature set; S9: Input the final multimodal feature set into the trained classification model and output the prediction result of whether the breast cancer patient has achieved pathological complete remission after neoadjuvant chemotherapy.

2. The method for predicting pCR in breast cancer using multimodal MRI fusion according to claim 1, characterized in that: In step S4, the feature quality enhancement mechanism is implemented through the following three constraints: Intra-class compactness constraint is used to make the features of samples of the same class move closer to the class center of that class; Inter-class separability constraints are used to maximize the Euclidean distance between class centers of different classes. Consistency constraints are used to maintain the consistency of feature representations of similar samples before and after data augmentation.

3. The method for predicting pCR in breast cancer using multimodal MRI fusion according to claim 1, characterized in that: In step S4, the multimodal fusion employs a dual-tower attention network. The dynamic contrast enhancement sequence and the diffusion weighted sequence are encoded by two parallel three-dimensional ResNet branches, and the nonlinear complementary information between the two sequences is captured through a multimodal attention block.

4. The method for predicting pCR in breast cancer using multimodal MRI fusion according to claim 1, characterized in that: In step S8, the screening includes: performing variance threshold filtering on features, univariate feature selection based on F statistic and correlation filtering, and eliminating redundant features; When the number of candidate features exceeds a preset threshold, an integrated voting mechanism is used for final feature selection.

5. The method for predicting pCR in breast cancer using multimodal MRI fusion according to claim 1, characterized in that: In step S9, the classification model is an L2-regularized logistic regression model.

6. The method for predicting pCR in breast cancer using multimodal MRI fusion according to claim 1, characterized in that: In step S1, the clinicopathological data includes T stage, estrogen receptor status, progesterone receptor status, HER2 status, Ki-67 index, and neoadjuvant chemotherapy regimen.

7. The method for predicting pCR in breast cancer using multimodal MRI fusion according to claim 1, characterized in that: In step S5, the texture features include gray-level co-occurrence matrix features and gray-level run-length matrix features.

8. A multimodal MRI fusion-based pCR prediction system for breast cancer, characterized in that: include: The data acquisition module is used to acquire multi-parameter magnetic resonance imaging data and clinicopathological data of breast cancer patients after neoadjuvant chemotherapy. The multi-parameter magnetic resonance imaging data includes at least dynamic contrast-enhanced sequences and diffusion-weighted sequences. The preprocessing module is used to perform format standardization, spatial alignment / registration, resampling, and intensity normalization on the multi-parameter magnetic resonance imaging data. The deep feature extraction module is used to extract dynamic contrast-enhanced deep features and diffusion-weighted deep features from the preprocessed dynamic contrast-enhanced sequence and diffusion-weighted sequence, respectively. The multimodal fusion and quality enhancement module is used to perform multimodal fusion on the dynamic contrast-enhanced depth features and diffusion-weighted depth features, and to make the feature distribution of similar samples more compact and the feature interval of dissimilar samples larger through the feature quality enhancement mechanism, and output the quality-enhanced multimodal depth features. A radiomics feature extraction module is used to extract radiomics features from the multi-parameter magnetic resonance imaging data; The differential radiomics calculation module is used to calculate the difference between the radiomics characteristics after treatment and the radiomics characteristics before treatment, given pre-treatment imaging data, as the differential radiomics feature. The clinical feature encoding module is used to perform structured encoding on the clinicopathological data and generate clinical feature vectors. The feature filtering and fusion module is used to filter and fuse the quality-enhanced multimodal deep features, radiomics features, differential radiomics features, and clinical feature vectors to obtain the final multimodal feature set. The prediction output module is used to input the final multimodal feature set into the trained classification model and output the prediction result of whether the breast cancer patient has achieved pathological complete remission after neoadjuvant chemotherapy.

9. The multimodal MRI fusion-based breast cancer pCR prediction system according to claim 8, characterized in that: The multimodal fusion and quality enhancement module includes a dual-tower attention network and a feature quality enhancement submodule. The dual-tower attention network uses two parallel 3D ResNet branches to encode the dynamic contrast enhancement sequence and the diffusion weighted sequence, respectively, and performs feature fusion through multimodal attention blocks. The feature quality enhancement submodule is used to apply intra-class compactness constraints, inter-class separability constraints, and consistency constraints.

10. The multimodal MRI fusion-based breast cancer pCR prediction system according to claim 8, characterized in that: The classification model is an L2-regularized logistic regression model.

Images