Breast cancer treatment response prediction system based on multi-time-sequence images

Abstract

The invention relates to the technical field of medical image analysis, and discloses a breast cancer treatment response prediction system based on multi-time-sequence images. According to the method, preoperative MRI, clinical pathology data and RNA-seq data meeting quality control standards are obtained through dynamic case screening, traditional image features, multi-scale Hessian matrix blood vessel-structure features and topological persistent coherent curvature features are extracted through multi-plane segmentation, and key features are obtained through three-level screening. A multi-factor logistic model fusing radiomics, clinical features and immune scoring is further constructed, a pCR probability value and a feature contribution heat map are output, and MRI data quality control, real-time analysis and interactive report generation are achieved through a matched computer system. The method breaks through the limitation of single modal analysis, dynamically associates the image features with the molecular mechanism, and provides interpretable decision support for precision medical treatment.

Description

Technical Field

[0001] The present invention relates to the field of medical imaging technology, and particularly to a breast cancer treatment response prediction system based on multi-temporal imaging images. Background Art

[0002] In recent years, the widespread application of technologies including mammography, ultrasound, MRI, etc. has provided a non-invasive and relatively accurate method for predicting the NAT of breast cancer, among which the application of MRI is the most common. However, due to subjective differences in manual film reading and a large workload of film reading, the accuracy and efficiency of manual diagnosis of MRI are not ideal. At the same time, the deep image features of MRI can more accurately predict the efficacy of NAT, which are also unrecognizable by the naked eye. Existing research mainly relies on the MRI images of patients before neoadjuvant treatment or the images before surgery to predict the treatment outcome. However, according to the latest literature statistics, during the period from January 2021 to December 2024, only five AI-based studies have explored the application of longitudinal MRI images in predicting the outcome of NAC. These studies have considered the importance of longitudinal data to a certain extent, but there are still the following deficiencies: lack of sufficient modeling of the tumor interactivity and dynamic characteristics during the treatment process. Analyzing only single or limited numbers of time point data is difficult to fully reflect the evolution process of the tumor at different stages, the model has insufficient biological interpretability, and some models can give prediction results, but lack corresponding analysis of the biological mechanisms behind the prediction, making it difficult to gain full trust and promotion in clinical practice. Summary of the Invention

[0003] To solve the technical problem of the disconnection between traditional breast imaging examinations and potential pathophysiology in the prior art, the present application provides a breast cancer treatment response prediction system based on multi-temporal imaging images, including: a data preprocessing module for standardizing multi-temporal breast MRI images, including image registration, histogram normalization, and frequency domain amplitude spectrum replacement operations; a sub-region segmentation module that uses a three-dimensional spatio-temporal network to perform multi-level sub-region segmentation on the tumor and the peritumoral region, and the network includes a residual U-Net architecture and a spatio-temporal transformer; a feature extraction module that constructs a siamese network structure based on a pre-trained medical imaging model to process the original images and sub-region segmentation maps of T0 and T1 respectively; a spatio-temporal fusion module that integrates a spatial attention mechanism and a temporal self-attention mechanism to achieve multi-temporal feature fusion through differential feature comparison and the Transformer architecture; a prediction output module that generates the treatment response probability and visualizes the key spatio-temporal feature regions.

[0004] In one embodiment, the three-dimensional spatio-temporal network includes: a downsampling and upsampling module that preserves detailed information through skip connections between layers; a spatio-temporal converter composed of a spatial converter and a temporal converter, which respectively process single-phase spatial features and cross-stage dynamic changes; an adaptive weight mechanism is used to process missing time-series data, and the feature weights are dynamically adjusted through an attention mask.

[0005] In one embodiment, the siamese network structure specifically includes: dual input channels that respectively receive the original MRI image and the sub-region segmentation map; a parallel convolution branch includes depthwise separable convolution and a channel attention module; the feature concatenation layer adopts a cross-modal fusion strategy and performs channel dimension concatenation at the stage3 of ResNet-50.

[0006] In one embodiment, the spatio-temporal fusion module includes: a differential feature calculation layer that aligns T0 / T1 feature maps through a registration algorithm; a time interval embedding layer that converts the treatment cycle into a position encoding; a multi-head attention mechanism includes 8 attention heads and calculates the correlation matrix of spatio-temporal features.

[0007] In one embodiment, the pre-trained model adopts a contrastive learning strategy: constructing a contrastive learning dataset for breast MRI, using a momentum contrast framework for parameter update, and separating anatomical features and device-related features according to a feature decoupling loss function.

[0008] In one embodiment, the system further includes an interpretability module: gradient-weighted class activation mapping visualizes key regions, transcriptomics matching analysis identifies pCR-related molecular pathways, and attention weight heatmaps show key decision points in the time dimension.

[0009] The present invention also provides an embodiment, a method for predicting breast cancer treatment response based on any one of the breast cancer treatment response prediction systems based on multi-temporal imaging images, including: obtaining multi-temporal DCE-MRI images and clinical data of a patient; performing frequency-domain amplitude spectrum normalization and tumor sub-region segmentation; extracting spatio-temporal features and calculating the difference matrix before and after treatment; generating a prediction probability through time attention weighting; outputting a pCR status prediction and a molecular pathway analysis report.

[0010] In one embodiment, the frequency-domain amplitude spectrum normalization specifically includes: calculating the average amplitude spectrum of the training set by grouping according to tumor size; replacing the amplitude spectrum after performing Fourier transform on the test set images; retaining the phase information for inverse transformation to reconstruct the normalized images.

[0011] The present invention also provides an embodiment, an electronic device, including a memory, a processor, and a computer program stored on the memory, and when the processor executes the program, it implements the steps of the method for predicting breast cancer treatment response based on any one of the above-mentioned multi-temporal imaging images.

[0012] The present invention also provides an embodiment, a computer-readable storage medium storing a computer program, and when the program is executed by a processor, it implements the steps of any one of the breast cancer treatment response prediction methods based on multi-temporal image images.

[0013] Beneficial effects

[0014] The present invention provides a breast cancer treatment response prediction system based on multi-temporal image images, which comprehensively improves the prediction efficiency of breast cancer treatment response through multi-dimensional technological innovation. The system integrates a three-dimensional spatio-temporal segmentation network and a dynamic attention mechanism. In terms of precise modeling, it adopts a hybrid architecture of a residual U-Net and a spatio-temporal transformer, combined with a differential feature contrast algorithm, which can capture the key volume change rate of treatment. The spatio-temporal joint modeling improves the prediction stability compared with traditional methods. For the problem of multi-center application, the innovative frequency-domain amplitude spectrum normalization technology separates and replaces device-related features through Fourier transform. In terms of interpretability, the system innovatively integrates image features and molecular pathway analysis. The visualization shows that the model decision region highly coincides with the pathological cell distribution. The time attention curve accurately locates 14 - 21 days as the key observation window, and the pathway enrichment analysis reveals a strong correlation between the activation of IFN-γ signal and the prediction result, providing a new direction for biomarker discovery. On this basis, through the optimization of depthwise separable convolution and dynamic pruning technology, the processing time for the full process of a single case is compressed from the traditional 15 minutes to 3 minutes. The model has a small video memory occupancy and can run in real time on conventional GPU devices. Verified by 327 clinical cases in 5 domestic tertiary hospitals, the system reduces the misjudgment rate of pCR, helps doctors determine the treatment plan in advance on average, and its accuracy, universality, and clinical applicability provide breakthrough technical support for the individualized treatment of breast cancer. Brief description of the drawings

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on the structures shown in these drawings.

[0016] Figure 1 It is a block diagram of a breast cancer treatment response prediction system based on multi-temporal image images provided by the first embodiment of the present invention;

[0017] Figure 2 It is an AUC graph of a multi-dimensional radiomics model provided by the first embodiment of the present invention;

[0018] Figure 3 It is a GSEA pathway enrichment graph provided by another embodiment of the present invention;

[0019] Figure 4 Working step diagram of the breast cancer treatment response prediction method based on multi-temporal image images provided by another embodiment of the present invention;

[0020] Figure 5 Visual display diagram of the reference case provided by the embodiment of the present invention. Detailed implementation manners

[0021] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0022] It should be noted that if there are directional indications (such as up, down, left, right, front, back...) involved in the embodiments of the present invention, the directional indications are only used to explain the relative positional relationship and movement conditions between components in a specific posture. If the specific posture changes, the directional indications will also change accordingly.

[0023] In addition, if there are descriptions such as "first", "second", etc. involved in the embodiments of the present invention, the descriptions of "first", "second", etc. are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include at least one of such features. In addition, if "and / or" or "and / or" appears throughout the text, its meaning includes three parallel solutions. Taking "A and / or B" as an example, it includes solution A, solution B, or a solution that satisfies both A and B at the same time. In addition, the technical solutions between the various embodiments can be combined with each other, but it must be based on the fact that those of ordinary skill in the art can implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the protection scope required by the present invention.

[0024] Embodiment 1

[0025] Reference Figures 1 - 3, A breast cancer treatment response prediction system based on multi-temporal image images, comprising: a data preprocessing module for standardizing multi-temporal breast MRI images, including image registration, histogram standardization, and frequency-domain amplitude spectrum replacement operations; a sub-region segmentation module that uses a three-dimensional spatio-temporal network to perform multi-level sub-region segmentation on the tumor and peritumoral regions, and the network includes a residual U-Net architecture and a spatio-temporal transformer; a feature extraction module that constructs a siamese network structure based on a pre-trained medical image model to process the original images and sub-region segmentation maps of T0 and T1 respectively; a spatio-temporal fusion module that integrates a spatial attention mechanism and a temporal self-attention mechanism to achieve multi-temporal feature fusion through differential feature comparison and a Transformer architecture; a prediction output module that generates treatment response probabilities and visualizes key spatio-temporal feature regions.

[0026] First, a patient screening step is required. The following are the inclusion criteria for screening: Inclusion criteria: Female patients, aged over 18 years; The patient has been pathologically and histologically diagnosed with invasive breast cancer without distant organ metastasis; The patient has undergone tumor surgical resection and axillary lymph node dissection; Breast magnetic resonance examination has been performed.

[0027] Exclusion criteria: Lack of MRI data (T0) before and / or after NAT (T1): Pre-NAC refers to MRI examination within 2 weeks before the start of NAC; T1: Early NAC refers to MRI examination within 1 - 2 cycles after the completion of NAC); Poor MRI quality or missing DCE-MRI images; No surgical results after NAT surgery.

[0028] According to the above criteria, data such as magnetic resonance images and clinical characteristics of cases meeting the inclusion criteria are collected. Collect breast MR imaging of patients using a 1.5T or 3.0T field strength magnet and the ACRIN 6698 database. Within 1 month after the completion of NAT, the pCR status of each target tumor is determined according to the surgical pathological results. pCR is defined as the absence of invasive cancer (ypT0 / is, ypN0) in the breast and axillary lymph nodes at the time of surgery, allowing for the presence of residual ductal carcinoma in situ (DCIS) (ypTis). For multi-center cohorts, a standardized pCR definition is used.

[0029] The content of clinical feature information collection includes: age at diagnosis, number of tumors, tumor size, pathological type, clinical tumor stage, clinical lymph node stage, ER status, PR status, HER2 status, molecular subtype, surgical method, and chemotherapy regimen. A three-dimensional spatio-temporal network for breast tumor segmentation was developed using the PyTorch framework. The model takes the original DCE-MRI images and subtracted images as inputs, aiming to capture the dynamic changes of tumors at different stages. The model adopts a U-Net architecture with residual blocks, consisting of 4 downsampling and upsampling modules, and uses skip connections to preserve detailed information. The spatio-temporal transformer consists of a spatial transformer and a temporal transformer, which simulate the spatial features of a single phase and the dynamic changes at different stages respectively. The missing stages are automatically processed through the attention mechanism, enhancing the robustness of the model. The Adam optimizer was used during the training process. The initial learning rate of the whole-breast segmentation model was 0.002, and that of the tumor segmentation model was 0.001. The learning rate was halved every 50 epochs, and the model was trained for a total of 300 epochs with a batch size of 2. The training process included more than 200,000 iterations to optimize resource usage and improve learning efficiency. The code of the model is publicly available on GitHub. Subsequently, two experienced radiologists used ITK-SNAP software to review the automatically segmented tumor ROIs of multi-center data, and resolved the differences through consensus. The Dice similarity coefficient was calculated to evaluate the similarity between the automatic segmentation of the model and the ROIs adjusted by the radiologists.

[0030] Before analyzing the MRI images, standardized preprocessing operations were first performed. Specifically, considering the differences in intensity distribution and data type between the source file and the label file, in this embodiment, resampling strategies using nearest neighbor interpolation and linear interpolation were respectively adopted, so as to ensure the resampling quality of the source file while avoiding distortion of the binary label data after interpolation. In addition, to improve the image contrast and quality and reduce the intensity differences between different centers, the present invention introduced a histogram normalization method to perform intensity calibration on the MRI images of each center. Subsequently, image registration was performed through affine transformation combined with mutual information metric, achieving effective alignment under different imaging modalities or different scanning conditions, and improving the accuracy of image comparison and analysis. It is worth mentioning that according to specific requirements, the present invention respectively carried out different strategies based on the whole image and the tumor region during the registration process: the former can provide a more complete image view for subsequent deep learning and attention mechanisms; the latter reduces the attention range by only focusing on the tumor and the surrounding area, and obtains more accurate results in the subsequent spectrum replacement link. Due to the differences in scanning equipment and protocols among multi-center data, to further eliminate the impact of these differences on the model, the present invention performed additional processing on the multi-center images. After performing Fourier transform on the images, the amplitude spectrum was used as the core index for clustering, and the tumors in the training set were grouped according to three sizes: large, medium, and small, and the average amplitude spectrum of each group was statistically calculated. Subsequently, for the images of the same size category in the test set, the present invention replaced their amplitude spectra with the average amplitude spectra of the corresponding categories in the training set, so that the images of different centers are more consistent in frequency domain distribution, thereby reducing the intensity changes caused by inconsistent scanning conditions. After the above preprocessing and amplitude spectrum replacement, the breast MRI data has been effectively improved in terms of space, intensity, and cross-center consistency, laying a stable foundation for the subsequent training and inference of the model.

[0031] When establishing the MRI-STI model, the working steps include that the model processes longitudinal multi-phase MRI scans at the pre-NAC (T0) and early NAC (T1) stages, combines whole-tumor and sub-region segmentation as well as spatial and temporal attention mechanisms to predict the NAC response. For internal and external validation, the model is trained and internally validated on the PC (GZ and LZ datasets) and EVC, and validated using three independent external cohorts (DG, YT, YN datasets), followed by PVC (GZ prospective dataset). The performance metrics include AUC and AP to evaluate the model efficacy. For prognosis prediction, the correlation between the model prediction score and recurrence-free survival (RFS) and overall survival (OS) is evaluated in the ACRIN 6657 / I-SPY1 dataset. The biological basis of the MRI-STI model is pathologically validated using the histopathological data of the main cohort and molecularly validated through the transcriptome analysis of ACRIN 6698 / I-SPY2 to explore the related biological mechanisms. Note: PC: main cohort; EVC: external validation cohort; PVC: prospective validation cohort; RFS: recurrence-free survival; OS: overall survival.

[0032] It is developed based on the medical large-scale pre-trained model MedicalNet. First, large-scale contrastive learning pre-training is carried out using the publicly available breast MRI dataset. More than 2,000 breast imaging samples are collected and continuously trained for more than 100 epochs on this data. In this way, the model learns rich general features in a wider image distribution, providing a more robust initialization for the subsequent specific data of the present invention. The pre-trained model is finally used as a feature extractor and directly embedded into the overall framework, laying a solid foundation for the construction of the core network.

[0033] On this basis, the overall architecture of the siamese network is constructed in this embodiment to respectively extract and compare the features of two images of the patient before (T0) and after (T1) treatment. The network sets two input heads at the input layer: one input head receives the preprocessed and cropped original image, and the other input head receives the corresponding sub-region segmentation map. By respectively extracting the features of the original image and the sub-region image and splicing them in the specified dimension, the model can simultaneously learn the macroscopic information of the whole tumor and the peritumoral region and the fine spatial information of the sub-region, so as to obtain a more differential feature expression.

[0034] For single-timepoint images, a spatial attention mechanism is introduced inside the model to identify the most critical parts for prediction in 3D images. Since breast images may vary significantly in size and shape, simple global attention alone is difficult to cover all important lesions. Therefore, we combine sub-region segmentation cues to help the model focus on specific key regions and form overall features by stitching together multiple local features. Subsequently, the model registers and calculates the differences between the features of T0 and T1 to obtain temporal change information. During fusion, temporal self-attention is also introduced through the multi-head attention mechanism of the Transformer architecture, combining the time-axis information (i.e., the treatment interval between the two imaging sessions) with the differential features, enabling the model to dynamically focus on the spatio-temporal feature combinations that have the most impact on prediction. During the model training process, a strategy of gradually unfreezing layers is adopted to fully utilize the feature advantages of the pre-trained model and avoid overfitting. Specifically, in the initial stage, only the network parameters of the few layers closest to the output end are fine-tuned, while the weights of the pre-trained layers are kept mostly frozen; as training progresses, the more front network layers are gradually unfrozen to allow deeper adaptation and fusion of the deep features with the data of this task. In this way, the general features learned from pre-training can be stably retained, and the model can gradually adapt to the new feature distribution in the later stage, thus achieving a balance between fast convergence and high-precision prediction. At the same time, it avoids the model from quickly fitting on the training machine, achieving the effect of both taking into account accuracy and ensuring generalization.

[0035] The system is based on the medical pre-trained model MedicalNet for secondary development. Through a contrastive learning strategy, transfer learning is carried out on a large breast imaging dataset, enabling the model to have the ability to extract general imaging features. On this basis, a dual-branch siamese network structure is designed to process the temporal imaging data before treatment (T0) and during treatment (T1) respectively. Each branch contains two parallel input channels: the main channel receives the preprocessed original three-dimensional tumor image, and the auxiliary channel inputs the corresponding sub-region segmentation atlas to achieve the collaborative extraction of overall features and local features.

[0036] A multi-level attention mechanism is integrated inside the network to enhance the feature expression ability. In the spatial dimension, the system combines deformable convolution with a region cue module, enabling the model to dynamically focus on the heterogeneous regions inside the tumor and changes in the peritumoral microenvironment. By constructing a hierarchical spatial attention map, the network can identify key morphological changes such as tumor margin infiltration and internal necrosis. In the time dimension, the system introduces a temporal difference module to calculate the dynamic changes between the feature maps of the T0 and T1 stages, and embeds the time interval parameter into the Transformer encoder to establish a time decay weighting mechanism. This design enables the model to automatically evaluate the importance of feature changes at different treatment stages, such as the association patterns between early treatment responses and late-stage drug resistance development.

[0037] The training process of the model optimizes parameter learning using a progressive thawing strategy. In the initial stage, the basic convolutional layers of the pre-trained model are frozen, and only the high-level network and attention module are fine-tuned to retain the general feature extraction ability. As the training iteration progresses, the underlying network layers are gradually thawed to enable the model to gradually adapt to the tumor spatio-temporal evolution pattern unique to this task. The optimization process uses an adaptive learning rate adjustment algorithm and combines gradient clipping techniques to prevent overfitting and ensure stable convergence under limited sample conditions. To verify the prediction performance of the system, the study adopted a multi-center cross-validation scheme. The data from the main center was divided into a training set and an internal validation set, and at the same time, the datasets of four independent external validation centers were introduced. After all the data passed through a unified preprocessing process, they were input into the model to evaluate the generalization ability of the system under different populations and device environments. The prediction results were compared and analyzed with the gold standard of postoperative pathological evaluation, and the prediction efficacy of the model for pathological complete remission (pCR) was calculated. The system demonstrated excellent discriminative performance on the internal validation set, and its prediction results were highly consistent with the pathological evaluation. The external validation results further confirmed that the model maintained stable performance in the image data obtained from different regions and different scanning devices, proving the effectiveness of the frequency domain replacement strategy and the multi-center preprocessing scheme. Compared with the existing single-timepoint prediction models, this system significantly improved the recognition sensitivity of early treatment response by integrating multi-temporal dynamic features, especially being able to accurately distinguish the advantageous population that may achieve pCR at the initial stage of treatment. To achieve the interpretability of the model prediction results, in this embodiment, deep features and transcriptomic data were jointly analyzed. Through gene set enrichment analysis (GSEA), it was found that in the cases predicted by the model to be pCR positive, anti-tumor immune-related pathways such as the T cell receptor signaling pathway and interferon-γ-mediated immune response were significantly activated. On the contrary, in the predicted drug-resistant group, tumor adaptation pathways such as RNA processing modification and epigenetic regulation were overly active. These findings highly coincided with the biological characteristics such as the degree of immune cell infiltration and tumor heterogeneity in the tumor microenvironment, confirming that the image features extracted by the model have clear biological significance. This system achieved dynamic monitoring and early warning of treatment response. Through visualizing the spatio-temporal attention weight map, clinicians can intuitively observe the image evolution patterns such as the weakening of enhancement in the tumor core area and the reduction of the peritumoral edema range during the treatment process, and these features showed a significant correlation with the degree of pathological remission. In addition, the changing trend of the sub-region features output by the system provided a quantitative basis for the adjustment of individualized treatment strategies. For example, for tumor sub-regions with persistent metabolic activity, local intensified radiotherapy or targeted therapy could be recommended.

[0038] In the specific implementation of this embodiment, clinical institutions can upload patients' multi-time-point MRI data to the system platform through a standardized interface. The preprocessing module automatically completes image registration, intensity standardization, and tumor region segmentation, generating structured data containing the original images and sub-region atlases. The feature extraction engine calls the pre-trained model parameters to calculate the spatio-temporal feature vectors at the T0 and T1 stages respectively, and quantifies the dynamic change patterns related to treatment response through differential operations. The prediction module synthesizes spatio-temporal attention weights and clinical parameters to generate individualized treatment response probabilities and visual interpretation reports. The entire processing flow realizes end-to-end automation, and the generation of prediction reports can be completed within a clinically acceptable time from the input of raw data. The system supports seamless docking with the hospital PACS system, and the prediction results can be directly embedded in the electronic medical record system to provide real-time decision support for the multi-disciplinary treatment team. For special cases, the system provides auxiliary analysis tools such as attention heat maps and feature evolution curves to help clinicians deeply understand the basis for model decisions and achieve the organic combination of artificial intelligence and clinical experience.

[0039] Embodiment 2

[0040] Reference Figure 4 , the present invention also provides an embodiment, a breast cancer treatment response prediction method based on any one of the breast cancer treatment response prediction systems based on multi-temporal imaging images, including: obtaining patients' multi-temporal DCE-MRI images and clinical data; performing frequency-domain amplitude spectrum standardization and tumor sub-region segmentation; extracting spatio-temporal features and calculating the difference matrix before and after treatment; generating prediction probabilities through time attention weighting; outputting pCR status prediction and molecular pathway analysis reports.

[0041] In one of the embodiments, the frequency-domain amplitude spectrum standardization specifically includes: calculating the average amplitude spectrum of the training set by grouping according to tumor size; replacing the amplitude spectrum after performing Fourier transform on the test set images; retaining the phase information for inverse transformation to reconstruct the standardized images.

[0042] In the specific implementation of this embodiment, 323 patients with pathologically confirmed invasive breast cancer were included and divided into a training set and an internal validation set according to a ratio of 4:1. Subsequently, 95, 131, 83, and 64 cases of data from other centers were included;

[0043] Perform the above-mentioned preprocessing on the image and 3D cropping to retain key information such as the tumor region and sub-regions within the tumor. Through the above model construction method, input the main cohort images and finally construct a new adjuvant chemotherapy efficacy prediction model based on multi-temporal spatio-temporal fusion. The AUC of the main sequence is 0.981, and the external AUCs are 0.923, 0.892, and 0.913 respectively; risk stratification is performed using the optimal cut-off value determined according to the logistics model prediction. GSEA shows that immune-related pathways are significantly up-regulated in the pCR tumors predicted by the model, including PD-1 signaling, interferon-γ signaling, and complement activation( Figure 2 a). Conversely, pathways related to tumor progression and drug resistance, such as epigenetic regulation and RNA processing, are down-regulated( Figure 2 b). Further analysis of the differentially expressed genes (DEGs) found that CD3D (p = 0.0054) and CD8A (p = 0.0019) were significantly overexpressed in pCR tumors, indicating a stronger immune response in tumors that are more likely to respond to NAC. Correlation analysis further confirmed that the features extracted by deep learning are closely related to immune-related pathways, emphasizing that the model effectively captures tumor-immune interactions that affect chemotherapy outcomes.

[0044] Reference Figure 5 , Visualization of two cases. Part a in the figure shows that patient A achieved pathological complete remission (pCR), and part b in the figure shows that patient B did not achieve pCR after neoadjuvant chemotherapy (NAC). For each patient, each column shows the following information: dynamic contrast-enhanced magnetic resonance imaging (DCE-MRI) images before (NAC-T0) and after (NAC-T1) treatment, heat maps showing the spatial distribution of tumor features in the whole tumor and three regions (WT, R1, R2, R3), and the reconstructed tumor volume over time and the corresponding heat maps showing the changes. The STI model correctly predicted the treatment outcomes of both patients. Patient A was diagnosed with pCR, and patient B was diagnosed with non-pCR, as shown by the prediction scores (0.984 and 0.019 respectively). Note: WT: heat map of the whole tumor; R1, R2, and R3: heat maps of the three sub-regions.

[0045] The present invention also provides an embodiment, an electronic device, including a memory, a processor, and a computer program stored on the memory. When the processor executes the program, it implements the steps of any of the breast cancer treatment response prediction methods based on multi-temporal image images.

[0046] The present invention also provides an embodiment, a computer-readable storage medium, storing a computer program, and when the program is executed by a processor, it implements the steps of any of the breast cancer treatment response prediction methods based on multi-temporal image images.

[0047] In summary, the present invention provides a breast cancer treatment response prediction system based on multi-temporal image images, through a siamese network architecture that fuses multi-temporal breast images and differential features, and an interpretability enhancement design that combines spatial and temporal attention mechanisms. In the network, feature extraction and contrast of differences are performed on the overall and sub-region images respectively, and time interval information is embedded in the differential features to improve the early prediction ability of breast cancer treatment response and the clinical interpretability of the model.

[0048] The above are only the preferred embodiments of the present invention, and do not limit the patent scope of the present invention. Any equivalent structural transformation made under the inventive concept of the present invention by using the content of the specification and drawings of the present invention, or direct / indirect application in other related technical fields, is included in the patent protection scope of the present invention.

Claims

1. A breast cancer treatment response prediction system based on multi-temporal image images, characterized in that, Including: A data preprocessing module for normalizing multi-temporal breast MRI images, including image registration, histogram normalization, and frequency-domain amplitude spectrum replacement operations; A sub-region segmentation module that uses a three-dimensional spatio-temporal network to perform multi-level sub-region segmentation on tumors and peritumoral regions, including a residual U-Net architecture and a spatio-temporal transformer; A feature extraction module that constructs a siamese network structure based on a pre-trained medical image model to process the original images and sub-region segmentation maps of T0 and T1 respectively; A spatio-temporal fusion module that integrates a spatial attention mechanism and a temporal self-attention mechanism to achieve multi-temporal feature fusion through differential feature comparison and a Transformer architecture; A prediction output module that generates treatment response probabilities and visualizes key spatio-temporal feature regions.

2. The breast cancer treatment response prediction system based on multi-temporal image images according to claim 1, wherein The three-dimensional spatio-temporal network includes: a downsampling and upsampling module that retains detailed information through skip connections between layers; the spatio-temporal transformer consists of a spatial transformer and a temporal transformer, which process single-phase spatial features and cross-stage dynamic changes respectively; an adaptive weight mechanism is used to process missing temporal data, and the feature weights are dynamically adjusted through an attention mask.

3. The breast cancer treatment response prediction system based on multi-temporal image images according to claim 1, characterized in that The siamese network structure specifically includes: dual input channels that receive the original MRI image and the sub-region segmentation map respectively; parallel convolutional branches include depthwise separable convolutions and channel attention modules; the feature concatenation layer adopts a cross-modal fusion strategy to perform channel dimension concatenation at the stage3 of ResNet-50.

4. The breast cancer treatment response prediction system based on multi-temporal image images according to claim 1, characterized in that, The spatio-temporal fusion module includes: a differential feature calculation layer that aligns feature maps through a registration algorithm; a time interval embedding layer that converts the treatment cycle into a position encoding; a multi-head attention mechanism includes 8 attention heads to calculate the correlation matrix of spatio-temporal features.

5. The breast cancer treatment response prediction system based on multi-temporal image images according to claim 1, characterized in that, The pre-trained model adopts a contrastive learning strategy: constructing a contrastive learning dataset for breast MRI, using a momentum contrast framework for parameter update, and separating anatomical features and device-related features according to a feature decoupling loss function.

6. The breast cancer treatment response prediction system based on multi-temporal image images according to claim 1, characterized in that, It also includes an interpretability module: gradient-weighted class activation mapping visualizes key regions, transcriptomics matching analysis identifies pCR-related molecular pathways, and attention weight heatmaps show key decision points in the time dimension.

7. A breast cancer treatment response prediction method based on multi-temporal image, applied to the breast cancer treatment response prediction system based on multi-temporal image according to claims 1-6, characterized in that Including: Obtaining multi-temporal DCE-MRI images and clinical data of patients; performing frequency-domain amplitude spectrum normalization and tumor sub-region segmentation; extracting spatio-temporal features and calculating the difference matrix before and after treatment; generating prediction probabilities through temporal attention weighting; outputting a pCR status prediction and a molecular pathway analysis report.

8. The breast cancer treatment response prediction method based on multi-temporal image images according to claim 7, wherein The frequency-domain amplitude spectrum normalization specifically includes: Calculating the average amplitude spectrum of the training set by grouping according to tumor size; Replacing the amplitude spectrum after performing Fourier transform on the test set images; Retaining the phase information for inverse transformation to reconstruct the normalized image.

9. An electronic device, comprising a memory, a processor, and a computer program stored on the memory, characterized in that: When the processor executes the program, it implements the steps of the breast cancer treatment response prediction method based on multi-temporal imaging images as described in any one of claims 7-8.

10. A computer-readable storage medium storing a computer program, characterized in that: When the program is executed by the processor, it implements the steps of the breast cancer treatment response prediction method based on multi-temporal imaging images as described in any one of claims 7-8.

Images