Breast cancer recurrence risk prediction method and system based on multi-modal data missing interpolation and gene interpretability enhancement
By using generative adversarial networks and gene-information-driven feature selection, the problems of missing data and insufficient interpretability in multimodal breast cancer recurrence risk prediction models are solved, achieving stable predictive performance and biological transparency, and supporting clinical applications.
Patent Information
- Application Number
- CN202610140001.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-02-02
- Publication Date
- 2026-03-06
AI Technical Summary
Existing multimodal breast cancer recurrence risk prediction models show performance degradation when faced with missing data and lack biological interpretability, affecting clinical application.
Generative adversarial networks (GANs) are used for adaptive imputation to generate realistic missing modal features. Combined with gene-information-driven feature selection and a visualization rendering engine, the biological interpretability of the model is enhanced.
By effectively utilizing existing modal information to fill in missing data, the robustness and clinical reliability of the model are improved, intuitive and visual auxiliary diagnostic information is provided, and personalized treatment decisions are supported.
Smart Images

Figure CN121617637A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of medical image processing and artificial intelligence technology, specifically to a method and system for predicting the risk of breast cancer recurrence based on multimodal data imputation and enhanced gene interpretability. Background Technology
[0002] Triple-negative breast cancer (TNBC) has always been a challenge in breast cancer diagnosis and treatment due to its aggressiveness, high recurrence rate, and lack of specific therapeutic targets. With the rapid development of medical imaging technology, a vast amount of multimodal data has been accumulated clinically, including ultrasound (US), mammography (MG), digital breast computed tomography (DBT), and magnetic resonance imaging (MRI). Single imaging modalities are limited by their imaging principles (e.g., ultrasound is insensitive to microcalcifications, and X-rays are limited in diagnosing dense breasts), making it difficult to comprehensively capture the heterogeneous characteristics of tumors. In contrast, multimodal imaging data can complement each other from different dimensions such as anatomical structure, blood perfusion, functional information, and tissue stiffness, providing a more comprehensive perspective for clinical diagnosis and treatment decisions.
[0003] In recent years, multimodal fusion technology has become a research hotspot. For example, a study published in *Nature Biomedical Engineering* in 2024 successfully developed the BMU-Net multimodal machine learning model by integrating MG, US, and clinical data. This model performed exceptionally well in breast cancer risk stratification and diagnostic tasks, achieving performance close to that of a pathologist's initial diagnosis. Another study in the same year also confirmed that a multimodal radiomics model combining MRI and MG has significant advantages in predicting axillary lymph node metastasis. Our previous research team significantly improved the accuracy of breast cancer diagnosis by constructing a radiomics model combining MG and MRI; a study combining US and MRI multimodal data further increased diagnostic accuracy from 77.9% to 85.6%. These results fully demonstrate the value of multimodal data in precision medicine.
[0004] However, despite the promising prospects of multimodal fusion, two key technical bottlenecks remain in practical clinical applications, limiting the robustness and clinical applicability of predictive models: First, there is a contradiction between the requirement for "completeness" of multimodal data and the "normality of missing data" in clinical practice. In long-term postoperative follow-up or retrospective studies, it is difficult to ensure that every patient completes a full set of (US / MG / DBT / MRI) examinations at every time point due to factors such as patient economic status, compliance, or hospital equipment scheduling. Existing multimodal models typically require complete input data modalities, and often adopt direct sample removal or simple zero / mean imputation strategies when faced with missing data. Sample removal leads to the waste of valuable data and model bias, while simple imputation ignores the potential nonlinear mapping relationships between different modalities and introduces noise, seriously affecting the accuracy of the model in predicting recurrence risk. Currently, how to utilize generative artificial intelligence technology to mine known modal information to reconstruct or approximate the features of missing modalities is a key direction for solving this problem.
[0005] Secondly, there is a gap between the "black box" nature of existing deep learning models and their biological interpretability. While deep learning has achieved significant success in quantitative image analysis, its decision-making process lacks transparency. Physicians and researchers struggle to understand the logic behind the model's output (i.e., what features the model uses to determine high risk), which severely hinders clinicians' trust in AI-assisted systems. More importantly, current interpretive methods (such as heatmaps) are largely limited to visual "localization," lacking qualitative analysis at the molecular biological level. Breast cancer recurrence (especially TNBC) is closely related to the expression of specific genes. Imaging genomics research shows that imaging phenotypes are macroscopic reflections of genotypes. However, most existing recurrence prediction models are driven solely by pure imaging features, failing to effectively link these features to the microscopic genetic and molecular mechanisms of tumors. If the imaging features extracted by the model cannot be proven to have clear biological significance, the model's predictions will be difficult to use to truly guide precise clinical decision-making. Summary of the Invention
[0006] To address the shortcomings of existing technologies, the present invention aims to provide a method and system for predicting breast cancer recurrence risk based on multimodal data imputation and enhanced gene interpretability.
[0007] To achieve the above objectives, the present invention provides the following technical solution: a method for predicting breast cancer recurrence risk based on multimodal data missing imputation and enhanced gene interpretability, comprising the following steps: S1: First, acquire multimodal imaging data of the patient before neoadjuvant therapy, during treatment, the last time before the end of treatment, and during postoperative follow-up. The multimodal imaging data includes at least two or more of US, MG, DBT, and MR, and corresponding clinicopathological data and gene detection data are collected simultaneously. Next, preprocess the acquired image data, including denoising, grayscale normalization, and histogram equalization, and use the TorchIO toolkit to perform data augmentation operations, including Gaussian blur, random rotation, and mirror flipping, to expand the sample size. Subsequently, use a finely tuned pre-trained large model for intelligent segmentation of the image data of different dimensions: the MedSAM 2D model is used for two-dimensional images (such as MG and US), and the SAM-Med 3D model is used for three-dimensional images (such as DBT and MRI), thereby obtaining accurate regions of interest (ROIs) for lesions. Furthermore, morphological dilation operations are performed using spherical structural elements to generate a peritumoral region mask reflecting the tumor microenvironment. Finally, based on the lesions and peritumoral masks, PyRadiomics radiomics features and ResNet-50 deep learning features were extracted in parallel. SLIC supervoxel segmentation combined with GMM clustering algorithm was used to extract habitat features that reflect the ecological diversity inside the tumor. Finally, a high-dimensional multimodal feature matrix that can comprehensively reflect the spatiotemporal heterogeneity of the tumor was constructed.
[0008] S2: To address the common problem of missing modalities in clinical follow-up, this invention designs an adaptive imputation mechanism. First, the input feature matrix is scanned to generate a binary modality missing mask with the same dimensions as the feature matrix, where "1" indicates the presence of modality data and "0" indicates missing modality data. Then, a pre-trained Generative Adversarial Network (GAN) model is constructed and invoked. This GAN model includes a generator and a discriminator. The generator receives modality features and random noise vectors present at the current time point as input, and synthesizes fitted feature vectors for missing modalities by learning the nonlinear mapping relationship between modalities. The discriminator distinguishes between generated features and real features. During the training phase, the generator parameters are optimized by jointly minimizing the L1 loss function and the adversarial loss function, ensuring that the generated feature distribution approximates the biological distribution of the real data as closely as possible. During the inference phase, the trained generator is used to fill in the missing terms in the feature matrix, thereby outputting a structurally complete temporal feature matrix without discarding samples.
[0009] To enhance the biological interpretability of the model and select key features, this invention constructs a multi-task deep neural network with ResNet-50 as the shared underlying layer. This network comprises two independent branches: a gene expression prediction branch and a recurrence risk prediction branch. During model training, biological constraints are introduced, and a binary cross-entropy loss function is calculated using gene expression labels to supervise the training of the gene prediction branch, forcing the shared encoder to capture image phenotypes closely related to gene mutations. Simultaneously, an automatic feature selection strategy is designed to select a "key feature subset" that simultaneously meets the following three conditions: (1) There is a significant statistical association with the expression of genes related to relapse (high gene predictive contribution). (2) It has a high weight contribution in the recurrence risk prediction branch (high recurrence prediction weight); (3) Low correlation between feature groups (low redundancy).
[0010] S4: The selected key features are serialized and sinusoidal positional encoding is added to preserve temporal order information, then input into a time-series prediction model based on the Transformer architecture. This invention innovatively introduces a modality missing mask into the Transformer's self-attention layer. When calculating attention weights, this masking mechanism dynamically adjusts weight allocation: smaller weights are automatically assigned to imputed features (Mask=0), while larger weights are assigned to the original, true features (Mask=1), effectively reducing the cumulative impact of imputation errors on long-term time-series prediction results. Furthermore, the model incorporates a temporal consistency loss function to constrain the prediction results at adjacent time points to maintain trend consistency, avoiding drastic fluctuations in risk probability, and finally calculating and outputting a smooth-transitioning recurrence risk probability.
[0011] S5: To assist physicians in intuitively understanding the model's decision-making basis, this invention integrates a visualization rendering engine. On one hand, it uses Grad-CAM technology to calculate gradients and generate activation-like heatmaps on the original images (US, MG, DBT, or MRI), highlighting the anatomical regions that contribute most to recurrence risk prediction, achieving "visual localization." On the other hand, it uses the SHAP algorithm to calculate the contribution of each input feature to the prediction result, generating a feature importance ranking map that intuitively displays the association path between image features and specific genes and recurrence risk, achieving "logical qualitative analysis." This multi-dimensional visualization interaction significantly enhances the system's transparency and clinical acceptability.
[0012] The beneficial effects of this invention are: 1. This invention innovatively introduces an adaptive imputation mechanism based on Generative Adversarial Networks (GANs), which can utilize existing modal features to mine nonlinear mapping relationships between modalities and generate highly realistic representations of missing modal features. This not only avoids the waste of valuable clinical data and maximizes the available sample size, but also, compared to traditional mean imputation or zero imputation, the features generated by GANs are more consistent with the real biological distribution, significantly reducing the noise introduced by imputation. This allows the system to maintain stable predictive performance even in real-world clinical scenarios (where incomplete examinations are common).
[0013] 2. This invention proposes a gene-information-driven feature selection strategy. Unlike traditional feature extraction that relies solely on labels (relapse / non-relapse), this invention introduces key gene expression states as supervisory signals during the training phase, forcing the model to select imaging features that can both predict relapse and have a clear molecular biological basis. This design not only achieves dimensionality reduction and redundancy removal of imaging features, but more importantly, it endows abstract features with clear biological meaning, demonstrating the association between imaging phenotypes and microscopic genotypes, thereby greatly enhancing doctors' trust and acceptability of the model's prediction results.
[0014] 3. This system integrates an advanced visualization rendering engine, capable of outputting intuitive anatomical heatmaps (Grad-CAM) and feature contribution maps (SHAP). The heatmap directly highlights high-risk areas on the original images, helping doctors quickly locate key lesion sites; the feature map clearly displays the logical chain of "genetic status - imaging features - recurrence risk." This interactive presentation transforms complex AI calculation results into easily understandable diagnostic information for doctors, effectively supporting doctor-patient communication and personalized treatment decisions. Attached Figure Description
[0015] Figure 1 This is a technical roadmap for the present invention.
[0016] Figure 2 This is a schematic diagram of multimodal data missing interpolation according to the present invention.
[0017] Figure 3 This is a schematic diagram of the image feature screening and multi-source information fusion based on gene information driven by the present invention.
[0018] Figure 4 This is a schematic diagram of the multimodal image feature screening and recurrence risk prediction model construction method of the present invention. Detailed Implementation
[0019] like Figure 1As shown, this invention discloses a method and system for predicting breast cancer recurrence risk based on multimodal data missing imputation and enhanced gene interpretability, which includes the following 5 steps: S1, This embodiment details the specific process of constructing a high-quality multimodal temporal feature matrix, providing standardized mathematical input for subsequent models. First, the research subjects are defined as patients with pathologically confirmed TNBC, and a time index set is established. These correspond to the pre-neoadjuvant therapy, during treatment, pre-operative, and post-operative follow-up periods, respectively. A modal index set is defined. These correspond to US, MG, DB, and MRI, respectively.
[0020] To address the heterogeneity of multi-source data, a normalization preprocessing step is performed: all 3D images are resampled to an isotropic voxel space, and denoising algorithms are first used to denoise the images. Subsequently, Z-score normalization is applied to the image voxels. For arbitrary modalities... Image Its body elements Normalized gray value The calculation is as follows:
[0021] in, and These are the global grayscale mean and standard deviation within the image, respectively.
[0022] Then, the lesion mask is obtained using the finely tuned MedSAM 2D and SAM-Med 3D large models. To capture the tumor microenvironment, a morphological dilation algorithm is introduced to construct a peritumoral mask. .set up For a spherical structural element with a radius of 6 mm, the expansion operation is defined as follows: Then the final region of interest (peritumoral region + lesion region) is determined. The calculation formula is:
[0023] Based on the extracted region of interest Generate ROI images containing only the lesion and peritumoral region. Then, three types of features (deep learning features, radiomics features, and tumor habitat features) are extracted in parallel. 1) Deep Learning Feature Extraction: High-dimensional abstract features are extracted using a deep learning network (ResNet-50) pre-trained on large-scale medical images. The images... Input deep neural network encoder Extracting the output of the deep hidden layers of the network as deep learning feature vectors The formula is as follows:
[0024] 2) Radiomics Feature Extraction: High-throughput feature computation is performed on the ROI region to generate a radiomics feature set. This module uses radiomics algorithms to calculate a comprehensive feature vector from the ROI. This vector contains the following feature subset:
[0025] in: Features of the first-order histogram (mean, variance, skewness, kurtosis); Shape characteristics (volume, surface area, compactness); Texture features (calculated based on GLCM, GLRLM, and GLSZM matrices); These are filtering features based on Gaussian and wavelet transforms.
[0026] 3) The tumor habitat feature extraction module is configured to quantify the spatial heterogeneity within the tumor and generate an ecological diversity index feature set. Its processing flow includes three steps: supervoxel pre-segmentation, unsupervised clustering, and diversity index calculation.
[0027] C1) Hypervoxel pre-segmentation uses a linear iterative clustering algorithm to divide the tumor region into N hypervoxel subregions. Define the... The pixel and the The combined distance of each cluster center as follows:
[0028]
[0029]
[0030] in Indicates the intensity value of multimodal image. Represents the spatial coordinates of a pixel. The distance is the image intensity value. Spatial distance; The preset grid step size, A compactness parameter is used to balance intensity similarity and spatial proximity.
[0031] C2) Habitat subregion clustering optimization: Based on the image features of subregions, the K-means algorithm is used for clustering, and the optimal number of clusters is determined by the Calinski-Harabasz (CH) index. (i.e., the optimal number of habitats). The CH index corresponding to different k values is calculated, and the k value that maximizes the CH index is selected as the optimal number of clusters. The formula for calculating the CH index is:
[0032] in: This represents the current number of clusters. This represents the total number of samples in the sub-region. Represents the trace of a matrix; The cluster scatter matrix is the matrix of scatter within the cluster. Let be the inter-cluster scatter matrix, defined as follows:
[0033]
[0034] Here, Indicates the first The sample set of each cluster For the first The center of each cluster, The global center of all samples. For the first Number of samples in each cluster.
[0035] C3) Feature set generation, selecting features that... The largest Value as Based on the optimal clustering result, a tumor ecological diversity index feature set is generated. .
[0036] After obtaining the above three types of features, feature fusion and tensor quantization operations are performed, such as... Figure 3 As shown. For a specific point in time. and specific modes The normalized three types of feature vectors are concatenated and spliced to obtain the comprehensive feature vector of that mode at that moment. :
[0037] Finally, iterate through all time points and modalities, if the first... Time of the first If the modal data actually exists, then the calculated data will be... Fill in the corresponding positions in the tensor; if data is missing, explicitly set that position to a vector of all zeros. This constructs the initial multimodal temporal feature tensor. This tensor fully preserves the structural information and missing state distribution of the multimodal time series data, and is directly used as the input to the GAN interpolation network in step S2.
[0038] S2, This embodiment details how to use a GAN network architecture to process the initial feature tensor output from the previous step. The logic for performing high-fidelity completion, such as Figure 2 As shown in the figure. This process aims to infer and reconstruct the feature representation of missing modalities by utilizing existing multimodal data distribution patterns.
[0039] First, the input tensor definition and mask matrix are constructed. The input data directly inherits the initial multimodal temporal feature tensor constructed in the previous step. The system first executes the mask generation program to construct and... Binary mask tensors with identical dimensions Its construction logic is as follows: traverse each modal channel of the tensor, if the... Time of the first If the modal data actually exists in clinical practice (i.e., a non-zero vector), then the mask slice corresponding to that channel is used. Set all values to 1; if data is missing, set all values to 0. Then, a GAN network is constructed, which includes a generator. ) and a discriminator Both improve the biological realism of the imputed data through adversarial training. The generator of the GAN network... It uses the RRDB architecture. Its input is a mask tensor. Filtered observation features (i.e., the missing position is zero), the generator outputs a fitted feature tensor for the missing mode through a nonlinear mapping layer. Discriminator A VGG network is employed, with its input being a complete feature tensor (containing either a real multimodal feature matrix or multimodal feature moments generated by the generator). Discriminator. The output layer is a Sigmoid activation function, which outputs a probability scalar between [0, 1], representing the probability that the input feature belongs to the true modality feature matrix.
[0040] To ensure that the generated features both approximate the distribution of the real data (constrained by the discriminator) and maintain structural consistency with the known observed data (constrained by the reconstruction loss). This embodiment constructs a composite loss function. Specifically, the overall training objective function of the GAN network is to minimize the generator's loss and maximize the discriminator's gain, as shown in the following formula:
[0041] in, To balance hyperparameters and combat loss. A binary cross-entropy approach is used to drive the generator to "deceive" the discriminator, making the feature distributions it generates difficult to distinguish.
[0042] In the formula, The distribution representing the true feature data, This represents the distribution of data generated in the simulation.
[0043] Reconstruction loss Using L1 norm constraints, only in the mask Values are generated by calculating values in regions (i.e., real-world modal data). Compared with the true value The difference between them is calculated using the following formula:
[0044] In the formula, This refers to the Hadamard product.
[0045] After training, use the trained generator For the initial feature tensor The missing terms in the feature tensor are filled in to obtain the completed feature tensor. Its calculation logic is as follows:
[0046] This tensor will serve as the standard input for step S3.
[0047] Step S3 aims to enhance the model's biological interpretability and screen key features. By introducing gene expression numbers as an auxiliary supervision signal, the model is forced to screen for image features with a clear molecular biological basis, such as... Figure 4 As shown.
[0048] This embodiment constructs a neural network architecture that includes a shared encoder and dual-branch output: 1) Shared feature encoder: The ResNet-50 architecture is used as the shared feature encoder. Backbone network. Input is the interpolated feature tensor output from step S2. Map it to a high-dimensional latent feature vector The encoder is responsible for extracting deep abstract representations of the image.
[0049] 2) Gene prediction branch: Consists of fully connected layers and a sigmoid activation function. Its task is to predict genes based on latent feature vectors. Predict the mutation status (high expression / low expression) of key genes.
[0050] 3) Recurrence Risk Branch: Composed of fully connected layers and a sigmoid activation function. Its task is to determine the recurrence risk based on the same latent feature vector. Predict the probability of a patient's recurrence risk.
[0051] To enable gene information to guide image feature extraction, a joint loss function is established. To simultaneously optimize network weights:
[0052] in, Binary cross-entropy loss for recurrence prediction; Hyperparameters for adjusting the strength of gene supervision. The binary cross-entropy loss function is used to supervise the prediction of gene mutation states. The specific formula is as follows:
[0053] In the formula, For the sample size, For the first The true genetic label of each sample Predict probabilities for the model.
[0054] After the network training converges, extract the weights of the fully connected layers with recurrent branches. and gene branch fully connected layer weights To select features that can predict relapse, reflect gene status, and are non-redundant, we define the feature... Overall importance score :
[0055] The formula contains three key constraints: 1) Numerator : Ensure that the feature contributes significantly to the risk of recurrence.
[0056] 2) Numerator : Ensure that the characteristics are highly correlated with gene expression.
[0057] 3) Denominator terms By using the Pearson correlation coefficient to penalize features that are highly correlated with other features, the selected feature set is ensured to have low redundancy and high complementarity.
[0058] in accordance with The features are sorted in descending order, and the top-K features are extracted to construct the final key feature sequence. This serves as the input for step S4.
[0059] S4: This embodiment utilizes the Transformer architecture to handle long temporal dependencies and introduces a special attention mechanism for the "pseudo-data" generated by interpolation in step S2 to eliminate accumulated errors. Since the Transformer itself does not have the ability to handle temporal order, it first processes the input feature sequence... Add sine position encoding To preserve chronological information:
[0060] in, By using sine and cosine functions of different frequencies, the model is ensured to be perceptive. (Before neoadjuvant therapy) to The evolution of (postoperative follow-up) over time.
[0061] To distinguish between real observation data and GAN-impaired data and prevent the model from over-relying on impaired values, this embodiment improves the standard self-attention mechanism by introducing a dynamic mask matrix corresponding to the feature sequence. (Its value originates from step S2) The corrected attention score is calculated as follows:
[0062] in, These are the query, key, and value matrices, respectively. This is the scaling factor. The key lies in the mask penalty term. Its definition is:
[0063] By introducing penalty items ( (where the value is a large positive number), during the Softmax normalization step, the weights corresponding to the time steps generated by the interpolation will be significantly compressed. This allows the model to automatically reduce its focus on the interpolated data when aggregating global information, and to mainly rely on the real observation data for inference, thereby improving the robustness of the prediction.
[0064] At the same time, to avoid the recurrence risk probability predicted by the model from being at adjacent time points (such as... arrive The occurrence of drastic fluctuations that violate biological laws introduces a loss of time consistency. Constraint model output recurrence probability :
[0065] in The output features of the Transformer encoder are mapped to recurrence risk probabilities through a fully connected layer. The trained model can then output the patient's recurrence risk probability throughout the entire follow-up period. arrive Recurrence risk probability sequence The system stratifies patients based on preset clinical thresholds: S5, this step aims to transform the "black box" decision-making process of deep learning models into visual information that clinicians can understand, specifically including two dimensions: anatomical localization visualization and feature attribution visualization.
[0066] (1) For samples predicted by the model to have a high risk of relapse, the convolutional neural network (ResNet-50) from which features were extracted in step S1 is backtracked using Grad-CAM technology. First, the relapse risk score is calculated. (i.e., prediction probability) (Relative to the feature map of the last convolutional layer of ResNet-50) The gradient is calculated. The importance weight of the k-th feature map to the target class is calculated using global average pooling. :
[0067] In the formula, This represents the total number of pixels in the feature map. This is the gradient value. Subsequently, the feature map is linearly weighted and summed using this weight, and a ReLU activation function is applied to remove negative activations (i.e., only regions that positively contribute to recurrence are retained), generating an anatomical heatmap. :
[0068] Finally, Upsampled to the resolution of the original image (US, MG, DBT, or MRI) and overlaid for display. The highlighted areas in the heatmap are the "high-risk lesion areas" or "peritumoral invasion areas" of interest to the model, assisting doctors in precise localization.
[0069] (2) To quantitatively explain the specific contribution of "gene-image" features to relapse risk, this embodiment employs the SHAP (Shapley Additive Explanations) algorithm. For the key feature sequences screened in step S3... Each feature in the dataset has a Shapley value. Marginal contribution calculation based on game theory:
[0070] In the formula, The set of all input features. For feature subset, This is a prediction model.
[0071] The system according to The sign and magnitude of a radiomics feature are used to generate a ranking plot of feature importance. If a certain radiomics feature... Furthermore, if the value is large and the feature is identified as highly correlated with the gene in step S3, the system will output an association prompt.
[0072] The present invention also discloses a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method for improving the interpretability of a breast cancer recurrence risk prediction model based on gene information-driven feature screening.
[0073] The readable storage medium may include a readable medium in the form of volatile memory, such as random access memory (RAM) and / or cache memory, and may further include read-only memory (ROM). Program code is stored thereon, which is executed by a processor to cause the processor to perform the steps of the embodiments described in this specification.
[0074] The memory may also include programs / utilities having a set (at least one) of program modules, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.
[0075] A bus can represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus that uses any of the various bus structures.
[0076] The present invention also discloses an electronic device, including a memory, a processor, and a computer program stored in the memory. When the processor executes the program, it implements the above-mentioned method for improving the interpretability of the breast cancer recurrence risk prediction model based on gene information-driven feature screening.
[0077] The electronic device can also communicate with one or more external devices (e.g., keyboards, pointing devices, Bluetooth devices, etc.), one or more devices that enable a user to interact with the electronic device, and / or any device that enables the electronic device to communicate with one or more other computing devices (e.g., routers, modems, etc.). This communication can be performed via input / output (I / O) interfaces. Furthermore, the electronic device can communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter. The network adapter communicates with other modules of the electronic device via a bus. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with the electronic device, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.
[0078] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.
[0079] The embodiments should not be regarded as limitations on the present invention, but any improvements made based on the spirit of the present invention should be within the protection scope of the present invention.
Claims
1. A breast cancer recurrence risk prediction method based on multi-modal data missing imputation and gene interpretability enhancement, characterized in that: It comprises the following steps: S1: Multimodal time series data acquisition and feature extraction, acquiring multimodal time series data of patients at different time points, after data enhancement preprocessing, using fine-tuned pre-trained large model combined with morphological algorithm, respectively processing different modal image data to obtain the region of interest; Based on the region of interest, ResNet-50 deep learning features, PyRadiomics imageomics features, and tumor habitat features based on simple linear iterative clustering super voxel segmentation and Gaussian mixture model clustering are extracted in parallel; Finally, a high-dimensional multimodal feature matrix reflecting the spatiotemporal heterogeneity of tumor is constructed; S2: Missing data imputation based on generative adversarial network, generating a binary mask representing the existence state of each modality; Using a pre-trained generative adversarial network, the generator performs nonlinear mapping and synthesizes the feature vector of the missing modality according to the existing modality features; And through the adversarial training of the discriminator, ensure that the generated features conform to the true biological distribution, so as to output a complete time series feature matrix structure; S3: Key feature screening and fusion driven by genes, construct a multi-task deep neural network with ResNet-50 as the shared bottom layer, and perform joint supervised training through the gene prediction branch and the recurrence risk branch; Automatically select a key feature subset that meets the high contribution of gene prediction, high weight of recurrence prediction, and low correlation between groups, realize the fusion and dimensionality reduction of image phenotype and microscopic gene molecular features; S4: Time series prediction based on dynamic mask attention, input the filtered feature sequence into the Transformer architecture model integrated with position encoder, use the dynamic mask attention mechanism to automatically reduce the weight of the interpolated generated features in the inference stage to reduce the cumulative error; And combine the time consistency loss function to output a smooth recurrence risk probability; S5: Visualization interaction and clinical decision support, using a visualization rendering engine, generating a risk area anatomical heat map superimposed on the original image based on the Grad-CAM algorithm and generating a feature importance map based on the SHAP algorithm to show the "gene, image and recurrence association path, assist clinical decision making. 2.The breast cancer recurrence risk prediction method based on multi-modal data missing imputation and gene interpretability enhancement according to claim 1, wherein, The multimodal time series data obtained in step S1 includes: multimodal image data of patients before neoadjuvant therapy, during therapy, the last time before therapy ends, and during postoperative follow-up, including breast ultrasound, mammography, digital breast tomosynthesis, and multi-parameter magnetic resonance imaging sequence data, as well as synchronous acquisition of clinical pathology and gene detection data, The preprocessing of the multimodal time series data includes at least one of denoising, gray scale normalization and histogram equalization, and data enhancement operations are implemented using the TorchIO toolkit, including at least one of Gaussian blur, random rotation and mirror flipping.
3. The breast cancer recurrence risk prediction method based on multimodal data missing imputation and gene interpretability enhancement of claim 2, wherein: The determination process of the region of interest is: for two-dimensional and three-dimensional image data, respectively using pre-trained MedSAM 2D and SAM-Med 3D models for lesion segmentation, obtaining lesion mask, and using spherical structure element for morphological dilation to generate peritumoral region mask.
4. The breast cancer recurrence risk prediction method based on multi-modal data missing imputation and gene interpretability enhancement of claim 3, wherein: Based on the lesion mask and the peritumoral region mask, the following three types of features are extracted in parallel to construct a multi-dimensional feature set reflecting tumor heterogeneity: imageomics features extracted based on PyRadiomics, deep features extracted based on ResNet-50, and tumor habitat features extracted based on SLIC super voxel segmentation and GMM clustering method. 5.The breast cancer recurrence risk prediction method based on multi-modal data missing imputation and gene interpretability enhancement according to claim 1, wherein, The specific implementation of the step S2 comprises: First, the input feature matrix is scanned to generate a modality missing mark mask consistent with the dimension of the feature matrix, which is used to mark the missing situation of each modality data at each time point. If there is modality data at a certain time point, it is marked as 1, and if it is missing, it is marked as 0; Subsequently, a generative adversarial network including a generator and a discriminator is constructed, wherein the generator takes the existing modality features at the current time point and a noise vector as input, learns the mapping relationship from the available modalities to the missing modalities, and outputs the fitted feature vector of the missing modalities, and the discriminator is used to distinguish between the generated features output by the generator and the real features; In the training stage, the difference between the generated features and the real features is measured by minimizing the L1 loss, and joint optimization is performed combined with the adversarial loss function, so that the generated feature distribution approximates the biological distribution of the real data. In the inference stage, the trained generator is used to fill in the missing items in the feature matrix. 6.The breast cancer recurrence risk prediction method based on multi-modal data missing imputation and gene interpretability enhancement according to claim 1, wherein, The specific steps of the step S3 comprise: S31, a deep neural network including a shared encoder and two independent branches is constructed, wherein the shared encoder adopts ResNet-50 architecture and is used to extract deep abstract representations from breast medical images, and the two independent branches are respectively a gene expression prediction branch and a recurrence risk prediction branch, which respectively receive the features output by the shared encoder and perform independent prediction; S32, a binary cross-entropy loss function is calculated using gene expression labels to supervise the training of the gene expression prediction branch, forcing the shared encoder to capture image features related to gene mutations; S33, a three-condition feature screening strategy is implemented during the training process, and only image features that meet the following conditions are retained to the final feature set, satisfy the statistically significant correlation with recurrence-related gene expression; have high weight contribution in the recurrence risk prediction branch; low correlation between feature groups to provide complementary information.
7. The breast cancer recurrence risk prediction method based on multimodal data missing imputation and gene interpretability enhancement of claim 1, wherein, The step S4 comprises: The feature matrix after interpolation and screening is serialized and sine position encoding is added to retain the time sequence information, and then input into the Transformer encoder; A modality missing mask is introduced in the self-attention layer of the Transformer, which gives lower weight to the interpolated features and higher weight to the original existing features when calculating the attention weight, thereby reducing the cumulative error brought by the interpolation process; At the same time, a temporal consistency loss function is introduced to constrain the model, so that the prediction results of adjacent time points remain consistent in trend and avoid sharp fluctuations. 8.The breast cancer recurrence risk prediction method based on multi-modal data missing imputation and gene interpretability enhancement according to claim 1, wherein, The step S5 comprises: Integrate a visualization rendering engine, on the one hand, use Grad-CAM technology to generate class activation maps on the original US, MG, DBT or MRI images based on gradients, and highlight the anatomical regions that contribute most to the prediction of recurrence risk; In another aspect, the SHAP value is used to calculate the contribution of each input feature to the prediction result of the recurrence risk, and a feature importance ranking graph is generated to intuitively present the correlation between the image features, genes and the recurrence risk, thereby assisting clinical decision-making.
9. An electronic device, comprising: The device comprises a processor and a memory, and the memory stores a computer program which, when executed by the processor, implements the breast cancer recurrence risk prediction method based on multi-modal data missing imputation and gene interpretability enhancement according to any one of claims 1 to 8.
10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program which, when executed by the processor, implements the breast cancer recurrence risk prediction method based on multi-modal data missing imputation and gene interpretability enhancement according to any one of claims 1 to 8.
Citation Information
Patent Citations
Guided multi-modal image genetics data feature analysis method
CN114202075A
Multi-modal medical missing data completion method and device based on data relevance mining
CN116795826A
Thyroid tumor classification and recurrence risk prediction method based on deep learning
CN120600315A
Endometrial cancer molecular typing deep learning prediction algorithm based on multi-modal data fusion
CN120674070A
Cited By
A medical electronic medical record missing value collaborative imputation method based on a knowledge graph
CN122177335A
Nutrition decision-making method and system integrating diagnosis and treatment evaluation based on multi-task learning
CN122290895A
